diff --git a/docs/dev/rest-api/endpoints.md b/docs/dev/rest-api/endpoints.md index 8b65f080..5428b194 100644 --- a/docs/dev/rest-api/endpoints.md +++ b/docs/dev/rest-api/endpoints.md @@ -17,7 +17,7 @@ Start the server with [`unshackle serve`](../../guide/cli-reference.md). By defa - **CORS.** Every response carries permissive CORS headers, and the server answers `OPTIONS` preflight requests automatically. - **Compression.** JSON responses of 256 bytes or more are gzip-compressed when the client sends `Accept-Encoding: gzip`. A [slow response](#slow-responses) is not compressed. - **Service allowlist.** The effective allowlist for your API key filters the service-facing endpoints (the intersection of the global `serve.services` list and your per-key list). The server treats services you cannot use as unknown. -- **Remote services.** A full-mode server can run a service through one of its own `remote_services` servers (see [Remote sessions](remote-sessions.md)). `GET /api/config` lists their names in `remote_services`, and `GET /api/services?remote=` lists that server's services, each with `remote` set to the name. Add `"remote": ""` to `POST /api/search`, `/api/list-titles`, `/api/list-tracks` or `/api/download`, and the server opens a remote session with its own key for that server: the remote logs in and resolves titles, tracks and licences, and a download still runs, decrypts and muxes here. The tag may be any case or an alias, and your allowlist applies to it. An unknown name is `400 INVALID_PARAMETERS`, a server that does not answer is `SERVICE_UNAVAILABLE`, and a tag it does not offer is `INVALID_SERVICE`. List calls for the same title, profile, proxy and service options share one remote session, which closes after 120 seconds unused. An optional `client_name` names your client on the remote server's dashboard as `serve ()`. A `--remote-only` server does not relay. A remote login that waits on you (a pair code or URL to approve on another device) cannot answer through a list call, so poll `GET /api/remote/logins` alongside it: each entry carries the login's `notice`, and its `prompt` when it asks a question (an OTP), which you answer with `POST /api/remote/logins/{id}/input` and `{"response": "..."}`. A remote download job shows the same text as its own `notice`, and asks a question (an OTP) through `input_prompt` as a local job does. +- **Remote services.** A full-mode server can run a service through one of its own `remote_services` servers (see [Remote sessions](remote-sessions.md)). `GET /api/config` lists their names in `remote_services`, and `GET /api/services?remote=` lists that server's services, each with `remote` set to the name. Add `"remote": ""` to `POST /api/search`, `/api/list-titles`, `/api/list-tracks` or `/api/download`, and the server opens a remote session with its own key for that server: the remote logs in and resolves titles, tracks and licences, and a download still runs, decrypts and muxes here. The tag may be any case or an alias, and your allowlist applies to it. An unknown name is `400 INVALID_PARAMETERS`, a server that does not answer is `SERVICE_UNAVAILABLE`, and a tag it does not offer is `INVALID_SERVICE`. List calls for the same title, profile, proxy and service options share one remote session, which closes after 120 seconds unused. An optional `client_name` names your client on the remote server's dashboard as `serve ()`. A relayed search opens no remote session. The server sends its own credentials, cookies and cache for the `profile` with the search. A service that searches with no login gets none of them. A `--remote-only` server does not relay. A remote login that waits on you (a pair code or URL to approve on another device) cannot answer through a list call, so poll `GET /api/remote/logins` alongside it: each entry carries the login's `notice`, and its `prompt` when it asks a question (an OTP), which you answer with `POST /api/remote/logins/{id}/input` and `{"response": "..."}`. A remote download job shows the same text as its own `notice`, and asks a question (an OTP) through `input_prompt` as a local job does. !!! warning "Error responses have two shapes" Structured application errors (raised inside a handler) return `{"status": "error", "error_code": "...", "message": "...", "timestamp": "..."}`. The authentication middleware, however, returns a **different** shape: `{"status": 401, "message": "..."}` with an integer `status` and no `error_code`. The same middleware answers `429` in that shape when an API key goes over its rate limit. Clients must accept both shapes. The [Error responses](#error-responses) section at the bottom of this page gives the full error model. diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index 257eedbd..35d737b1 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -250,12 +250,15 @@ unshackle logs in to the service before the search. A service that declares a se | `-p`, `--profile` | Profile for credentials and cookies. | | `--proxy` | Proxy URI, country or location code, or `provider:region` (e.g. `controld:ca`). | | `--no-proxy` | Force-disable all proxy use. | +| `--remote` | Search on a remote unshackle server. The server logs in to the service. unshackle sends it your credentials, cookies and cache for the profile, and saves the cache that the server sends back. A service that searches with no login gets none of them. | +| `--server` | Name a remote server from the `remote_services` config. | !!! example ```shell unshackle search EXAMPLE "My Show" unshackle search -p myprofile EXAMPLE "Another Show" unshackle search --proxy nordvpn:ca SERVICE "query" + unshackle search --remote --server my_server SERVICE "query" ``` --- diff --git a/tests/remote/unit/test_remote_search.py b/tests/remote/unit/test_remote_search.py new file mode 100644 index 00000000..a76d4be2 --- /dev/null +++ b/tests/remote/unit/test_remote_search.py @@ -0,0 +1,71 @@ +"""RemoteService.search: the body ``search --remote`` posts, and the results it yields.""" + +from __future__ import annotations + +from types import SimpleNamespace +from typing import Any + +import pytest + +from unshackle.core import remote_service +from unshackle.core.remote_service import RemoteService + +pytestmark = pytest.mark.unit + + +class FakeClient: + server_url = "https://server.example" + + def __init__(self) -> None: + self.sent: dict[str, Any] = {} + + def post(self, endpoint: str, data: dict[str, Any]) -> dict[str, Any]: + self.sent = {"endpoint": endpoint, **data} + return {"results": [{"id": "t1", "title": "Show", "label": "2019"}], "cache": {"token": "x"}} + + +def make_service(params: dict[str, Any], server_accounts: Any = None) -> tuple[RemoteService, FakeClient, list]: + svc = RemoteService.__new__(RemoteService) + client = FakeClient() + saved: list = [] + svc.service_tag = "TEST" + svc.title_id = "query" + svc.client = client # type: ignore[assignment] + svc.ctx = SimpleNamespace(parent=SimpleNamespace(params=params), obj=None) # type: ignore[assignment] + svc._server_accounts = server_accounts + svc._anonymous_search = False + svc.load_cache_files = lambda profile=None: {"token": "old"} # type: ignore[method-assign] + svc.save_returned_cache = lambda data, profile=None: saved.append(data) # type: ignore[method-assign] + return svc, client, saved + + +def test_search_sends_client_login_and_yields_results(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(remote_service, "load_credentials_for_transport", lambda tag, profile: {"username": "u"}) + monkeypatch.setattr(remote_service, "load_cookies_for_transport", lambda tag, profile: None) + svc, client, saved = make_service({"profile": "p1", "no_proxy": True}) + + results = list(svc.search()) + + assert client.sent == { + "endpoint": "/api/search", + "service": "TEST", + "query": "query", + "profile": "p1", + "credentials": {"username": "u"}, + "cache": {"token": "old"}, + "no_proxy": True, + } + assert [(r.id, r.title, r.label) for r in results] == [("t1", "Show", "2019")] + assert saved == [{"token": "x"}] + + +@pytest.mark.parametrize( + ("server_accounts", "anonymous"), [({"global": True}, False), (None, True)], ids=["server-account", "anonymous"] +) +def test_search_sends_no_login(server_accounts: Any, anonymous: bool) -> None: + svc, client, _ = make_service({}, server_accounts) + svc._anonymous_search = anonymous + + list(svc.search()) + + assert client.sent == {"endpoint": "/api/search", "service": "TEST", "query": "query"} diff --git a/unshackle/commands/search.py b/unshackle/commands/search.py index 7763cdb3..bea3f599 100644 --- a/unshackle/commands/search.py +++ b/unshackle/commands/search.py @@ -39,8 +39,29 @@ from unshackle.core.utils.redact import mask_proxy help="Proxy URI to use. If a 2-letter country is provided, it will try get a proxy from the config.", ) @click.option("--no-proxy", is_flag=True, default=False, help="Force disable all proxy use.") +@click.option( + "--remote", + is_flag=True, + default=False, + is_eager=True, + help="Search on a remote unshackle server instead of with local service code.", +) +@click.option( + "--server", + type=str, + default=None, + is_eager=True, + help="Name of the remote server from remote_services config (if multiple configured).", +) @click.pass_context -def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, proxy: Optional[str] = None): +def search( + ctx: click.Context, + no_proxy: bool, + profile: Optional[str] = None, + proxy: Optional[str] = None, + remote: bool = False, + server: Optional[str] = None, +): if not ctx.invoked_subcommand: raise ValueError("A subcommand to invoke was not specified, the main code cannot continue.") @@ -52,14 +73,14 @@ def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, pr if profile: log.info(f"Using profile: '{profile}'") - with console.status("Loading Service Config...", spinner="dots"): - service_config_path = Services.get_path(service) / config.filenames.config - if service_config_path.exists(): - service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8")) - log.info("Service Config loaded") - else: - service_config = {} - merge_dict(config.services.get(service), service_config) + service_config: dict = {} + if not remote: + with console.status("Loading Service Config...", spinner="dots"): + service_config_path = Services.get_path(service) / config.filenames.config + if service_config_path.exists(): + service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8")) + log.info("Service Config loaded") + merge_dict(config.services.get(service), service_config) proxy_providers: list[Proxy] = [] if no_proxy: @@ -91,12 +112,12 @@ def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, pr @search.result_callback() -def result(service: Service, profile: Optional[str] = None, **_: Any) -> None: +def result(service: Service, profile: Optional[str] = None, remote: bool = False, **_: Any) -> None: log = logging.getLogger("search") service_tag = service.__class__.__name__ - if not service.ANONYMOUS_SEARCH: + if not remote and not service.ANONYMOUS_SEARCH: with console.status("Authenticating with Service...", spinner="dots"): cookies = dl.get_cookie_jar(service_tag, profile) credential = dl.get_credentials(service_tag, profile) @@ -117,7 +138,7 @@ def result(service: Service, profile: Optional[str] = None, **_: Any) -> None: result_text += f"\n[bright_black]id: {result.id}[/]" search_results.add(result_text + "\n") - cookie_file = None if service.ANONYMOUS_SEARCH else dl.get_cookie_path(service_tag, profile) + cookie_file = None if remote or service.ANONYMOUS_SEARCH else dl.get_cookie_path(service_tag, profile) if cookie_file: dl.save_cookies(cookie_file, service.session.cookies) diff --git a/unshackle/core/api/handlers.py b/unshackle/core/api/handlers.py index 4eda644a..fd89d443 100644 --- a/unshackle/core/api/handlers.py +++ b/unshackle/core/api/handlers.py @@ -481,11 +481,11 @@ def relay_error(exc: Any, service: str) -> APIError: return APIError(APIErrorCode.SERVICE_ERROR, exc.message, details={"service": service, "remote_code": exc.kind}) -def setup_remote_service( +def relay_service( data: Dict[str, Any], server: str, profile: Optional[str], title_id: str, request: Optional[web.Request] = None ) -> Any: - """Open and authenticate a remote session and fetch its titles; the caller must close() it.""" - from unshackle.core.remote_service import RemoteError, make_remote_service + """A RemoteService for ``data["service"]`` on a ``remote_services`` server, with this server's login files.""" + from unshackle.core.remote_service import make_remote_service # the same server_proxy gate a local service gets: without it the providers stay empty proxy_param, providers = resolve_handler_proxy(data, data["service"], request) @@ -500,7 +500,16 @@ def setup_remote_service( ) ctx = click.Context(click.Command(data["service"]), parent=parent_ctx, obj=parent_ctx.obj) # this process is long-lived and serves local services too, so the remote's overrides stay out of config - service_instance = make_remote_service(ctx, server, data["service"], title_id, data, apply_overrides=False) + return make_remote_service(ctx, server, data["service"], title_id, data, apply_overrides=False) + + +def setup_remote_service( + data: Dict[str, Any], server: str, profile: Optional[str], title_id: str, request: Optional[web.Request] = None +) -> Any: + """Open and authenticate a remote session and fetch its titles; the caller must close() it.""" + from unshackle.core.remote_service import RemoteError + + service_instance = relay_service(data, server, profile, title_id, request) login = RemoteLogin(caller_key(request), server, data["service"], title_id, service_instance) service_instance._input_bridge = login.bridge _remote_logins[login.id] = login @@ -725,14 +734,21 @@ def remote_services_info(name: str, allowed: Optional[List[str]]) -> List[Dict[s return [{**svc, "remote": name} for svc in remote_list(name) if allowed is None or svc.get("tag") in allowed] -def remote_search(server: str, service: str, query: str) -> Dict[str, Any]: - """Run a search on a ``remote_services`` server with this server's key for it.""" - from unshackle.core.remote_service import RemoteError, remote_client +def remote_search( + data: Dict[str, Any], server: str, query: str, request: Optional[web.Request] = None +) -> Dict[str, Any]: + """Run a search on a ``remote_services`` server, sending this server's login files as ``search --remote`` does.""" + from unshackle.core.remote_service import RemoteError + service_instance = relay_service(data, server, client_profile(data), query, request) try: - return remote_client(server).post("/api/search", {"service": service, "query": query}) + results = [result.to_dict() for result in service_instance.search()] except RemoteError as exc: - raise relay_error(exc, service) from None + raise relay_error(exc, data["service"]) from None + finally: + service_instance.session.close() + service_instance.client.session.close() + return {"results": results, "count": len(results)} def setup_list_service( @@ -851,15 +867,7 @@ def run_service_search( results: List[Dict[str, Any]] = [] try: for result in service_instance.search(): - results.append( - { - "id": result.id, - "title": result.title, - "description": result.description, - "label": result.label, - "url": result.url, - } - ) + results.append(result.to_dict()) except NotImplementedError: raise APIError( APIErrorCode.SERVICE_ERROR, @@ -1861,7 +1869,8 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] = remote = await asyncio.to_thread(remote_server, data, request) if data.get("remote") else None if remote: - return web.json_response(await asyncio.to_thread(remote_search, remote[0], remote[1]["tag"], query)) + data = {**data, "service": remote[1]["tag"]} + return web.json_response(await asyncio.to_thread(remote_search, data, remote[0], query, request)) # get_tag echoes an unknown tag back, so it can never be falsy; validate_service # is the check that actually resolves the service directory and the allowlist. diff --git a/unshackle/core/remote_service.py b/unshackle/core/remote_service.py index d134a323..d87be35a 100644 --- a/unshackle/core/remote_service.py +++ b/unshackle/core/remote_service.py @@ -41,6 +41,7 @@ from unshackle.core.constants import AnyTrack from unshackle.core.credential import Credential from unshackle.core.drm import cbcs, real_kids from unshackle.core.proxies.resolve import is_loopback +from unshackle.core.search_result import SearchResult from unshackle.core.session import MAX_BACKOFF from unshackle.core.titles import Title_T, Titles_T, remap_titles from unshackle.core.titles.episode import Episode, Series @@ -852,6 +853,7 @@ def make_remote_service( info = next((s for s in remote_service_list(server_name) or [] if s.get("tag") == service_tag), {}) services_config["_server_accounts"] = info.get("server_accounts") services_config["_client_config"] = info.get("client_config") + services_config["_anonymous_search"] = info.get("anonymous_search") params = remote_service_params(info, data) return RemoteService(ctx, service_tag, title_id, url, api_key, services_config, service_params=params) @@ -1099,6 +1101,7 @@ class RemoteService: self._server_device = False self._local_vaults_only = False self._server_accounts: Optional[dict] = services_config.get("_server_accounts") + self._anonymous_search = bool(services_config.get("_anonymous_search")) # read before apply_service_config merges the server's values into the same section own = config.services.get(service_tag) or {} self._client_config = {k: own[k] for k in services_config.get("_client_config") or [] if k in own} @@ -1144,6 +1147,43 @@ class RemoteService: def title(self) -> str: return self.title_id + def search(self) -> Iterator[SearchResult]: + """Search on the server, which logs in to the service itself with the login material sent here.""" + params = self.ctx.parent.params if self.ctx.parent else {} + profile = params.get("profile") + data: Dict[str, Any] = {"service": self.service_tag, "query": self.title_id} + if profile: + data["profile"] = profile + if not self._anonymous_search: + data.update(self.client_login(profile)) + if params.get("no_proxy"): + data["no_proxy"] = True + elif params.get("proxy"): + data["proxy"] = resolve_remote_proxy_arg(params["proxy"], getattr(self.ctx.obj, "proxy_providers", None)) + self.require_reachable_proxy(data["proxy"]) + response = self.client.post("/api/search", data) + self.save_returned_cache(response.get("cache") or {}, profile) + for result in response.get("results") or []: + yield SearchResult( + result["id"], result["title"], result.get("description"), result.get("label"), result.get("url") + ) + + def client_login(self, profile: Optional[str]) -> Dict[str, Any]: + """The credentials, cookies and cache to send, or nothing when the server manages the account.""" + if self._server_accounts is not None: + return {} + fields = { + "credentials": load_credentials_for_transport(self.service_tag, profile), + "cookies": load_cookies_for_transport(self.service_tag, profile), + "cache": self.load_cache_files(profile), + } + return {k: v for k, v in fields.items() if v} + + def require_reachable_proxy(self, proxy: Optional[str]) -> None: + """Refuse a proxy on this machine when the server is elsewhere: the server cannot reach it.""" + if proxy and is_loopback(proxy) and not is_loopback(self.client.server_url): + raise click.ClickException("That proxy is on your machine, so --remote cannot use it.") + def authenticate(self, cookies: Optional[CookieJar] = None, credential: Optional[Credential] = None) -> None: self.credential = credential profile = self.ctx.parent.params.get("profile") if self.ctx.parent else None @@ -1153,13 +1193,7 @@ class RemoteService: create_data: Dict[str, Any] = {"service": self.service_tag, "title_id": self.title_id} if self._server_accounts is None: - credentials = load_credentials_for_transport(self.service_tag, profile) - if credentials: - create_data["credentials"] = credentials - - cookies_text = load_cookies_for_transport(self.service_tag, profile) - if cookies_text: - create_data["cookies"] = cookies_text + create_data.update(self.client_login(profile)) else: self.credential = None self.log.info( @@ -1199,9 +1233,7 @@ class RemoteService: country = re.match(r"^(?:[a-z]+:)?([a-z]{2})\d*$", query, re.IGNORECASE) if country: create_data["proxy_region"] = country.group(1).lower() - if create_data.get("proxy") and is_loopback(create_data["proxy"]) and not is_loopback(self.client.server_url): - # the local CLI resolves the proxy, so a local-only URI is useless to a server elsewhere - raise click.ClickException("That proxy is on your machine, so --remote cannot use it.") + self.require_reachable_proxy(create_data.get("proxy")) local_proxy = None if no_proxy else (proxy or create_data.get("proxy")) if local_proxy: self._session.proxies["all"] = local_proxy @@ -1282,10 +1314,6 @@ class RemoteService: if self._client_config: create_data["service_config"] = self._client_config - cache_data = self.load_cache_files(profile) if self._server_accounts is None else None - if cache_data: - create_data["cache"] = cache_data - from unshackle.core import __code_hash__, __version__ secrets = [v for v in (create_data.get("credentials") or {}).values() if isinstance(v, str) and len(v) >= 4] diff --git a/unshackle/core/search_result.py b/unshackle/core/search_result.py index 6320e553..537b5d50 100644 --- a/unshackle/core/search_result.py +++ b/unshackle/core/search_result.py @@ -40,5 +40,15 @@ class SearchResult: self.label = label self.url = url + def to_dict(self) -> dict[str, Union[str, int, None]]: + """The result as the REST API returns it.""" + return { + "id": self.id, + "title": self.title, + "description": self.description, + "label": self.label, + "url": self.url, + } + __all__ = ("SearchResult",)