feat(search): add --remote to search through a remote server

The client sends its credentials, cookies and cache, but none for an anonymous-search service. The API relay uses the same path.

docs/dev/rest-api/endpoints.md

@@ -17,7 +17,7 @@ Start the server with [`unshackle serve`](../../guide/cli-reference.md). By defa
 - **CORS.** Every response carries permissive CORS headers, and the server answers `OPTIONS` preflight requests automatically.
 - **Compression.** JSON responses of 256 bytes or more are gzip-compressed when the client sends `Accept-Encoding: gzip`. A [slow response](#slow-responses) is not compressed.
 - **Service allowlist.** The effective allowlist for your API key filters the service-facing endpoints (the intersection of the global `serve.services` list and your per-key list). The server treats services you cannot use as unknown.
-- **Remote services.** A full-mode server can run a service through one of its own `remote_services` servers (see [Remote sessions](remote-sessions.md)). `GET /api/config` lists their names in `remote_services`, and `GET /api/services?remote=<name>` lists that server's services, each with `remote` set to the name. Add `"remote": "<name>"` to `POST /api/search`, `/api/list-titles`, `/api/list-tracks` or `/api/download`, and the server opens a remote session with its own key for that server: the remote logs in and resolves titles, tracks and licences, and a download still runs, decrypts and muxes here. The tag may be any case or an alias, and your allowlist applies to it. An unknown name is `400 INVALID_PARAMETERS`, a server that does not answer is `SERVICE_UNAVAILABLE`, and a tag it does not offer is `INVALID_SERVICE`. List calls for the same title, profile, proxy and service options share one remote session, which closes after 120 seconds unused. An optional `client_name` names your client on the remote server's dashboard as `serve (<client_name>)`. A `--remote-only` server does not relay. A remote login that waits on you (a pair code or URL to approve on another device) cannot answer through a list call, so poll `GET /api/remote/logins` alongside it: each entry carries the login's `notice`, and its `prompt` when it asks a question (an OTP), which you answer with `POST /api/remote/logins/{id}/input` and `{"response": "..."}`. A remote download job shows the same text as its own `notice`, and asks a question (an OTP) through `input_prompt` as a local job does.
+- **Remote services.** A full-mode server can run a service through one of its own `remote_services` servers (see [Remote sessions](remote-sessions.md)). `GET /api/config` lists their names in `remote_services`, and `GET /api/services?remote=<name>` lists that server's services, each with `remote` set to the name. Add `"remote": "<name>"` to `POST /api/search`, `/api/list-titles`, `/api/list-tracks` or `/api/download`, and the server opens a remote session with its own key for that server: the remote logs in and resolves titles, tracks and licences, and a download still runs, decrypts and muxes here. The tag may be any case or an alias, and your allowlist applies to it. An unknown name is `400 INVALID_PARAMETERS`, a server that does not answer is `SERVICE_UNAVAILABLE`, and a tag it does not offer is `INVALID_SERVICE`. List calls for the same title, profile, proxy and service options share one remote session, which closes after 120 seconds unused. An optional `client_name` names your client on the remote server's dashboard as `serve (<client_name>)`. A relayed search opens no remote session. The server sends its own credentials, cookies and cache for the `profile` with the search. A service that searches with no login gets none of them. A `--remote-only` server does not relay. A remote login that waits on you (a pair code or URL to approve on another device) cannot answer through a list call, so poll `GET /api/remote/logins` alongside it: each entry carries the login's `notice`, and its `prompt` when it asks a question (an OTP), which you answer with `POST /api/remote/logins/{id}/input` and `{"response": "..."}`. A remote download job shows the same text as its own `notice`, and asks a question (an OTP) through `input_prompt` as a local job does.
 
 !!! warning "Error responses have two shapes"
     Structured application errors (raised inside a handler) return `{"status": "error", "error_code": "...", "message": "...", "timestamp": "..."}`. The authentication middleware, however, returns a **different** shape: `{"status": 401, "message": "..."}` with an integer `status` and no `error_code`. The same middleware answers `429` in that shape when an API key goes over its rate limit. Clients must accept both shapes. The [Error responses](#error-responses) section at the bottom of this page gives the full error model.

docs/guide/cli-reference.md

@@ -250,12 +250,15 @@ unshackle logs in to the service before the search. A service that declares a se
 | `-p`, `--profile` | Profile for credentials and cookies. |
 | `--proxy` | Proxy URI, country or location code, or `provider:region` (e.g. `controld:ca`). |
 | `--no-proxy` | Force-disable all proxy use. |
+| `--remote` | Search on a remote unshackle server. The server logs in to the service. unshackle sends it your credentials, cookies and cache for the profile, and saves the cache that the server sends back. A service that searches with no login gets none of them. |
+| `--server` | Name a remote server from the `remote_services` config. |
 
 !!! example
     ```shell
     unshackle search EXAMPLE "My Show"
     unshackle search -p myprofile EXAMPLE "Another Show"
     unshackle search --proxy nordvpn:ca SERVICE "query"
+    unshackle search --remote --server my_server SERVICE "query"
     ```
 
 ---

tests/remote/unit/test_remote_search.py

@@ -0,0 +1,71 @@
+"""RemoteService.search: the body ``search --remote`` posts, and the results it yields."""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from typing import Any
+
+import pytest
+
+from unshackle.core import remote_service
+from unshackle.core.remote_service import RemoteService
+
+pytestmark = pytest.mark.unit
+
+
+class FakeClient:
+    server_url = "https://server.example"
+
+    def __init__(self) -> None:
+        self.sent: dict[str, Any] = {}
+
+    def post(self, endpoint: str, data: dict[str, Any]) -> dict[str, Any]:
+        self.sent = {"endpoint": endpoint, **data}
+        return {"results": [{"id": "t1", "title": "Show", "label": "2019"}], "cache": {"token": "x"}}
+
+
+def make_service(params: dict[str, Any], server_accounts: Any = None) -> tuple[RemoteService, FakeClient, list]:
+    svc = RemoteService.__new__(RemoteService)
+    client = FakeClient()
+    saved: list = []
+    svc.service_tag = "TEST"
+    svc.title_id = "query"
+    svc.client = client  # type: ignore[assignment]
+    svc.ctx = SimpleNamespace(parent=SimpleNamespace(params=params), obj=None)  # type: ignore[assignment]
+    svc._server_accounts = server_accounts
+    svc._anonymous_search = False
+    svc.load_cache_files = lambda profile=None: {"token": "old"}  # type: ignore[method-assign]
+    svc.save_returned_cache = lambda data, profile=None: saved.append(data)  # type: ignore[method-assign]
+    return svc, client, saved
+
+
+def test_search_sends_client_login_and_yields_results(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setattr(remote_service, "load_credentials_for_transport", lambda tag, profile: {"username": "u"})
+    monkeypatch.setattr(remote_service, "load_cookies_for_transport", lambda tag, profile: None)
+    svc, client, saved = make_service({"profile": "p1", "no_proxy": True})
+
+    results = list(svc.search())
+
+    assert client.sent == {
+        "endpoint": "/api/search",
+        "service": "TEST",
+        "query": "query",
+        "profile": "p1",
+        "credentials": {"username": "u"},
+        "cache": {"token": "old"},
+        "no_proxy": True,
+    }
+    assert [(r.id, r.title, r.label) for r in results] == [("t1", "Show", "2019")]
+    assert saved == [{"token": "x"}]
+
+
+@pytest.mark.parametrize(
+    ("server_accounts", "anonymous"), [({"global": True}, False), (None, True)], ids=["server-account", "anonymous"]
+)
+def test_search_sends_no_login(server_accounts: Any, anonymous: bool) -> None:
+    svc, client, _ = make_service({}, server_accounts)
+    svc._anonymous_search = anonymous
+
+    list(svc.search())
+
+    assert client.sent == {"endpoint": "/api/search", "service": "TEST", "query": "query"}

unshackle/commands/search.py

@@ -39,8 +39,29 @@ from unshackle.core.utils.redact import mask_proxy
     help="Proxy URI to use. If a 2-letter country is provided, it will try get a proxy from the config.",
 )
 @click.option("--no-proxy", is_flag=True, default=False, help="Force disable all proxy use.")
+@click.option(
+    "--remote",
+    is_flag=True,
+    default=False,
+    is_eager=True,
+    help="Search on a remote unshackle server instead of with local service code.",
+)
+@click.option(
+    "--server",
+    type=str,
+    default=None,
+    is_eager=True,
+    help="Name of the remote server from remote_services config (if multiple configured).",
+)
 @click.pass_context
-def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, proxy: Optional[str] = None):
+def search(
+    ctx: click.Context,
+    no_proxy: bool,
+    profile: Optional[str] = None,
+    proxy: Optional[str] = None,
+    remote: bool = False,
+    server: Optional[str] = None,
+):
     if not ctx.invoked_subcommand:
         raise ValueError("A subcommand to invoke was not specified, the main code cannot continue.")
 
@@ -52,14 +73,14 @@ def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, pr
     if profile:
         log.info(f"Using profile: '{profile}'")
 
-    with console.status("Loading Service Config...", spinner="dots"):
+    service_config: dict = {}
-        service_config_path = Services.get_path(service) / config.filenames.config
+    if not remote:
-        if service_config_path.exists():
+        with console.status("Loading Service Config...", spinner="dots"):
-            service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8"))
+            service_config_path = Services.get_path(service) / config.filenames.config
-            log.info("Service Config loaded")
+            if service_config_path.exists():
-        else:
+                service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8"))
-            service_config = {}
+                log.info("Service Config loaded")
-        merge_dict(config.services.get(service), service_config)
+    merge_dict(config.services.get(service), service_config)
 
     proxy_providers: list[Proxy] = []
     if no_proxy:
@@ -91,12 +112,12 @@ def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, pr
 
 
 @search.result_callback()
-def result(service: Service, profile: Optional[str] = None, **_: Any) -> None:
+def result(service: Service, profile: Optional[str] = None, remote: bool = False, **_: Any) -> None:
     log = logging.getLogger("search")
 
     service_tag = service.__class__.__name__
 
-    if not service.ANONYMOUS_SEARCH:
+    if not remote and not service.ANONYMOUS_SEARCH:
         with console.status("Authenticating with Service...", spinner="dots"):
             cookies = dl.get_cookie_jar(service_tag, profile)
             credential = dl.get_credentials(service_tag, profile)
@@ -117,7 +138,7 @@ def result(service: Service, profile: Optional[str] = None, **_: Any) -> None:
             result_text += f"\n[bright_black]id: {result.id}[/]"
             search_results.add(result_text + "\n")
 
-    cookie_file = None if service.ANONYMOUS_SEARCH else dl.get_cookie_path(service_tag, profile)
+    cookie_file = None if remote or service.ANONYMOUS_SEARCH else dl.get_cookie_path(service_tag, profile)
     if cookie_file:
         dl.save_cookies(cookie_file, service.session.cookies)
 

unshackle/core/api/handlers.py

@@ -481,11 +481,11 @@ def relay_error(exc: Any, service: str) -> APIError:
     return APIError(APIErrorCode.SERVICE_ERROR, exc.message, details={"service": service, "remote_code": exc.kind})
 
 
-def setup_remote_service(
+def relay_service(
     data: Dict[str, Any], server: str, profile: Optional[str], title_id: str, request: Optional[web.Request] = None
 ) -> Any:
-    """Open and authenticate a remote session and fetch its titles; the caller must close() it."""
+    """A RemoteService for ``data["service"]`` on a ``remote_services`` server, with this server's login files."""
-    from unshackle.core.remote_service import RemoteError, make_remote_service
+    from unshackle.core.remote_service import make_remote_service
 
     # the same server_proxy gate a local service gets: without it the providers stay empty
     proxy_param, providers = resolve_handler_proxy(data, data["service"], request)
@@ -500,7 +500,16 @@ def setup_remote_service(
     )
     ctx = click.Context(click.Command(data["service"]), parent=parent_ctx, obj=parent_ctx.obj)
     # this process is long-lived and serves local services too, so the remote's overrides stay out of config
-    service_instance = make_remote_service(ctx, server, data["service"], title_id, data, apply_overrides=False)
+    return make_remote_service(ctx, server, data["service"], title_id, data, apply_overrides=False)
+
+
+def setup_remote_service(
+    data: Dict[str, Any], server: str, profile: Optional[str], title_id: str, request: Optional[web.Request] = None
+) -> Any:
+    """Open and authenticate a remote session and fetch its titles; the caller must close() it."""
+    from unshackle.core.remote_service import RemoteError
+
+    service_instance = relay_service(data, server, profile, title_id, request)
     login = RemoteLogin(caller_key(request), server, data["service"], title_id, service_instance)
     service_instance._input_bridge = login.bridge
     _remote_logins[login.id] = login
@@ -725,14 +734,21 @@ def remote_services_info(name: str, allowed: Optional[List[str]]) -> List[Dict[s
     return [{**svc, "remote": name} for svc in remote_list(name) if allowed is None or svc.get("tag") in allowed]
 
 
-def remote_search(server: str, service: str, query: str) -> Dict[str, Any]:
+def remote_search(
-    """Run a search on a ``remote_services`` server with this server's key for it."""
+    data: Dict[str, Any], server: str, query: str, request: Optional[web.Request] = None
-    from unshackle.core.remote_service import RemoteError, remote_client
+) -> Dict[str, Any]:
+    """Run a search on a ``remote_services`` server, sending this server's login files as ``search --remote`` does."""
+    from unshackle.core.remote_service import RemoteError
 
+    service_instance = relay_service(data, server, client_profile(data), query, request)
     try:
-        return remote_client(server).post("/api/search", {"service": service, "query": query})
+        results = [result.to_dict() for result in service_instance.search()]
     except RemoteError as exc:
-        raise relay_error(exc, service) from None
+        raise relay_error(exc, data["service"]) from None
+    finally:
+        service_instance.session.close()
+        service_instance.client.session.close()
+    return {"results": results, "count": len(results)}
 
 
 def setup_list_service(
@@ -851,15 +867,7 @@ def run_service_search(
     results: List[Dict[str, Any]] = []
     try:
         for result in service_instance.search():
-            results.append(
+            results.append(result.to_dict())
-                {
-                    "id": result.id,
-                    "title": result.title,
-                    "description": result.description,
-                    "label": result.label,
-                    "url": result.url,
-                }
-            )
     except NotImplementedError:
         raise APIError(
             APIErrorCode.SERVICE_ERROR,
@@ -1861,7 +1869,8 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] =
 
     remote = await asyncio.to_thread(remote_server, data, request) if data.get("remote") else None
     if remote:
-        return web.json_response(await asyncio.to_thread(remote_search, remote[0], remote[1]["tag"], query))
+        data = {**data, "service": remote[1]["tag"]}
+        return web.json_response(await asyncio.to_thread(remote_search, data, remote[0], query, request))
 
     # get_tag echoes an unknown tag back, so it can never be falsy; validate_service
     # is the check that actually resolves the service directory and the allowlist.

unshackle/core/remote_service.py

@@ -41,6 +41,7 @@ from unshackle.core.constants import AnyTrack
 from unshackle.core.credential import Credential
 from unshackle.core.drm import cbcs, real_kids
 from unshackle.core.proxies.resolve import is_loopback
+from unshackle.core.search_result import SearchResult
 from unshackle.core.session import MAX_BACKOFF
 from unshackle.core.titles import Title_T, Titles_T, remap_titles
 from unshackle.core.titles.episode import Episode, Series
@@ -852,6 +853,7 @@ def make_remote_service(
     info = next((s for s in remote_service_list(server_name) or [] if s.get("tag") == service_tag), {})
     services_config["_server_accounts"] = info.get("server_accounts")
     services_config["_client_config"] = info.get("client_config")
+    services_config["_anonymous_search"] = info.get("anonymous_search")
     params = remote_service_params(info, data)
     return RemoteService(ctx, service_tag, title_id, url, api_key, services_config, service_params=params)
 
@@ -1099,6 +1101,7 @@ class RemoteService:
         self._server_device = False
         self._local_vaults_only = False
         self._server_accounts: Optional[dict] = services_config.get("_server_accounts")
+        self._anonymous_search = bool(services_config.get("_anonymous_search"))
         # read before apply_service_config merges the server's values into the same section
         own = config.services.get(service_tag) or {}
         self._client_config = {k: own[k] for k in services_config.get("_client_config") or [] if k in own}
@@ -1144,6 +1147,43 @@ class RemoteService:
     def title(self) -> str:
         return self.title_id
 
+    def search(self) -> Iterator[SearchResult]:
+        """Search on the server, which logs in to the service itself with the login material sent here."""
+        params = self.ctx.parent.params if self.ctx.parent else {}
+        profile = params.get("profile")
+        data: Dict[str, Any] = {"service": self.service_tag, "query": self.title_id}
+        if profile:
+            data["profile"] = profile
+        if not self._anonymous_search:
+            data.update(self.client_login(profile))
+        if params.get("no_proxy"):
+            data["no_proxy"] = True
+        elif params.get("proxy"):
+            data["proxy"] = resolve_remote_proxy_arg(params["proxy"], getattr(self.ctx.obj, "proxy_providers", None))
+            self.require_reachable_proxy(data["proxy"])
+        response = self.client.post("/api/search", data)
+        self.save_returned_cache(response.get("cache") or {}, profile)
+        for result in response.get("results") or []:
+            yield SearchResult(
+                result["id"], result["title"], result.get("description"), result.get("label"), result.get("url")
+            )
+
+    def client_login(self, profile: Optional[str]) -> Dict[str, Any]:
+        """The credentials, cookies and cache to send, or nothing when the server manages the account."""
+        if self._server_accounts is not None:
+            return {}
+        fields = {
+            "credentials": load_credentials_for_transport(self.service_tag, profile),
+            "cookies": load_cookies_for_transport(self.service_tag, profile),
+            "cache": self.load_cache_files(profile),
+        }
+        return {k: v for k, v in fields.items() if v}
+
+    def require_reachable_proxy(self, proxy: Optional[str]) -> None:
+        """Refuse a proxy on this machine when the server is elsewhere: the server cannot reach it."""
+        if proxy and is_loopback(proxy) and not is_loopback(self.client.server_url):
+            raise click.ClickException("That proxy is on your machine, so --remote cannot use it.")
+
     def authenticate(self, cookies: Optional[CookieJar] = None, credential: Optional[Credential] = None) -> None:
         self.credential = credential
         profile = self.ctx.parent.params.get("profile") if self.ctx.parent else None
@@ -1153,13 +1193,7 @@ class RemoteService:
         create_data: Dict[str, Any] = {"service": self.service_tag, "title_id": self.title_id}
 
         if self._server_accounts is None:
-            credentials = load_credentials_for_transport(self.service_tag, profile)
+            create_data.update(self.client_login(profile))
-            if credentials:
-                create_data["credentials"] = credentials
-
-            cookies_text = load_cookies_for_transport(self.service_tag, profile)
-            if cookies_text:
-                create_data["cookies"] = cookies_text
         else:
             self.credential = None
             self.log.info(
@@ -1199,9 +1233,7 @@ class RemoteService:
                 country = re.match(r"^(?:[a-z]+:)?([a-z]{2})\d*$", query, re.IGNORECASE)
                 if country:
                     create_data["proxy_region"] = country.group(1).lower()
-        if create_data.get("proxy") and is_loopback(create_data["proxy"]) and not is_loopback(self.client.server_url):
+        self.require_reachable_proxy(create_data.get("proxy"))
-            # the local CLI resolves the proxy, so a local-only URI is useless to a server elsewhere
-            raise click.ClickException("That proxy is on your machine, so --remote cannot use it.")
         local_proxy = None if no_proxy else (proxy or create_data.get("proxy"))
         if local_proxy:
             self._session.proxies["all"] = local_proxy
@@ -1282,10 +1314,6 @@ class RemoteService:
             if self._client_config:
                 create_data["service_config"] = self._client_config
 
-        cache_data = self.load_cache_files(profile) if self._server_accounts is None else None
-        if cache_data:
-            create_data["cache"] = cache_data
-
         from unshackle.core import __code_hash__, __version__
 
         secrets = [v for v in (create_data.get("credentials") or {}).values() if isinstance(v, str) and len(v) >= 4]

unshackle/core/search_result.py

@@ -40,5 +40,15 @@ class SearchResult:
         self.label = label
         self.url = url
 
+    def to_dict(self) -> dict[str, Union[str, int, None]]:
+        """The result as the REST API returns it."""
+        return {
+            "id": self.id,
+            "title": self.title,
+            "description": self.description,
+            "label": self.label,
+            "url": self.url,
+        }
+
 
 __all__ = ("SearchResult",)