Files
unshackle/tests/remote/unit/test_download_gates_redaction.py
AviDev 1a3cd09fc8 fix(api): repair REST API downloads, add /services flags & live progress (#113)
* feat(api): live download phase, granular progress, swallowed-failure detection, per-request CDM

- Tee Track.download progress so the job reports real percentage (not just 5/100%) and a
  human-readable phase ('downloading video 1080p') via the new job.phase field.
- Detect a swallowed download-worker failure (dl.result() prints 'Download Failed' but
  exits 0) and raise, so the job is marked failed instead of completed-with-no-output.
- Per-request CDM override (dl_instance.cdm_override; get_cdm prefers it) so a job can use
  a specific CDM device without mutating shared config.

* feat(api): expose service capability flags + auth methods in /services

needs_auth / has_search / has_drm derived from which Service hooks are overridden, and
auth_methods inferred from what the service's authenticate() body references (cookies /
credentials), so clients can show only the relevant auth options per service.

* feat(api): per-request credential injection for downloads

Accept a 'credential' ('user:pass') job parameter and feed it into the credentials map that
dl.get_credentials() reads, so a client can authenticate a download without persisting
anything to disk. (Kept on the deployment branch; the PR branch uses the client-sent path.)

* feat(api): gate per-request CDM override behind serve.cdm_overrides

A per-request `cdm` selects a server-side device, so honour it only when allow-listed.
`serve.cdm_overrides` opts in: a list permits those device names, or `true` permits any
(single trusted client). Unset/false rejects every override with 403, so an arbitrary device
can't be selected by default.

* fix(api): redact credentials and proxy userinfo in serialized job parameters

Job parameters can carry a raw user:pass credential and a proxy URL with embedded userinfo;
mask them wherever parameters are serialized for an API response so secrets don't leak via
the job-detail endpoint. Also read skipped-subtitle / download-failure state from the dl
instance instead of scraping stdout, and drop the dead n_m3u8dl percent branch.

* feat(api): prefer explicit AUTH_METHODS class var over source inference in /services

Inferring auth methods from authenticate() source mostly returns both options because services
call super().authenticate(cookies, credential). Prefer an explicit AUTH_METHODS class var when a
service declares one, falling back to inference.

* style: use plain hyphens instead of em-dashes in comments

* feat(api): gate per-job credentials, isolate caches, scrub error fields

Address review feedback on #113:
- Gate per-request credential/credentials behind serve.allow_job_credentials (default off,
  403 when not opted in), mirroring the existing serve.cdm_overrides CDM gate.
- Isolate the token cache per credential: when a per-job credential is set, namespace
  config.directories.cache by its hash in the worker, so two clients on the same service
  with different credentials can't share a cached session.
- Scrub the credential, its password half, and proxy userinfo out of error_message,
  error_details, error_traceback and worker_stderr before they leave via the job-detail API.
- Remove the unused _execute_download_sync in-process path (would have leaked one job's
  credential into the shared global config).
- Document serve.cdm_overrides and serve.allow_job_credentials in the example config.
- Add tests for both gates (403 default, allowlist pass) and the parameter/error redaction.

* fix(dl): flag download_failed when result() swallows a worker failure

dl.result() catches a download-worker exception, reports it, and returns
without re-raising so the CLI still exits cleanly. An embedding caller (the
API worker) had no way to tell the title actually failed and would report
the job completed with no output. Expose a download_failed flag, set in the
swallow path, that the worker reads after result() returns.

* feat(api): surface skipped subtitles and pass skip_subtitle_errors

Thread skip_subtitle_errors from the job into dl() so the API can opt into
non-fatal subtitle handling, and report which subtitles were skipped: store
them on the job (dl.SkippedSubtitle dicts) and include them in job details
so a client can tell the user which weren't available.

---------

Co-authored-by: Avi Cohen <avraham.coh770@gmail.com>
2026-06-08 11:38:08 -06:00

143 lines
5.2 KiB
Python

"""Unit tests for the /api/download security gates (per-request CDM + credential overrides)
and the secret redaction applied to job parameters and error/stderr fields."""
from __future__ import annotations
from datetime import datetime
import pytest
from aiohttp import web
from unshackle.core.api import handlers
from unshackle.core.api.download_manager import (
DownloadJob,
JobStatus,
_redact_parameters,
_redact_text,
_secret_values,
)
from unshackle.core.api.errors import APIError, APIErrorCode
pytestmark = pytest.mark.unit
# ---------- redaction ----------
def test_redact_parameters_masks_secrets_and_proxy_userinfo():
params = {
"service": "ATV",
"credential": "user:hunter2",
"password": "pw",
"token": "tok",
"api_key": "ak",
"proxy": "http://bob:secret@proxy.example:8080",
"quality": "1080p",
}
red = _redact_parameters(params)
assert red["credential"] == "***"
assert red["password"] == "***"
assert red["token"] == "***"
assert red["api_key"] == "***"
assert red["proxy"] == "http://***@proxy.example:8080"
assert red["quality"] == "1080p" # non-secret left intact
assert params["credential"] == "user:hunter2" # original dict not mutated
def test_redact_parameters_masks_credentials_dict():
assert _redact_parameters({"credentials": {"default": "u:p"}})["credentials"] == "***"
def test_secret_values_includes_password_half_and_dict_values():
secrets = _secret_values({"credential": "user:hunter2", "credentials": {"d": "alice:wonder"}})
assert "user:hunter2" in secrets # full credential
assert "hunter2" in secrets # password half of user:pass
assert "alice:wonder" in secrets # value from the credentials map
def test_redact_text_scrubs_credential_and_proxy_from_free_text():
params = {"credential": "user:hunter2", "proxy": "http://bob:secret@p:1"}
out = _redact_text("auth failed for user:hunter2 via http://bob:secret@p:1", params)
assert "hunter2" not in out
assert "bob:secret@" not in out
assert "***" in out
def test_redact_text_passthrough_without_secrets():
assert _redact_text("plain error", {}) == "plain error"
assert _redact_text(None, {}) is None
def test_to_dict_full_details_redacts_error_fields_and_parameters():
job = DownloadJob(
job_id="j1",
status=JobStatus.FAILED,
created_time=datetime(2026, 1, 1),
service="ATV",
title_id="t",
parameters={"credential": "user:hunter2"},
)
job.error_message = "login failed for user:hunter2"
job.worker_stderr = "Traceback ... user:hunter2 ..."
d = job.to_dict(include_full_details=True)
assert "hunter2" not in d["error_message"]
assert "hunter2" not in d["worker_stderr"]
assert d["parameters"]["credential"] == "***"
# ---------- gates ----------
class _PastGate(Exception):
"""Raised by the stubbed Services.load to prove a request got past the gate into the try block."""
@pytest.fixture
def stub_handler(monkeypatch):
"""Make the service valid and make the first call after the gate (Services.load) explode, so a
forbidden request raises APIError *before* the try block and an allowed one is caught inside it."""
monkeypatch.setattr(handlers, "validate_service", lambda tag, request=None: tag)
def _boom(*_args, **_kwargs):
raise _PastGate()
monkeypatch.setattr(handlers.Services, "load", _boom)
return monkeypatch
async def test_cdm_override_forbidden_by_default(stub_handler):
stub_handler.setattr(handlers.config, "serve", {})
with pytest.raises(APIError) as ei:
await handlers.download_handler({"service": "ATV", "title_id": "t", "cdm": "dev"})
assert ei.value.error_code == APIErrorCode.FORBIDDEN
async def test_cdm_override_allowed_when_enabled(stub_handler):
stub_handler.setattr(handlers.config, "serve", {"cdm_overrides": True})
# passing the gate reaches the stubbed Services.load, whose error is caught and returned as a response
resp = await handlers.download_handler({"service": "ATV", "title_id": "t", "cdm": "dev"})
assert isinstance(resp, web.Response)
async def test_cdm_override_allowlist_permits_only_named_device(stub_handler):
stub_handler.setattr(handlers.config, "serve", {"cdm_overrides": ["good"]})
assert isinstance(
await handlers.download_handler({"service": "ATV", "title_id": "t", "cdm": "good"}), web.Response
)
with pytest.raises(APIError) as ei:
await handlers.download_handler({"service": "ATV", "title_id": "t", "cdm": "other"})
assert ei.value.error_code == APIErrorCode.FORBIDDEN
async def test_credential_forbidden_by_default(stub_handler):
stub_handler.setattr(handlers.config, "serve", {})
with pytest.raises(APIError) as ei:
await handlers.download_handler({"service": "ATV", "title_id": "t", "credential": "u:p"})
assert ei.value.error_code == APIErrorCode.FORBIDDEN
async def test_credential_allowed_when_enabled(stub_handler):
stub_handler.setattr(handlers.config, "serve", {"allow_job_credentials": True})
resp = await handlers.download_handler({"service": "ATV", "title_id": "t", "credential": "u:p"})
assert isinstance(resp, web.Response)