Compare commits

...
5 Commits
Author SHA1 Message Date
imSp4rky 0eaefcaa2b fix(proxy): name the reason when the Control D forwarder answers 502 2026-10-21 18:35:26 -06:00
imSp4rky a7e68b7d20 fix(tracks): keep a locked segment file from hiding the download error
On Windows a worker can still hold a .!dev file open when a track fails; cleanup now ignores that OSError and still stashes the segments.
2026-10-21 18:25:50 -06:00
imSp4rky 5086dafff3 feat(api)!: add a server drain and never answer with a 5xx
POST/DELETE /api/dashboard/drain refuses new sessions and jobs (SERVER_DRAINING, 409) and warns open sessions. A CDN can replace a 5xx body, so server errors are now 422, retryable ones and the tunnel check 424, and a new middleware turns any 5xx into 422.

BREAKING CHANGE: INTERNAL_ERROR, DOWNLOAD_ERROR, WORKER_ERROR and the fallback return 422 instead of 500; NETWORK_ERROR and SERVICE_UNAVAILABLE return 424 instead of 503; /api/tunnel/check failures return 424 instead of 502.
2026-10-21 17:15:33 -06:00
imSp4rky 23614fd938 feat(remote): add --local-proxy to send server requests through the client
A reverse tunnel over the remote connection: the server's service traffic leaves through the client's own network. Only the client dials. Adds the tunnel-test command.
2026-10-21 17:09:37 -06:00
imSp4rky afaff8e070 feat(api): tag each error with a ref and count errors by service
Every structured error carries a short ref, appended to its message and logged with the request; /api/dashboard/status gains per-service error counts. Gzip now compresses in place.
2026-10-21 14:39:37 -06:00
33 changed files with 1775 additions and 153 deletions
+9 -2
View File
@@ -39,11 +39,13 @@ unshackle serve --remote-only --quiet
## Endpoints
All are `GET`.
All are `GET`, except the two `drain` routes.
| Path | Returns |
|------|---------|
| `/api/dashboard/status` | Version, `code_hash`, bind host and port, `mode` (`full`, `api_only`, `remote_only`), uptime, `requests_total`, `requests_rejected` (401s), `requests_by_key` (label → count), configured service count (the same services as `/api/dashboard/services`), session count, `max_sessions` (`null` when the operator configured no limit), `session_ttl`, job counts by status. The service count includes services that failed to import - use `/api/dashboard/services` for the number that loaded. |
| `/api/dashboard/status` | Version, `code_hash`, bind host and port, `mode` (`full`, `api_only`, `remote_only`), uptime, `requests_total`, `requests_rejected` (401s), `requests_by_key` (label → count), `errors` (error counts by service tag, then by `error_code`), configured service count (the same services as `/api/dashboard/services`), session count, `max_sessions` (`null` when the operator configured no limit), `session_ttl`, job counts by status. `draining` is `true` while a drain is on. The service count includes services that failed to import - use `/api/dashboard/services` for the number that loaded. |
| `POST /api/dashboard/drain` | Starts a drain before a restart and returns the `/status` body. The server then refuses each new remote session, download job and job retry with `SERVER_DRAINING` (409). Open remote sessions and jobs continue, and each open remote session gets a warning in its service log, which the client shows to the user. Poll `/status` until `sessions` and the queued and downloading jobs are 0, then restart. The drain ends when the process stops. A second call changes nothing. |
| `DELETE /api/dashboard/drain` | Cancels the drain and returns the `/status` body. The server accepts new work again. |
| `/api/dashboard/sessions` | Every live remote session: `id`, `owner` (username or masked key), `creator_ip`, `service`, `title_id`, `titles`, `tracks`, `title` (the display name of the title the client is on: the last one it asked tracks for, else the first resolved title), `auth_status`, `auth_error`, `server_account` (the server profile lent to the remote session, else `null`), `server_cdm` (`false` when the client's own device licenses the remote session), `server_cdm_max_height` (the tallest video the server CDM licenses live, else `null`), `client`, `actions`, `created_at` and `last_accessed` (ISO 8601) with `created_ts` and `last_accessed_ts` (Unix epoch, the same clock as log `ts`), `age_seconds`, `idle_seconds`, `log_seq`. `client` is whatever the remote client sent as `client` in its session create request. The CLI sends `version`, `code_hash` (the commit the client runs, `null` when its source cannot be read), `platform` and `argv`. `argv` is the command line the user ran: proxy and URL userinfo, secret query parameters and credential values become `***`, home and install paths shorten as in the logs, and the line is cut at 3000 characters. An older client sends less or nothing and the field is `{}`. `log_seq` is the last sequence number in the remote session's service log - poll `/api/dashboard/sessions/{id}/logs` when it changes. `actions` is the session's request log, newest last, capped at 500: `ts`, `method`, `action` (`titles`, `tracks`, `segments`, `license`, `prompt`, …), `query`, `status`, `ms`, `bytes_in`, `bytes_out`. |
| `/api/dashboard/jobs` | Every download job with full detail, regardless of owner. Empty in `--remote-only` mode. |
| `/api/dashboard/logs` | `{"seq": N, "records": [...]}`: the last 1000 log records. `?since=<seq>` returns only newer records; `?level=WARNING` sets the minimum level; `?logger=serve` keeps one logger and its children (`aiohttp.access` is the noisiest). `ts` is a Unix epoch in seconds. |
@@ -54,6 +56,11 @@ All are `GET`.
| `/api/dashboard/health` | Preflight: whether this instance could finish a download. Cached 30 seconds. |
| `/api/dashboard/events` | Server-Sent Events stream. With `?since=<seq>` it returns `{"seq", "stats", "events": [...]}` instead: every event after `seq` (the server keeps the last 20000, log records included, so a poll every few seconds is safe even under a busy access log) as one JSON burst, so a UI that polls every few seconds still sees a remote session that opened and closed in between. |
`errors` counts every structured error response since the server started, for example
`{"EXAMPLE": {"SERVICE_ERROR": 4}, "-": {"SESSION_NOT_FOUND": 1}}`. The service is the one of
the remote session, else the `service` field of the JSON body. `-` holds an error with no
installed service. Each error also logs its [`ref`](errors.md#find-an-error-from-its-ref).
`requests_rejected` counts 401 responses; those never appear in `requests_by_key`. The
label is the `username` from the `users` entry when one is set, else the first four
characters of the API key.
+3 -3
View File
@@ -131,7 +131,7 @@ Field notes:
| Status | Meaning |
| --- | --- |
| `200` | Service list returned. |
| `500` | Unexpected server error (`INTERNAL_ERROR`). |
| `422` | Unexpected server error (`INTERNAL_ERROR`). |
### `POST /api/search`
@@ -296,7 +296,7 @@ These keys are sent only when the title carries them, so a title without them se
| `400` | `INVALID_SERVICE` | Unknown or disallowed service. |
| `401` | `AUTH_FAILED` | Authentication with the service failed. |
| `404` | `NOT_FOUND` | Title not found. |
| `500` | `INTERNAL_ERROR` | Unexpected error. |
| `422` | `INTERNAL_ERROR` | Unexpected error. |
### `POST /api/list-tracks`
@@ -1290,7 +1290,7 @@ The server serialises every structured error raised inside a handler to the same
The [error code catalogue](errors.md#error-code-catalogue) gives the default HTTP status of each error code.
When a generic (uncategorized) exception reaches a handler, unshackle inspects its message and type to pick a code. For example, messages mentioning "connection" or "timeout" become `NETWORK_ERROR` (503, retryable), "rate limit" or "429" become `RATE_LIMITED` (429, retryable), and anything unrecognized collapses to `INTERNAL_ERROR` (500).
When a generic (uncategorized) exception reaches a handler, unshackle inspects its message and type to pick a code. For example, messages mentioning "connection" or "timeout" become `NETWORK_ERROR` (424, retryable), "rate limit" or "429" become `RATE_LIMITED` (429, retryable), and anything unrecognized collapses to `INTERNAL_ERROR` (422).
---
+34 -18
View File
@@ -25,7 +25,8 @@ A structured error is always a JSON object with `status` set to the literal stri
{
"status": "error",
"error_code": "INVALID_PARAMETERS",
"message": "vcodec must be one of: H264, H265, ...",
"message": "vcodec must be one of: H264, H265, ... (ref 7f3a2c)",
"ref": "7f3a2c",
"timestamp": "2026-07-03T12:34:56.789012+00:00",
"details": { "field": "vcodec" },
"retryable": true,
@@ -40,7 +41,8 @@ A structured error is always a JSON object with `status` set to the literal stri
|---|---|---|---|
| `status` | string | Yes | Always the literal `"error"`. Lets a client distinguish an error body from a success body at a glance. |
| `error_code` | string | Yes | A stable, machine-readable code from the [catalogue below](#error-code-catalogue). Match on this, not on `message`. |
| `message` | string | Yes | A human-readable description of what went wrong. Wording may change between versions, so do not parse it. |
| `message` | string | Yes | A human-readable description of what went wrong, followed by `(ref <ref>)`. Wording may change between versions, so do not parse it. |
| `ref` | string | Yes | A short random reference for this one error. The server logs it with the request, so an operator can find the record from a user's report. |
| `timestamp` | string | Yes | ISO 8601 timestamp (UTC) of when the error response was built. |
| `details` | object | Only when non-empty | Structured, error-specific context (for example the invalid field, a list of available titles, or the set of currently active jobs). |
| `retryable` | boolean | Only when `true` | Present and set to `true` when retrying the same request may succeed later (network blips, rate limiting, transient outages). Absent otherwise. |
@@ -52,6 +54,17 @@ A structured error is always a JSON object with `status` set to the literal stri
every code maps to a default status (see the catalogue), and the pairing is
stable across releases.
### Find an error from its `ref`
For each error response, the server logs one `WARNING` line on the `api` logger:
```text
Error ref 7f3a2c: SERVICE_ERROR 422 POST /api/session/<id>/license service=EXAMPLE user=alice: Licence request failed: ...
```
Search the server log for `Error ref <ref>`. The handler's traceback, when there is one,
is directly above that line. The [developer dashboard](dashboard.md) log view also shows it.
### The `details` object
When present, `details` carries extra context specific to the error. It is only
@@ -87,15 +100,16 @@ When it is absent, assume the request will keep failing until you change somethi
Every code below is a member of the `APIErrorCode` enumeration. The **Status** column
is the default HTTP status the server returns for that code. A handler can override
the status in special cases (for example, a network error is always forced to `503`),
but the defaults hold for the overwhelming majority of responses.
the status in special cases, but the defaults hold for the overwhelming majority of responses.
The server never answers with a `5xx` status. A proxy such as Cloudflare can replace the
body of a `5xx` with its own page, and the JSON error is then lost. Server-side failures
use `422`, and the ones that a retry can fix use `424`. Read `error_code` to tell them apart.
A `5xx` that a client receives comes from a proxy in front of the server, for example
while the server restarts.
### Client errors (4xx)
`SERVICE_ERROR` and `DRM_ERROR` are upstream failures, but they use `422`, not `502`.
A proxy such as Cloudflare replaces the body of a `502` or `504` with its own page, and
the JSON error is then lost.
| Code | Status | Meaning |
|---|---|---|
| `INVALID_INPUT` | 400 | A required field is missing, or the request body could not be parsed. Also used as the generic "your input was malformed" code. |
@@ -112,6 +126,7 @@ the JSON error is then lost.
| `SESSION_NOT_FOUND` | 404 | The remote-dl session does not exist or has expired. |
| `TRACK_NOT_FOUND` | 404 | A referenced track ID is not present in the remote session. |
| `CONFLICT` | 409 | The target resource is in a state that disallows the action (retrying a running job, prioritising a non-queued job, clearing cache during an active download). |
| `SERVER_DRAINING` | 409 | The server is about to restart and starts no new remote session or download job. Open work continues. Retryable after the restart. |
| `SERVICE_ERROR` | 422 | The streaming service's API returned an error, or the service does not support the requested operation. |
| `DRM_ERROR` | 422 | DRM or license acquisition failed. |
| `RATE_LIMITED` | 429 | The streaming service is rate-limiting requests. Retryable. |
@@ -125,19 +140,20 @@ the JSON error is then lost.
the window is fixed, so an early retry only spends the next window. See
[Authentication](authentication.md#rate-limit-rejections).
### Server errors (5xx)
### Server errors (422 and 424)
| Code | Status | Meaning |
|---|---|---|
| `INTERNAL_ERROR` | 500 | An unexpected server-side error. This is the fallback for any exception the server could not categorise. |
| `DOWNLOAD_ERROR` | 500 | The download process itself failed. |
| `WORKER_ERROR` | 500 | The download worker subprocess errored out. |
| `NETWORK_ERROR` | 503 | A network connectivity problem (connection, timeout, DNS, TLS). Retryable. |
| `SERVICE_UNAVAILABLE` | 503 | The service is temporarily unavailable or under maintenance. Retryable. |
| `INTERNAL_ERROR` | 422 | An unexpected server-side error. This is the fallback for any exception the server could not categorise, and for an exception that no handler caught. |
| `DOWNLOAD_ERROR` | 422 | The download process itself failed. |
| `WORKER_ERROR` | 422 | The download worker subprocess errored out. |
| `NETWORK_ERROR` | 424 | A network connectivity problem (connection, timeout, DNS, TLS). Retryable. |
| `SERVICE_UNAVAILABLE` | 424 | The service is temporarily unavailable or under maintenance. Retryable. |
!!! note "Fallback status"
If a code is ever emitted without a mapped status, the server falls back to
`500`. In practice every code above has an explicit mapping.
`422`. In practice every code above has an explicit mapping. A middleware
in every server mode also changes any `5xx` answer, such as one from a CDM subapplication, to `422`.
## How the server produces errors
@@ -156,14 +172,14 @@ The classification is keyword-based, checked in order:
| If the error mentions... | It becomes | Retryable |
|---|---|---|
| `auth`, `login`, `credential`, `unauthorized`, `forbidden`, `token` | `AUTH_FAILED` (401) | no |
| `connection`, `timeout`, `network`, `unreachable`, `socket`, `dns`, `resolve` (or `ConnectionError`, `TimeoutError`, `URLError`, `SSLError`) | `NETWORK_ERROR` (503) | yes |
| `connection`, `timeout`, `network`, `unreachable`, `socket`, `dns`, `resolve` (or `ConnectionError`, `TimeoutError`, `URLError`, `SSLError`) | `NETWORK_ERROR` (424) | yes |
| `geofence`, `geoblock`, `region`, `not available in`, `territory` | `GEOFENCE` (403) | no |
| `not found`, `404`, `does not exist`, `invalid id` | `NOT_FOUND` (404) | no |
| `rate limit`, `too many requests`, `429`, `throttle` | `RATE_LIMITED` (429) | yes |
| `drm`, `license`, `widevine`, `playready`, `decrypt` | `DRM_ERROR` (422) | no |
| `service unavailable`, `503`, `maintenance`, `temporarily unavailable` | `SERVICE_UNAVAILABLE` (503) | yes |
| `service unavailable`, `503`, `maintenance`, `temporarily unavailable` | `SERVICE_UNAVAILABLE` (424) | yes |
| `invalid`, `malformed`, `validation` (or `ValueError`, `ValidationError`) | `INVALID_INPUT` (400) | no |
| anything else | `INTERNAL_ERROR` (500) | no |
| anything else | `INTERNAL_ERROR` (422) | no |
Categorised errors include a `reason` tag inside `details` (such as
`"network_connectivity"` or `"drm_failure"`) so a client can react without parsing
+2 -2
View File
@@ -443,8 +443,8 @@ The `error_code` maps to an HTTP status. The ones you will meet most often:
| `CONFLICT` | 409 | Illegal state transition (e.g. retry a running job) |
| `SERVICE_ERROR` | 422 | The service raised an error |
| `RATE_LIMITED` | 429 | Upstream throttling (retryable) |
| `INTERNAL_ERROR` | 500 | Unexpected server-side failure |
| `NETWORK_ERROR` | 503 | Connection/timeout reaching the service (retryable) |
| `INTERNAL_ERROR` | 422 | Unexpected server-side failure |
| `NETWORK_ERROR` | 424 | Connection/timeout reaching the service (retryable) |
The [error code catalogue](errors.md#error-code-catalogue) lists every code.
+12 -1
View File
@@ -146,10 +146,14 @@ hood the client walks a remote session through its lifecycle.
`controld://<resolver>@dns.controld.com`. The server counts it as a full proxy
URI and runs its own forwarder for it. A bare region skips Control D, because its
proxy is a forwarder on the client. See
[Control D](../../guide/proxies-and-vpn.md#control-d)
[Control D](../../guide/proxies-and-vpn.md#control-d).
The client also downloads playlists and segments through the same proxy,
as a local run does. `--no-proxy-download` and `--proxy-download` change
that for downloads only.
- A `--local-proxy` goes as `tunnel:<id>`. The client opens
`GET /api/tunnel` first, and the server replaces the ID with a loopback SOCKS5
proxy that relays to the client's own connection. See
[Send the server's requests through your machine](../../guide/proxies-and-vpn.md#send-the-servers-requests-through-your-machine).
- Track-selection hints (`range_`, `vcodec`, `quality`, `best_available`,
`generate_dv`) so the server fetches the right manifests
- Your language and audio codec selection (`-l`, `-vl`, `-al`, `-a`, `-fs`)
@@ -400,6 +404,11 @@ serve:
gets a persistent per-account cache instead, so the account's tokens outlive the
remote session. The server and the client both refuse a forwarded cache file that is not
JSON or that names a type other than a date or time, and the service then authenticates again.
- **A tunnel belongs to its API key.** The server accepts `tunnel:<id>` only from the API key
that opened the tunnel. Its SOCKS5 proxy listens on loopback only, and it needs a username and
password for that tunnel, so a raw `socks5h://127.0.0.1:<port>` proxy from a client fails.
- **Only the client dials.** The server ignores a request from the client to open a
connection, so a tunnel never gives a client access to the server's network.
- **The server redacts secrets in logs.** It redacts remote session IDs, service tags,
and other user values before it writes them to the log.
@@ -427,6 +436,8 @@ The server mounts all these routes, even in `--remote-only` mode. Paths use the
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/api/session/create` | Build + authenticate a service instance; returns a session ID immediately |
| `GET` | `/api/tunnel` | WebSocket that relays the connections of a `tunnel:<id>` proxy to the client; `?id=` reconnects |
| `GET` | `/api/tunnel/check?id=` | Exit IP address and lookup time through the caller's tunnel, for `unshackle tunnel-test` |
| `GET` | `/api/session/{id}/titles` | Fetch and cache the title list |
| `POST` | `/api/session/{id}/tracks` | Resolve tracks, manifests, chapters, headers/cookies for a title |
| `POST` | `/api/session/{id}/segments` | Resolve per-segment/track download descriptors |
+20
View File
@@ -207,6 +207,7 @@ order the service used, then renumbers the episodes into the order you asked for
| Flag | Description |
|---|---|
| `--proxy` | Proxy URI, a country or location code resolved from configured providers, or `provider:region` (e.g. `nordvpn:ca`, `gluetun:us`, `protonvpn:de:berlin`, `controld:yul`). |
| `--local-proxy` | With `--remote`, send the server's service requests through your machine's own connection. Works with no other proxy option. Check it first with `unshackle tunnel-test`. See [Proxies & VPN](proxies-and-vpn.md#send-the-servers-requests-through-your-machine). |
| `--no-proxy` | Force-disable all proxy use. |
| `--no-proxy-download` | Bypass the proxy for **all downloads** (manifest, licence, and auth stay proxied). |
| `--proxy-download` | Use a different proxy for **all downloads**, in the same forms as `--proxy` (manifest, licence, and auth stay on `--proxy`). |
@@ -250,6 +251,7 @@ unshackle logs in to the service before the search. A service that declares a se
|---|---|
| `-p`, `--profile` | Profile for credentials and cookies. |
| `--proxy` | Proxy URI, country or location code, or `provider:region` (e.g. `controld:ca`). |
| `--local-proxy` | With `--remote`, send the server's search requests through your machine's own connection. Works with no other proxy option. |
| `--no-proxy` | Force-disable all proxy use. |
| `--remote` | Search on a remote unshackle server. The server logs in to the service. unshackle sends it your credentials, cookies and cache for the profile, and saves the cache that the server sends back. A service that searches with no login gets none of them. |
| `--local` | Search with the local service code, also when the config sets `remote: true` for the service. |
@@ -285,6 +287,24 @@ The export file can be a `mediaexport` file from `dl --export`, an export from a
---
## `tunnel-test`
Check `--local-proxy` before a real `--remote` run. The command finds the IP address that this
machine exits from, opens a tunnel to the server, and makes the server look up its exit IP address
through the tunnel. It stops at the first step that fails.
```
unshackle tunnel-test [--server NAME]
```
| Flag | Description |
|---|---|
| `--server` | Name of the remote server from `remote_services` config, if you configure more than one. |
See [Proxies & VPN](proxies-and-vpn.md#send-the-servers-requests-through-your-machine).
---
## `setup`
Install the external tools and make a first config.
+80 -3
View File
@@ -22,11 +22,12 @@ a query to a real proxy URL: [basic static proxies](#basic-static-proxies),
## The `--proxy` flag
Four options on the `dl` command control proxying at download time:
Five options on the `dl` command control proxying at download time:
| Option | Effect |
|---|---|
| `--proxy` | The proxy to use. Either an explicit URI, or a query that unshackle resolves against your configured providers. |
| `--local-proxy` | With `--remote`, send the server's service requests through your machine's own connection. Works with no other proxy option. See [Send the server's requests through your machine](#send-the-servers-requests-through-your-machine). |
| `--no-proxy` | Force **all** proxy use off. No providers are initialised and no proxy query is resolved. |
| `--no-proxy-download` | Use the proxy for the manifest, licence, and authentication, but bypass it for the **downloads** themselves. |
| `--proxy-download` | Use `--proxy` for the manifest, licence, and authentication, and a different proxy for the **downloads**. Takes the same forms as `--proxy`. |
@@ -204,6 +205,75 @@ sees your own IP, but the service does not.
A `socks5://` proxy makes the DNS lookup of each host on your machine and gives no warning.
Use `socks5h://` to make the proxy do the DNS lookups too.
### Send the server's requests through your machine
With `--remote`, the server sends the service requests, and they leave from the server's own
connection. Add `--local-proxy` to send them through your machine's connection instead:
```shell
unshackle dl --remote --local-proxy EXAMPLE 81234567
```
A VPN app that covers your whole machine then also covers the server's requests, so the
service sees your VPN's IP address. `--local-proxy` needs `--remote` (or a service with
`remote: true`), and it works with no other proxy option: unshackle stops if you also give
`--proxy`, `--no-proxy`, `--proxy-download` or `--no-proxy-download`. A proxy option from the `dl`
config defaults does not count: `--local-proxy` turns it off for the run.
A `--proxy` on `localhost` or a loopback address, such as the local port of a VPN app, stops a
`--remote` run with `That proxy is on your machine, so --remote cannot use it`. Connect the VPN
app for the whole machine, then use `--local-proxy` without `--proxy`.
#### Test it first
`unshackle tunnel-test` checks the route before a real run. Add `--server <name>` to test a
server other than the default one:
```shell
unshackle tunnel-test
```
It does three steps, and it stops at the first one that fails:
1. It looks up the IP address that your machine exits from.
2. It opens a tunnel to the server.
3. It makes the server look up its exit IP address through the tunnel. That address must be
the same as in step 1, or in the same country when one lookup used IPv4 and the other IPv6.
It fails when the lookup left from the server's own address. It also shows how long one
lookup takes.
#### How the tunnel works
1. The client opens a WebSocket on the server, on the same host and port as the other
remote session requests. Only a client opens a tunnel. The server never connects to
a host for a client, so a tunnel gives no access to the server's network.
2. The server gives the service a SOCKS5 proxy on its loopback address, with a username and
password for that tunnel only. Another API key cannot use it.
3. The server sends each connection that the service opens through the tunnel. The client
connects to the host and relays the bytes.
The client logs `The server sends its requests through your connection on this machine`. You
do not configure ports: the server picks a free one, and your
machine opens none, because the client opens the connection to the server. A NAT router, a
firewall, or a Cloudflare tunnel in front of the server does not stop it.
The service traffic stays encrypted from the server to the service, so the client relays only
encrypted bytes. The client refuses to connect to `localhost`, and to a host name or IP address
that resolves only to a loopback, private or link-local address, so the server cannot reach
your local network.
!!! note "Each request crosses the tunnel twice"
A request from the server goes to your machine and then to the service, so it takes
longer than a request that the server sends directly. The segments do not go through the
tunnel: the client downloads them as before.
A service that uses a server account does not work with `--local-proxy`, because the server
logs in with its own account. If the tunnel drops, the client connects again, and a new request
from the server waits up to 10 seconds for it. A request that was in progress when the tunnel
dropped fails. The server closes a tunnel when its client has not connected for 60 seconds, and
the client then stops with `The tunnel to the server closed`. An API key can have 4 tunnels
open at once.
## Basic (static proxies)
The `Basic` proxy provider is pure static configuration, with no accounts and no network
@@ -392,11 +462,18 @@ it next needs them.
### Troubleshooting Control D
When the forwarder cannot connect to a host, it answers `502 Bad Gateway` and gives the reason in brackets.
Python-Requests shows it as `Tunnel connection failed: 502 Bad Gateway (<reason>)`. The forwarder also logs
each reason once as a warning, `Could not reach <host>:<port>: <reason>`, because some HTTP clients do not
show the reason.
| Symptom | Cause and fix |
|---|---|
| `Proxy check failed`, because the exit IP is your own IP. | The profile does not redirect. Make sure that the plan includes traffic redirection (*Full Control*, not *Some Control*). After a region change, wait for the TTL of the previous answer to end. |
| A `502 Bad Gateway` for one host. With `-d`, the log shows `Control D returned no A record for …`. | Control D gave no IPv4 address for that host. |
| A `502 Bad Gateway` for one host only. With `-d`, the log shows `Refused …, an address that Control D cannot redirect`. | The request names the host by IP address. The forwarder refuses it. See the note on IPv4 above. |
| `502 Bad Gateway (Control D returned no A record for …)` | Control D gave no IPv4 address for that host. |
| `502 Bad Gateway (an IP address skips Control D's DNS …)` | The request names the host by IP address. The forwarder refuses it. See the note on IPv4 above. |
| `502 Bad Gateway (Control D sent … to …, which did not answer: …)` | Control D gave an address for the host, but the connection to that address failed. This is usually a temporary fault on Control D's network. Try again, or use `--no-proxy-download` to download without the proxy. |
| `502 Bad Gateway (Control D DNS timed out)`, `(Control D DNS could not be reached …)` or `(Control D DNS answered HTTP …)` | unshackle got no answer from Control D's DNS at `dns.controld.com`. Check the network connection of this machine. |
| An HTTP `401` or `403` from `api.controld.com`. | The token is a *Read* token, or its *Allowed IPs* do not include this machine. Use a *Write* token. |
| `Control D needs both a profile and a resolver` or `Control D needs max_profiles of 1 or more` at startup. | The `controld` config is not valid. Give `profile` and `resolver` together, and a `max_profiles` of 1 or more. Every run fails until you correct it, also a run that does not use Control D. |
| `Could not read Control D's locations`. | unshackle could not get the location list. It tries again at the next Control D query. |
+67 -16
View File
@@ -6,6 +6,7 @@ from __future__ import annotations
import base64
import http.server
import logging
import re
import socket
import socketserver
@@ -13,6 +14,7 @@ import threading
import time
from types import SimpleNamespace
from typing import Iterator
from unittest.mock import Mock
import click
import pytest
@@ -30,6 +32,7 @@ from unshackle.core.proxies.controld import (
parse_a_records,
)
from unshackle.core.proxies.resolve import REGION, resolve_proxy
from unshackle.core.utilities import get_free_port
pytestmark = pytest.mark.unit
@@ -74,9 +77,12 @@ def origin(body: bytes) -> http.server.HTTPServer:
@pytest.fixture
def forwarder() -> Iterator[LocalProxy]:
"""A forwarder whose resolver sends every hostname to 127.0.0.1, as Control D sends it to its proxy."""
proxy = LocalProxy(SimpleNamespace(resolve=lambda host: "127.0.0.1"))
def forwarder(request: pytest.FixtureRequest) -> Iterator[LocalProxy]:
"""A forwarder whose resolver sends every hostname to 127.0.0.1, as Control D sends it to its proxy.
A test can pass its own resolver as the indirect parameter.
"""
proxy = LocalProxy(getattr(request, "param", None) or SimpleNamespace(resolve=lambda host: "127.0.0.1"))
threading.Thread(target=proxy.serve_forever, daemon=True).start()
yield proxy
proxy.shutdown()
@@ -134,8 +140,8 @@ def test_connect_tunnels_bytes_both_ways(forwarder: LocalProxy) -> None:
@pytest.mark.parametrize(
("target", "status"),
[
("1.2.3.4:443", b"502"), # an IP literal would bypass Control D
("[::1]:443", b"502"),
("1.2.3.4:443", b"502 Bad Gateway (an IP address skips Control D's DNS"),
("[::1]:443", b"502 Bad Gateway (an IP address skips Control D's DNS"),
("a.test:x", b"400"),
],
)
@@ -145,18 +151,63 @@ def test_connect_refuses_what_it_cannot_redirect(forwarder: LocalProxy, target:
assert client.recv(128).startswith(b"HTTP/1.1 " + status)
def test_an_unreachable_host_answers_502() -> None:
def refuse(host: str) -> str:
raise ConnectionError("no A record")
def failing_resolver(error: Exception) -> Resolver:
resolver = Resolver("https://dns.controld.com/SECRETID")
resolver.session = SimpleNamespace(get=Mock(side_effect=error))
return resolver
proxy = LocalProxy(SimpleNamespace(resolve=refuse))
threading.Thread(target=proxy.serve_forever, daemon=True).start()
try:
response = requests.get("http://unresolvable.test/", proxies={"http": proxy.uri}, timeout=10)
assert response.status_code == 502
finally:
proxy.shutdown()
proxy.server_close()
@pytest.mark.parametrize(
("error", "reason"),
[
(
requests.ConnectionError("HTTPSConnectionPool(host='dns.controld.com'): /SECRETID"),
"Control D DNS could not be reached (ConnectionError)",
),
(requests.Timeout("read timed out: dns.controld.com/SECRETID"), "Control D DNS timed out"),
(
requests.HTTPError("503 for url: /SECRETID", response=SimpleNamespace(status_code=503)),
"Control D DNS answered HTTP 503",
),
],
)
def test_a_failed_lookup_hides_the_resolver(error: Exception, reason: str) -> None:
with pytest.raises(ConnectionError) as e:
failing_resolver(error).resolve("a.test")
assert str(e.value) == reason and e.value.__suppress_context__ # a traceback must not print the URL
@pytest.mark.parametrize(
("forwarder", "reason"),
[
(
SimpleNamespace(resolve=Mock(side_effect=ConnectionError("Control D returned no A record for a.test"))),
"no A record for a.test",
),
(
failing_resolver(requests.ConnectionError("dns.controld.com: /SECRETID")),
"Control D DNS could not be reached",
),
# a CR/LF left in the reason would end the status line early and cut the reason off
(
SimpleNamespace(resolve=Mock(side_effect=ConnectionError("no A record\r\nX-Injected: 1"))),
"no A record X-Injected: 1",
),
(None, "Control D sent a.test to 127.0.0.1, which did not answer"),
],
indirect=["forwarder"],
)
def test_a_failed_tunnel_names_the_reason(forwarder: LocalProxy, reason: str, caplog: pytest.LogCaptureFixture) -> None:
caplog.set_level(logging.WARNING, logger="ControlD")
with pytest.raises(requests.exceptions.ProxyError) as e:
requests.get(f"https://a.test:{get_free_port()}/", proxies={"https": forwarder.uri}, timeout=10)
assert "502 Bad Gateway (" in str(e.value) and reason in str(e.value)
assert "SECRETID" not in str(e.value)
response = requests.get(f"http://a.test:{get_free_port()}/", proxies={"http": forwarder.uri}, timeout=10)
assert response.status_code == 502
# clients that drop the reason phrase still get it once in the log
warnings = [r.getMessage() for r in caplog.records if r.levelno == logging.WARNING]
assert len(warnings) == 1 and reason in warnings[0]
def stub_session(response: bytes, seen: list) -> object:
+42 -4
View File
@@ -2,7 +2,9 @@
from __future__ import annotations
import contextlib
import os
import shutil
import time
from functools import partial
from pathlib import Path
@@ -78,6 +80,12 @@ def failing_parser(track, ctx):
raise RuntimeError("simulated crash")
def cancelled_parser(track, ctx):
seed(ctx)
(ctx.save_dir / "1.mp4.p.!dev").write_bytes(b"partial")
raise DownloadCancelled
def test_failed_track_resumes_in_next_run(temp_root, monkeypatch):
monkeypatch.setattr(config, "continue_downloads", True)
@@ -170,6 +178,40 @@ def test_rename_failure_never_replaces_download_error(temp_root, monkeypatch):
run_download(monkeypatch, failing_parser)
def test_locked_segment_never_replaces_download_error(temp_root, monkeypatch):
monkeypatch.setattr(config, "continue_downloads", False)
rmtree = shutil.rmtree
def locked(path, *args, **kwargs):
if Path(path).name == SEG_NAME:
raise PermissionError(32, "being used by another process", str(path))
rmtree(path, *args, **kwargs)
monkeypatch.setattr(shutil, "rmtree", locked)
with task_temp_dir():
with pytest.raises(RuntimeError, match="simulated crash"):
run_download(monkeypatch, failing_parser)
@pytest.mark.parametrize("parser", [failing_parser, cancelled_parser])
def test_locked_partial_still_stashes(temp_root, monkeypatch, parser):
monkeypatch.setattr(config, "continue_downloads", True)
unlink = Path.unlink
def locked(self, missing_ok=False):
if self.name.endswith(".!dev"):
raise PermissionError(32, "being used by another process", str(self))
unlink(self, missing_ok=missing_ok)
with task_temp_dir():
monkeypatch.setattr(Path, "unlink", locked)
with contextlib.suppress(RuntimeError):
run_download(monkeypatch, parser)
assert (temp_root / RESUME_STASH / SEG_NAME / "0.mp4").read_bytes() == b"complete segment"
def test_sweep_removes_only_old_stash_entries(temp_root):
stash = temp_root / RESUME_STASH
old, fresh = stash / "old_segments", stash / "fresh_segments"
@@ -195,10 +237,6 @@ def test_no_stash_outside_task_dir(temp_root):
def test_cancelled_track_is_stashed(temp_root, monkeypatch):
monkeypatch.setattr(config, "continue_downloads", True)
def cancelled_parser(track, ctx):
seed(ctx)
raise DownloadCancelled
with task_temp_dir():
run_download(monkeypatch, cancelled_parser)
+1 -1
View File
@@ -61,4 +61,4 @@ def test_compresses_large_json_when_accepted() -> None:
assert resp.headers.get("Content-Encoding") == "gzip"
decompressed = gzip.decompress(resp.body)
assert json.loads(decompressed) == big
assert resp.headers["Content-Length"] == str(len(resp.body))
assert resp.content_length == len(resp.body)
+31
View File
@@ -281,6 +281,37 @@ async def test_dashboard_session_logs(aiohttp_client, dashboard_cfg) -> None:
await store.delete(entry.session_id)
async def test_drain_refuses_new_work_and_warns_open_sessions(aiohttp_client, dashboard_cfg) -> None:
from unshackle.core.api.download_manager import get_download_manager
from unshackle.core.api.errors import APIError
from unshackle.core.api.session_log import SessionLogBuffer
from unshackle.core.api.session_store import get_session_store
store = get_session_store()
entry = await store.create("EXAMPLE", object())
entry.log_buffer = SessionLogBuffer()
client = await aiohttp_client(make_app())
headers = {"X-Secret-Key": "dash-secret"}
try:
body = await (await client.post("/api/dashboard/drain", headers=headers)).json()
assert body["draining"] is True and body["sessions"] == 1
await client.post("/api/dashboard/drain", headers=headers)
assert [r["level"] for r in entry.log_buffer.since(0)] == ["WARNING"]
refused = await client.post("/api/session/create", json={})
assert refused.status == 409
assert (await refused.json())["error_code"] == "SERVER_DRAINING"
with pytest.raises(APIError, match="restarts for an update"):
get_download_manager().create_job("EXAMPLE", "t")
body = await (await client.delete("/api/dashboard/drain", headers=headers)).json()
assert body["draining"] is False
assert (await client.post("/api/session/create", json={})).status == 400
finally:
store.stop_drain()
await store.delete(entry.session_id)
async def test_dashboard_log_read_does_not_touch_the_session() -> None:
from unshackle.core.api.session_store import SessionStore
+42 -15
View File
@@ -40,20 +40,21 @@ def test_api_error_default_http_status_per_code() -> None:
APIErrorCode.TRACK_NOT_FOUND: 404,
APIErrorCode.CONFLICT: 409,
APIErrorCode.RATE_LIMITED: 429,
APIErrorCode.INTERNAL_ERROR: 500,
APIErrorCode.DOWNLOAD_ERROR: 500,
APIErrorCode.WORKER_ERROR: 500,
APIErrorCode.INTERNAL_ERROR: 422,
APIErrorCode.DOWNLOAD_ERROR: 422,
APIErrorCode.WORKER_ERROR: 422,
APIErrorCode.SERVICE_ERROR: 422,
APIErrorCode.DRM_ERROR: 422,
APIErrorCode.NETWORK_ERROR: 503,
APIErrorCode.SERVICE_UNAVAILABLE: 503,
APIErrorCode.NETWORK_ERROR: 424,
APIErrorCode.SERVICE_UNAVAILABLE: 424,
APIErrorCode.SERVER_DRAINING: 409,
}
# Lock that the cases above are the *complete* live set; a new/removed code trips this.
assert {c.name for c in cases} == {c.name for c in APIErrorCode}
for code, expected in cases.items():
assert APIError(code, "x").http_status == expected, code
# the edge proxy replaces a 502 or 504 body, so no code may default to either
assert not {502, 504} & set(cases.values())
# the edge proxy can replace a 5xx body, so no code may default to one
assert all(status < 500 for status in cases.values())
def test_api_error_explicit_http_status_overrides_default() -> None:
@@ -73,7 +74,7 @@ def test_build_error_response_from_api_error() -> None:
body = _body(resp)
assert body["status"] == "error"
assert body["error_code"] == "SESSION_NOT_FOUND"
assert body["message"] == "no such session"
assert body["message"] == f"no such session (ref {body['ref']})"
assert body["details"] == {"session_id": "abc"}
assert "retryable" not in body
assert "debug_info" not in body
@@ -88,10 +89,10 @@ def test_build_error_response_retryable_flag() -> None:
def test_build_error_response_from_generic_exception() -> None:
resp = build_error_response(RuntimeError("oops"))
assert resp.status == 500
assert resp.status == 422
body = _body(resp)
assert body["error_code"] == "INTERNAL_ERROR"
assert body["message"] == "oops"
assert body["message"] == f"oops (ref {body['ref']})"
def test_build_error_response_debug_mode_includes_traceback() -> None:
@@ -161,15 +162,15 @@ def test_handle_api_exception_categorizes_generic() -> None:
"exc, status, code, retryable",
[
(Exception("Invalid credentials provided"), 401, "AUTH_FAILED", False),
(Exception("Connection refused"), 503, "NETWORK_ERROR", True),
(TimeoutError("read timeout"), 503, "NETWORK_ERROR", True),
(Exception("Connection refused"), 424, "NETWORK_ERROR", True),
(TimeoutError("read timeout"), 424, "NETWORK_ERROR", True),
(Exception("Not available in your region"), 403, "GEOFENCE", False),
(Exception("Title not found"), 404, "NOT_FOUND", False),
(Exception("HTTP 429 too many requests"), 429, "RATE_LIMITED", True),
(Exception("DRM license fetch failed"), 422, "DRM_ERROR", False),
(Exception("503 service unavailable"), 503, "SERVICE_UNAVAILABLE", True),
(Exception("503 service unavailable"), 424, "SERVICE_UNAVAILABLE", True),
(ValueError("malformed body"), 400, "INVALID_INPUT", False),
(RuntimeError("totally novel failure xyz"), 500, "INTERNAL_ERROR", False),
(RuntimeError("totally novel failure xyz"), 422, "INTERNAL_ERROR", False),
],
)
def test_error_contract_generic_exception(exc: Exception, status: int, code: str, retryable: bool) -> None:
@@ -188,6 +189,32 @@ def test_error_contract_apierror_passthrough_full_shape() -> None:
assert resp.status == 429
body = _body(resp)
assert body["error_code"] == "RATE_LIMITED"
assert body["message"] == "slow down"
assert body["message"] == f"slow down (ref {body['ref']})"
assert body["details"] == {"retry_after": 5}
assert body["retryable"] is True
async def test_outer_middleware_never_answers_5xx(aiohttp_client) -> None:
from aiohttp import web
from unshackle.core.api.routes import no_5xx_middleware
async def broken(request: web.Request) -> web.Response:
raise KeyError(request.headers["X-Secret-Key"])
async def subapp_500(_request: web.Request) -> web.Response:
return web.json_response({"status": 500, "message": "device error"}, status=500)
app = web.Application(middlewares=[no_5xx_middleware])
app.router.add_get("/broken", broken)
app.router.add_get("/subapp", subapp_500)
client = await aiohttp_client(app)
resp = await client.get("/broken", headers={"X-Secret-Key": "leaky-key"})
assert resp.status == 422
body = await resp.json()
assert body["error_code"] == "INTERNAL_ERROR" and "leaky-key" not in json.dumps(body)
resp = await client.get("/subapp")
assert resp.status == 422 and (await resp.json())["message"] == "device error"
assert (await client.get("/missing")).status == 404
+43
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import logging
from types import SimpleNamespace
from typing import Any
@@ -18,6 +19,7 @@ class FakeClient:
def __init__(self) -> None:
self.sent: dict[str, Any] = {}
self.tunnel: Any = None
def post(self, endpoint: str, data: dict[str, Any]) -> dict[str, Any]:
self.sent = {"endpoint": endpoint, **data}
@@ -69,3 +71,44 @@ def test_search_sends_no_login(server_accounts: Any, anonymous: bool) -> None:
list(svc.search())
assert client.sent == {"endpoint": "/api/search", "service": "TEST", "query": "query"}
def test_local_proxy_sends_the_server_a_tunnel(monkeypatch: pytest.MonkeyPatch) -> None:
class FakeTunnel:
def __init__(self, server_url: str, headers: dict) -> None:
self.id = "abc"
def start(self) -> str:
return self.id
def stop(self) -> None:
self.stopped = True
monkeypatch.setattr(remote_service, "TunnelClient", FakeTunnel)
svc, client, _ = make_service({"local_proxy": True})
svc.log = logging.getLogger("test")
client.session = SimpleNamespace(headers={"X-Secret-Key": "k"}) # type: ignore[attr-defined]
list(svc.search())
assert client.sent["proxy"] == "tunnel:abc"
assert client.tunnel.stopped
def test_a_loopback_proxy_points_to_local_proxy(monkeypatch: pytest.MonkeyPatch) -> None:
import click
monkeypatch.setattr(remote_service, "resolve_remote_proxy_arg", lambda arg, providers=None: arg)
svc, _, _ = make_service({"proxy": "http://127.0.0.1:8888"}, {"global": True})
with pytest.raises(click.ClickException, match="--local-proxy without --proxy"):
list(svc.search())
def test_local_proxy_is_refused_for_a_server_account_service() -> None:
import click
svc, _, _ = make_service({"local_proxy": True}, {"global": True})
with pytest.raises(click.ClickException, match="its own account"):
list(svc.search())
@@ -558,11 +558,12 @@ def test_session_create_writes_nested_cache_and_skips_colliding_keys(tmp_path, m
"create_service_instance",
lambda *args, **kwargs: (SimpleNamespace(log=logging.getLogger("Svc")), None, None),
)
monkeypatch.setattr("unshackle.core.api.session_store.refuse_while_draining", lambda: None)
monkeypatch.setattr("unshackle.core.api.session_store.get_session_store", lambda: (_ for _ in ()).throw(Stop()))
cache = {"x": blob('"flat"'), "x.json/y": blob('"collides"'), "MSL/keys.v2": blob('"dotted"')}
response = asyncio.run(handlers.session_create_handler({"service": "TESTSVC", "title_id": "t", "cache": cache}))
assert response.status == 500 # the Stop stand-in fired after the cache write
assert response.status == 422 # the Stop stand-in fired after the cache write
written = {p.name: p.read_text() for p in tmp_path.rglob("*.json") if p.is_file()}
assert written == {"x.json": '"flat"', "keys.v2.json": '"dotted"'}
+30 -1
View File
@@ -9,6 +9,7 @@ from __future__ import annotations
import asyncio
import json
from collections import Counter, defaultdict
import pytest
from aiohttp import web
@@ -132,7 +133,7 @@ async def test_api_handler_maps_apierror(aiohttp_client) -> None:
body = await resp.json()
assert body["status"] == "error"
assert body["error_code"] == "NOT_FOUND"
assert body["message"] == "nope"
assert body["message"] == f"nope (ref {body['ref']})"
assert body["details"] == {"x": 1}
@@ -180,6 +181,7 @@ async def test_heartbeat_late_error_counts_as_rejected(aiohttp_client, monkeypat
monkeypatch.setattr(routes, "HEARTBEAT_INTERVAL", 0.05)
monkeypatch.setattr(stats, "keys", {})
monkeypatch.setattr(stats, "errors", defaultdict(Counter))
@heartbeat
async def slow_fail(request: web.Request) -> web.Response:
@@ -195,6 +197,33 @@ async def test_heartbeat_late_error_counts_as_rejected(aiohttp_client, monkeypat
sent = len(await resp.read())
(entry,) = stats.keys.values()
assert resp.status == 200 and entry.rejected == 1 and entry.bytes_out >= sent > 0
assert stats.errors == {"-": {"SERVICE_ERROR": 1}}
@pytest.mark.parametrize(("service", "counted_as"), [("EXAMPLE", "EXAMPLE"), ("NOT_INSTALLED", "-")])
async def test_error_is_counted_by_service_and_logged_with_its_ref(
aiohttp_client, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture, service: str, counted_as: str
) -> None:
"""A gzipped error still reaches the stats: counted under an installed service only, and logged with its ref."""
from unshackle.core.api.stats import stats, stats_middleware
monkeypatch.setattr(stats, "errors", defaultdict(Counter))
@api_handler
async def fail(request: web.Request) -> web.Response:
await request.json()
raise APIError(APIErrorCode.SERVICE_ERROR, "denied", details={"pad": "x" * 300})
app = web.Application(middlewares=[stats_middleware, compression_middleware])
app["debug_api"] = False
app.router.add_post("/fail", fail)
client = await aiohttp_client(app)
resp = await client.post("/fail", json={"service": service}, headers={"Accept-Encoding": "gzip"})
body = await resp.json()
assert resp.headers["Content-Encoding"] == "gzip"
assert stats.errors == {counted_as: {"SERVICE_ERROR": 1}}
assert f"Error ref {body['ref']}: SERVICE_ERROR 422 POST /fail service={counted_as}" in caplog.text
async def test_heartbeat_client_disconnect_is_quiet(
+389
View File
@@ -0,0 +1,389 @@
"""The reverse tunnel: a server-side SOCKS5 proxy whose connections the client dials and relays."""
from __future__ import annotations
import asyncio
import http.server
import socket
import socketserver
import threading
import time
from typing import Callable, Iterator
import aiohttp
import click
import pytest
import requests
import responses
from aiohttp import web
from unshackle.core import tunnel
from unshackle.core.api.routes import tunnel as tunnel_route
from unshackle.core.api.routes import tunnel_check
from unshackle.core.tunnel import (
CHECK_ROUTE,
OPEN,
ROUTE,
TUNNELS,
TunnelClient,
connect_public,
frame,
is_public_ip,
tunnel_proxy,
)
class Hello(http.server.BaseHTTPRequestHandler):
hits = 0
def do_GET(self) -> None:
Hello.hits += 1
body = b"hello through the tunnel"
self.send_response(200)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *_: object) -> None:
pass
@pytest.fixture
def target() -> Iterator[int]:
server = socketserver.ThreadingTCPServer(("127.0.0.1", 0), Hello)
threading.Thread(target=server.serve_forever, daemon=True).start()
yield server.server_address[1]
server.shutdown()
@pytest.fixture
def server() -> Iterator[str]:
loop = asyncio.new_event_loop()
app = web.Application()
app.router.add_get(ROUTE, tunnel_route)
app.router.add_get(CHECK_ROUTE, tunnel_check)
runner = web.AppRunner(app)
loop.run_until_complete(runner.setup())
site = web.TCPSite(runner, "127.0.0.1", 0)
loop.run_until_complete(site.start())
port = site._server.sockets[0].getsockname()[1] # type: ignore[union-attr]
threading.Thread(target=loop.run_forever, daemon=True).start()
yield f"http://127.0.0.1:{port}"
asyncio.run_coroutine_threadsafe(tunnel.close_tunnels(app), loop).result(5)
loop.call_soon_threadsafe(loop.stop)
def test_requests_through_the_tunnel_reach_the_client_side_target(
server: str, target: int, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(tunnel, "is_public_ip", lambda address: True) # the test target is on loopback
client = TunnelClient(server, {"X-Secret-Key": "alice"})
tunnel_id = client.start()
proxy = tunnel_proxy(tunnel_id, "alice")
assert proxy and proxy.startswith(f"socks5h://{tunnel_id}:") and "@127.0.0.1:" in proxy
for _ in range(3):
resp = requests.get(f"http://127.0.0.1:{target}/", proxies={"all": proxy}, timeout=10)
assert resp.text == "hello through the tunnel"
client.stop()
def test_a_tunnel_is_only_usable_by_its_owner(server: str) -> None:
client = TunnelClient(server, {"X-Secret-Key": "alice"})
tunnel_id = client.start()
assert tunnel_proxy(tunnel_id, "mallory") is None
assert tunnel_proxy("unknown", "alice") is None
client.stop()
def test_the_client_refuses_a_target_on_its_own_network(server: str, target: int) -> None:
client = TunnelClient(server, {"X-Secret-Key": "alice"})
proxy = tunnel_proxy(client.start(), "alice")
with pytest.raises(requests.exceptions.ConnectionError):
requests.get(f"http://127.0.0.1:{target}/", proxies={"all": proxy}, timeout=10)
client.stop()
@pytest.mark.parametrize(
("address", "public"),
[("8.8.8.8", True), ("127.0.0.1", False), ("192.168.1.10", False), ("10.0.0.1", False), ("::1", False)],
)
def test_is_public_ip(address: str, public: bool) -> None:
assert is_public_ip(address) is public
def test_localhost_is_refused() -> None:
with pytest.raises(PermissionError):
asyncio.run(connect_public("localhost", 443))
def test_a_name_that_resolves_to_a_local_address_is_refused(monkeypatch: pytest.MonkeyPatch) -> None:
async def resolve(self: object, host: str, port: int, **_: object) -> list:
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("192.168.1.1", port))]
monkeypatch.setattr(asyncio.BaseEventLoop, "getaddrinfo", resolve)
with pytest.raises(PermissionError):
asyncio.run(connect_public("rebind.example", 443))
def local_proxy_ctx(params: dict, typed: set) -> click.Context:
from click.core import ParameterSource
ctx = click.Context(click.Command("dl"))
ctx.params = {"local_proxy": True, **params}
for key in params:
ctx.set_parameter_source(key, ParameterSource.COMMANDLINE if key in typed else ParameterSource.DEFAULT_MAP)
return ctx
def test_local_proxy_works_with_no_other_proxy_option() -> None:
from unshackle.core.proxies.resolve import require_alone_local_proxy
with pytest.raises(click.UsageError, match="--proxy, --proxy-download"):
require_alone_local_proxy(
local_proxy_ctx({"remote": True, "proxy": "us", "proxy_download": "us"}, {"proxy", "proxy_download"})
)
with pytest.raises(click.UsageError, match="needs --remote"):
require_alone_local_proxy(local_proxy_ctx({"remote": False}, set()))
def test_local_proxy_turns_off_a_proxy_from_the_config_defaults() -> None:
from unshackle.core.proxies.resolve import require_alone_local_proxy
ctx = local_proxy_ctx({"remote": True, "proxy": "us", "no_proxy_download": True}, set())
require_alone_local_proxy(ctx)
assert ctx.params["proxy"] is None and ctx.params["no_proxy_download"] is False
def test_the_tunnel_port_refuses_a_proxy_uri_without_the_tunnel_credentials(
server: str, target: int, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(tunnel, "is_public_ip", lambda address: True)
client = TunnelClient(server, {"X-Secret-Key": "alice"})
tunnel_id = client.start()
port = TUNNELS[tunnel_id].port
for uri in (f"socks5h://127.0.0.1:{port}", f"socks5h://{tunnel_id}:wrong@127.0.0.1:{port}"):
with pytest.raises(requests.exceptions.ConnectionError):
requests.get(f"http://127.0.0.1:{target}/", proxies={"all": uri}, timeout=10)
client.stop()
def test_the_server_never_dials_for_a_client(server: str, target: int) -> None:
"""An OPEN from the client must not make the server connect anywhere: only the client dials."""
async def send_open() -> None:
async with aiohttp.ClientSession(headers={"X-Secret-Key": "mallory"}) as http:
async with http.ws_connect(server + ROUTE) as ws:
await ws.receive_json()
await ws.send_bytes(frame(1, OPEN, target.to_bytes(2, "big") + b"127.0.0.1"))
Hello.hits = 0
asyncio.run(send_open())
assert Hello.hits == 0
def test_the_check_route_is_only_for_the_tunnel_owner(server: str) -> None:
client = TunnelClient(server, {"X-Secret-Key": "alice"})
tunnel_id = client.start()
resp = requests.get(f"{server}{CHECK_ROUTE}?id={tunnel_id}", headers={"X-Secret-Key": "mallory"}, timeout=10)
assert resp.status_code == 404
client.stop()
@pytest.mark.parametrize(
("uri", "line"),
[
("socks5://127.0.0.1:1080", "Using a local proxy on port 1080"),
("http://localhost:8888", "Using a local proxy on port 8888"),
("http://1.2.3.4:8080", "Using explicit Proxy: http://1.2.3.4:8080"),
("http://localhost", "Using a local proxy"),
("http://127.0.0.1:99999", "Using a local proxy"),
],
)
def test_a_local_proxy_logs_only_its_port(uri: str, line: str) -> None:
from unshackle.core.proxies.resolve import describe_explicit_proxy
assert describe_explicit_proxy(uri) == line
def test_another_api_key_cannot_reattach_to_a_tunnel(server: str) -> None:
client = TunnelClient(server, {"X-Secret-Key": "alice"})
tunnel_id = client.start()
async def reattach() -> None:
async with aiohttp.ClientSession(headers={"X-Secret-Key": "mallory"}) as http:
async with http.ws_connect(f"{server}{ROUTE}?id={tunnel_id}"):
pass
with pytest.raises(aiohttp.WSServerHandshakeError) as e:
asyncio.run(reattach())
assert e.value.status == 404
client.stop()
def test_a_request_that_is_not_a_websocket_opens_no_tunnel(server: str) -> None:
before = len(TUNNELS)
resp = requests.get(server + ROUTE, headers={"X-Secret-Key": "alice"}, timeout=10)
assert resp.status_code == 400
assert len(TUNNELS) == before
def test_a_malformed_frame_still_leaves_the_tunnel_to_expire(server: str) -> None:
async def send_short_frame() -> str:
async with aiohttp.ClientSession(headers={"X-Secret-Key": "mallory"}) as http:
async with http.ws_connect(server + ROUTE) as ws:
tunnel_id = (await ws.receive_json())["id"]
await ws.send_bytes(b"\x01")
return str(tunnel_id)
tunnel_id = asyncio.run(send_short_frame())
deadline = time.monotonic() + 5
while TUNNELS[tunnel_id].expiry is None and time.monotonic() < deadline:
time.sleep(0.05)
assert TUNNELS[tunnel_id].expiry is not None
def test_an_api_key_has_a_tunnel_limit(server: str) -> None:
clients = [TunnelClient(server, {"X-Secret-Key": "bob"}) for _ in range(tunnel.MAX_TUNNELS)]
for client in clients:
client.start()
with pytest.raises(ConnectionError, match="429"):
TunnelClient(server, {"X-Secret-Key": "bob"}).start()
for client in clients:
client.stop()
def test_a_connection_that_stops_reading_is_dropped() -> None:
sent: list = []
class FakeWS:
closed = False
async def send_bytes(self, data: bytes) -> None:
sent.append(data)
async def flood() -> tunnel.Link:
link = tunnel.Link(FakeWS())
link.streams[7] = tunnel.new_queue()
for _ in range(tunnel.BACKLOG + 5):
await link.feed(7, b"x")
return link
link = asyncio.run(flood())
assert 7 not in link.streams
assert sent == [frame(7, tunnel.CLOSE)]
def test_a_download_job_refuses_a_tunnel_proxy(monkeypatch: pytest.MonkeyPatch) -> None:
from unshackle.core.api import handlers
from unshackle.core.api.errors import APIError
monkeypatch.setattr(handlers, "server_cdm_allowed", lambda request, service: True)
monkeypatch.setattr(handlers, "server_cdm_max_height", lambda request, service: None)
with pytest.raises(APIError, match="tunnel"):
handlers.enforce_download_gates({"service": "EXAMPLE", "proxy": "tunnel:abc"}, None)
@responses.activate
def test_a_server_error_after_the_tunnel_closed_names_the_tunnel() -> None:
from types import SimpleNamespace
from unshackle.core.remote_service import RemoteClient, RemoteError
responses.add(
responses.GET,
"http://srv:8786/api/session/x/titles",
json={"status": "error", "error_code": "SERVICE_ERROR", "message": "proxy refused"},
status=422,
)
client = RemoteClient("http://srv:8786", "")
client.tunnel = SimpleNamespace(lost=True) # type: ignore[assignment]
with pytest.raises(RemoteError) as e:
client.get("/api/session/x/titles")
assert e.value.kind == "TUNNEL_CLOSED"
def spy_links(monkeypatch: pytest.MonkeyPatch) -> list:
"""Record every Link the client makes, so a test can look at its open streams."""
links: list = []
class SpyLink(tunnel.Link):
def __init__(self, ws: object) -> None:
super().__init__(ws)
links.append(self)
monkeypatch.setattr(tunnel, "Link", SpyLink)
return links
def wait_until(check: "Callable[[], bool]", seconds: float = 3) -> bool:
deadline = time.monotonic() + seconds
while not check() and time.monotonic() < deadline:
time.sleep(0.05)
return check()
def test_a_dial_the_server_gave_up_on_is_dropped_by_the_client(
server: str, target: int, monkeypatch: pytest.MonkeyPatch
) -> None:
"""When the server times out an OPEN, the client must not keep the connection it makes later."""
monkeypatch.setattr(tunnel, "is_public_ip", lambda address: True)
monkeypatch.setattr(tunnel, "OPEN_TIMEOUT", 0.3)
real_connect = tunnel.connect_public
async def slow_connect(host: str, port: int) -> tuple:
await asyncio.sleep(1)
return await real_connect(host, port)
monkeypatch.setattr(tunnel, "connect_public", slow_connect)
links = spy_links(monkeypatch)
client = TunnelClient(server, {"X-Secret-Key": "alice"})
proxy = tunnel_proxy(client.start(), "alice")
with pytest.raises(requests.exceptions.ConnectionError):
requests.get(f"http://127.0.0.1:{target}/", proxies={"all": proxy}, timeout=10)
assert wait_until(lambda: not links[-1].streams and not client.relays)
client.stop()
def socks5_connect(proxy: str, host: bytes) -> int:
"""Send one SOCKS5 CONNECT for ``host`` with the credentials in ``proxy`` and return the reply code."""
from urllib.parse import urlsplit
parts = urlsplit(proxy)
user, password = (parts.username or "").encode(), (parts.password or "").encode()
with socket.create_connection(("127.0.0.1", parts.port or 0), timeout=10) as sock:
sock.sendall(b"\x05\x01\x02")
assert sock.recv(2) == b"\x05\x02"
sock.sendall(b"\x01" + bytes([len(user)]) + user + bytes([len(password)]) + password)
assert sock.recv(2) == b"\x01\x00"
sock.sendall(b"\x05\x01\x00\x03" + bytes([len(host)]) + host + (443).to_bytes(2, "big"))
return sock.recv(10)[1]
def test_a_hostname_the_resolver_rejects_closes_its_stream(server: str, monkeypatch: pytest.MonkeyPatch) -> None:
async def resolve(self: object, host: str, port: int, **_: object) -> list:
host.encode("idna") # as socket.getaddrinfo does, so a long label raises UnicodeError
return []
monkeypatch.setattr(asyncio.BaseEventLoop, "getaddrinfo", resolve)
monkeypatch.setattr(tunnel, "OPEN_TIMEOUT", 1)
links = spy_links(monkeypatch)
client = TunnelClient(server, {"X-Secret-Key": "alice"})
proxy = tunnel_proxy(client.start(), "alice")
assert proxy
assert socks5_connect(proxy, b"a" * 64 + b".example") == 5
assert wait_until(lambda: not links[-1].streams)
client.stop()
def test_a_failed_start_leaves_no_thread_behind() -> None:
silent = socket.socket()
silent.bind(("127.0.0.1", 0))
silent.listen()
try:
client = TunnelClient(f"http://127.0.0.1:{silent.getsockname()[1]}", {"X-Secret-Key": "alice"})
with pytest.raises(ConnectionError, match="timed out"):
client.start(timeout=0.5)
assert not client.thread.is_alive()
finally:
silent.close()
+13 -1
View File
@@ -62,10 +62,12 @@ from unshackle.core.providers.anilist import parse_anilist_ref
from unshackle.core.providers.tvdb import SEASON_TYPES, parse_int
from unshackle.core.proxies.resolve import (
REGION,
describe_explicit_proxy,
describe_proxy,
is_loopback,
load_proxy_providers,
pick_proxy,
require_alone_local_proxy,
resolve_proxy,
split_proxy_query,
)
@@ -922,6 +924,12 @@ class dl:
default=None,
help="Proxy URI to use. If a 2-letter country is provided, it will try to get a proxy from the config.",
)
@click.option(
"--local-proxy",
is_flag=True,
default=False,
help="With --remote, the server sends its service requests out through this machine's own connection (and any VPN on it). Works with no other proxy option.",
)
@click.option(
"--tag",
type=str,
@@ -1598,6 +1606,10 @@ class dl:
if cdm_info:
log_event("load_cdm", level="INFO", service=self.service, context={"cdm": cdm_info})
if ctx.params.get("local_proxy"):
require_alone_local_proxy(ctx)
proxy, no_proxy = ctx.params.get("proxy"), bool(ctx.params.get("no_proxy"))
# A server on this machine can reach a local proxy, so only a server elsewhere is guarded
self.remote_needs_public_proxy = self.is_remote and not is_loopback(server_url(ctx.params.get("server")))
@@ -1644,7 +1656,7 @@ class dl:
ctx.params["proxy_query"] = proxy_query
ctx.params["proxy_provider"] = requested_provider
else:
self.log.info(f"Using explicit Proxy: {mask_proxy(ctx.params['proxy'])}")
self.log.info(describe_explicit_proxy(ctx.params["proxy"]))
ctx.params["proxy_query"] = None
ctx.params["proxy_provider"] = None
+21 -3
View File
@@ -16,12 +16,19 @@ from unshackle.core.config import config
from unshackle.core.console import console
from unshackle.core.constants import context_settings
from unshackle.core.proxies.proxy import Proxy
from unshackle.core.proxies.resolve import REGION, describe_proxy, load_proxy_providers, pick_proxy, split_proxy_query
from unshackle.core.proxies.resolve import (
REGION,
describe_explicit_proxy,
describe_proxy,
load_proxy_providers,
pick_proxy,
require_alone_local_proxy,
split_proxy_query,
)
from unshackle.core.service import Service
from unshackle.core.services import Services
from unshackle.core.utils.click_types import ContextData
from unshackle.core.utils.collections import ci_get, merge_dict
from unshackle.core.utils.redact import mask_proxy
@click.command(
@@ -38,6 +45,12 @@ from unshackle.core.utils.redact import mask_proxy
default=None,
help="Proxy URI to use. If a 2-letter country is provided, it will try get a proxy from the config.",
)
@click.option(
"--local-proxy",
is_flag=True,
default=False,
help="With --remote, the server sends its service requests out through this machine's own connection (and any VPN on it). Works with no other proxy option.",
)
@click.option("--no-proxy", is_flag=True, default=False, help="Force disable all proxy use.")
@click.option(
"--remote/--local",
@@ -58,6 +71,7 @@ def search(
no_proxy: bool,
profile: Optional[str] = None,
proxy: Optional[str] = None,
local_proxy: bool = False,
remote: bool = False,
server: Optional[str] = None,
):
@@ -81,6 +95,10 @@ def search(
log.info("Service Config loaded")
merge_dict(ci_get(config.services, service), service_config)
if ctx.params.get("local_proxy"):
require_alone_local_proxy(ctx)
proxy, no_proxy = ctx.params.get("proxy"), bool(ctx.params.get("no_proxy"))
proxy_providers: list[Proxy] = []
if no_proxy:
ctx.params["proxy"] = None
@@ -106,7 +124,7 @@ def search(
ctx.params["proxy"] = proxy_uri
log.info(describe_proxy(proxy_provider, proxy_uri, query))
else:
log.info(f"Using explicit Proxy: {mask_proxy(ctx.params['proxy'])}")
log.info(describe_explicit_proxy(ctx.params["proxy"]))
ctx.obj = ContextData(config=service_config, cdm=None, proxy_providers=proxy_providers, profile=profile)
+11 -2
View File
@@ -16,7 +16,7 @@ from aiohttp import web
import unshackle.core.config as config_module
from unshackle.core import binaries
from unshackle.core.api import cors_middleware, setup_routes, setup_swagger
from unshackle.core.api import cors_middleware, no_5xx_middleware, setup_routes, setup_swagger
from unshackle.core.api.compression import compression_middleware
from unshackle.core.api.events import publish_refresh_events, publish_service_event
from unshackle.core.api.handlers import (
@@ -495,12 +495,19 @@ def serve(
log.info("Starting REST API server (pywidevine/pyplayready CDM disabled)")
if no_key:
app = web.Application(
middlewares=[cors_middleware, stats_middleware, dashboard_authentication, compression_middleware]
middlewares=[
cors_middleware,
no_5xx_middleware,
stats_middleware,
dashboard_authentication,
compression_middleware,
]
)
else:
app = web.Application(
middlewares=[
cors_middleware,
no_5xx_middleware,
stats_middleware,
dashboard_authentication,
api_key_authentication,
@@ -569,6 +576,7 @@ def serve(
app = web.Application(
middlewares=[
cors_middleware,
no_5xx_middleware,
stats_middleware,
cdm_log_middleware,
dashboard_authentication,
@@ -580,6 +588,7 @@ def serve(
app = web.Application(
middlewares=[
cors_middleware,
no_5xx_middleware,
stats_middleware,
cdm_log_middleware,
dashboard_authentication,
+45
View File
@@ -0,0 +1,45 @@
import logging
from typing import Optional
import click
from unshackle.core.remote_service import remote_client, start_tunnel
from unshackle.core.tunnel import CHECK_ROUTE
from unshackle.core.utils.ip_info import get_ip_info
from unshackle.core.utils.redact import safe_display_url
@click.command(short_help="Check that a remote server can send its requests through this machine.")
@click.option(
"--server", type=str, default=None, help="Name of the remote server from remote_services config (if multiple)."
)
def tunnel_test(server: Optional[str]) -> None:
"""Check --local-proxy before a real --remote run.
The command finds the exit IP of this machine, opens a tunnel to the server, and makes the server
look up its exit IP through the tunnel. The two must match.
"""
log = logging.getLogger("tunnel-test")
local = get_ip_info()
if not local:
raise click.ClickException("Step 1 failed: an IP lookup from this machine did not work.")
log.info(f"Step 1: this machine exits at {local['ip']} ({str(local.get('country', '?')).upper()})")
client = remote_client(server)
tunnel = start_tunnel(client)
log.info(f"Step 2: opened a tunnel to {safe_display_url(client.server_url)}")
try:
remote = client.get(f"{CHECK_ROUTE}?id={tunnel.id}")
finally:
tunnel.stop()
country = str(remote.get("country", "?")).upper()
# one exit can answer over IPv4 here and IPv6 there, so a matching country also passes
if remote.get("ip") != local["ip"] and remote.get("country") != local.get("country"):
raise click.ClickException(
f"Step 3 failed: the server's requests exit at {remote.get('ip')} ({country}), "
f"not where this machine exits. Report this with the IP addresses."
)
log.info(f"Step 3: the server's requests exit at {remote['ip']} ({country}), {remote.get('ms')} ms for one lookup")
log.info("The tunnel works. Add --local-proxy to a --remote command.")
+2 -2
View File
@@ -1,3 +1,3 @@
from unshackle.core.api.routes import cors_middleware, setup_routes, setup_swagger
from unshackle.core.api.routes import cors_middleware, no_5xx_middleware, setup_routes, setup_swagger
__all__ = ["setup_routes", "setup_swagger", "cors_middleware"]
__all__ = ["setup_routes", "setup_swagger", "cors_middleware", "no_5xx_middleware"]
+3 -9
View File
@@ -49,12 +49,6 @@ async def compression_middleware(request: web.Request, handler) -> web.StreamRes
if not level:
return response
compressed = gzip.compress(body, compresslevel=level)
headers = dict(response.headers)
headers["Content-Encoding"] = "gzip"
headers["Content-Length"] = str(len(compressed))
return web.Response(
body=compressed,
status=response.status,
headers=headers,
)
response.body = gzip.compress(body, compresslevel=level)
response.headers["Content-Encoding"] = "gzip"
return response
+3
View File
@@ -735,6 +735,9 @@ class DownloadQueueManager:
def create_job(self, service: str, title_id: str, owner_key: Optional[str] = None, **parameters) -> DownloadJob:
"""Make a new download job and add it to the queue."""
from unshackle.core.api.session_store import refuse_while_draining
refuse_while_draining()
job_id = str(uuid.uuid4())
job = DownloadJob(
job_id=job_id,
+24 -12
View File
@@ -7,6 +7,7 @@ and optional debug information for the unshackle REST API.
from __future__ import annotations
import secrets
import traceback
from datetime import datetime, timezone
from enum import Enum
@@ -46,6 +47,7 @@ class APIErrorCode(str, Enum):
DRM_ERROR = "DRM_ERROR"
DOWNLOAD_ERROR = "DOWNLOAD_ERROR"
SERVICE_UNAVAILABLE = "SERVICE_UNAVAILABLE"
SERVER_DRAINING = "SERVER_DRAINING"
WORKER_ERROR = "WORKER_ERROR"
@@ -78,7 +80,11 @@ class APIError(Exception):
@staticmethod
def default_http_status(error_code: APIErrorCode) -> int:
"""Map error codes to default HTTP status codes."""
"""Map error codes to default HTTP status codes.
No code maps to a 5xx: a CDN in front of the server can replace a 5xx body with its own page.
Server-side failures are 422, and the ones worth a retry are 424, which the client retries.
"""
status_map = {
APIErrorCode.INVALID_INPUT: 400,
APIErrorCode.INVALID_SERVICE: 400,
@@ -94,16 +100,17 @@ class APIError(Exception):
APIErrorCode.SESSION_NOT_FOUND: 404,
APIErrorCode.TRACK_NOT_FOUND: 404,
APIErrorCode.CONFLICT: 409,
APIErrorCode.SERVER_DRAINING: 409,
APIErrorCode.SERVICE_ERROR: 422,
APIErrorCode.DRM_ERROR: 422,
APIErrorCode.INTERNAL_ERROR: 422,
APIErrorCode.DOWNLOAD_ERROR: 422,
APIErrorCode.WORKER_ERROR: 422,
APIErrorCode.NETWORK_ERROR: 424,
APIErrorCode.SERVICE_UNAVAILABLE: 424,
APIErrorCode.RATE_LIMITED: 429,
APIErrorCode.INTERNAL_ERROR: 500,
APIErrorCode.NETWORK_ERROR: 503,
APIErrorCode.SERVICE_UNAVAILABLE: 503,
APIErrorCode.DOWNLOAD_ERROR: 500,
APIErrorCode.WORKER_ERROR: 500,
}
return status_map.get(error_code, 500)
return status_map.get(error_code, 422)
def build_error_response(
@@ -114,6 +121,9 @@ def build_error_response(
"""
Assemble a structured JSON error response.
Each error gets a short ``ref``, also appended to ``message`` so that every client prints it.
The stats middleware logs the ref with the request, so a reported ref finds the server-side record.
Args:
error: APIError or generic Exception to convert to response
debug_mode: Whether to include technical debug information
@@ -132,13 +142,15 @@ def build_error_response(
error_code = APIErrorCode.INTERNAL_ERROR.value
message = str(error) or "An unexpected error occurred"
details = {}
http_status = 500
http_status = APIError.default_http_status(APIErrorCode.INTERNAL_ERROR)
retryable = False
ref = secrets.token_hex(3)
response_data: dict[str, Any] = {
"status": "error",
"error_code": error_code,
"message": message,
"message": f"{message} (ref {ref})",
"ref": ref,
"timestamp": datetime.now(timezone.utc).isoformat(),
}
@@ -161,7 +173,9 @@ def build_error_response(
response_data["debug_info"] = debug_info
return web.json_response(response_data, status=http_status)
response = web.json_response(response_data, status=http_status)
response["error"] = {"code": error_code, "ref": ref, "message": message}
return response
def categorize_exception(
@@ -227,7 +241,6 @@ def categorize_exception(
message=f"Network error occurred: {redact_secrets(str(exc))}",
details={**context, "reason": "network_connectivity"},
retryable=True,
http_status=503,
)
if any(keyword in exc_str for keyword in ["not found", "404", "does not exist", "invalid id"]):
@@ -261,7 +274,6 @@ def categorize_exception(
message=f"Service temporarily unavailable: {redact_secrets(str(exc))}",
details={**context, "reason": "service_unavailable"},
retryable=True,
http_status=503,
)
if any(keyword in exc_str for keyword in ["invalid", "malformed", "validation"]) or exc_type in [
+83 -2
View File
@@ -38,6 +38,7 @@ from unshackle.core.session import utf8_text
from unshackle.core.titles import Episode, Movie, Song, Title_T
from unshackle.core.titles.music import SONG_FIELDS
from unshackle.core.tracks import Audio, Subtitle, Tracks, Video
from unshackle.core.tunnel import HEARTBEAT, MAX_MESSAGE, MAX_TUNNELS, TUNNELS, Tunnel, tunnel_proxy
from unshackle.core.utilities import declared_kwargs, kind_only_conflict, original_tag
from unshackle.core.utils.click_types import AUDIO_CODEC_LIST, QUALITY_LIST, SUBTITLE_CODEC, VIDEO_CODEC_LIST
from unshackle.core.utils.collections import ci_get
@@ -2376,6 +2377,12 @@ def enforce_download_gates(params: Dict[str, Any], request: Optional[web.Request
if params.get("proxy_download") is not None and not isinstance(params["proxy_download"], str):
raise APIError(APIErrorCode.INVALID_INPUT, "proxy_download must be a string.")
if any(str(params.get(key) or "").startswith("tunnel:") for key in ("proxy", "proxy_download")):
raise APIError(
APIErrorCode.INVALID_PROXY,
"A download job cannot use a tunnel: proxy, because a tunnel belongs to one --remote client run.",
)
if not server_proxy_allowed(request):
resolve_handler_proxy(params, params.get("service") or "", request)
if params.get("proxy_download"):
@@ -2709,10 +2716,27 @@ async def dashboard_status_handler(request: web.Request) -> web.Response:
"max_sessions": store.max_sessions,
"session_ttl": store.ttl,
"jobs": dict(jobs),
"draining": store.draining,
}
)
async def dashboard_drain_start_handler(request: web.Request) -> web.Response:
"""Start a drain before a restart; answer with the status body."""
from unshackle.core.api.session_store import get_session_store
get_session_store().start_drain()
return await dashboard_status_handler(request)
async def dashboard_drain_stop_handler(request: web.Request) -> web.Response:
"""Cancel the drain; answer with the status body."""
from unshackle.core.api.session_store import get_session_store
get_session_store().stop_drain()
return await dashboard_status_handler(request)
async def dashboard_sessions_handler(request: web.Request) -> web.Response:
"""Every live remote session, not owner-scoped."""
from unshackle.core.api.session_store import get_session_store
@@ -3850,8 +3874,9 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
remote session on the server with the authenticated service instance, fetches all
titles and tracks, and returns everything the client needs for track selection.
"""
from unshackle.core.api.session_store import get_session_store
from unshackle.core.api.session_store import get_session_store, refuse_while_draining
refuse_while_draining()
service_tag = data.get("service")
title_id = data.get("title_id")
profile = client_profile(data)
@@ -4527,7 +4552,7 @@ def resolve_handler_proxy(
if (
proxy_param
and re.match(r"^(https?://|socks|controld://)", str(proxy_param))
and re.match(r"^(https?://|socks|controld://|tunnel:)", str(proxy_param))
and server_account_for(request, normalized_service)
):
raise APIError(
@@ -4536,6 +4561,17 @@ def resolve_handler_proxy(
details={"service": normalized_service},
)
if isinstance(proxy_param, str) and proxy_param.startswith("tunnel:") and not no_proxy:
proxy_param = tunnel_proxy(
proxy_param.removeprefix("tunnel:"), request_secret_key(request) if request else None
)
if not proxy_param:
raise APIError(
APIErrorCode.INVALID_PROXY,
"The tunnel for this proxy is not open. Run the command again.",
details={"service": normalized_service},
)
if allowed and not no_proxy:
proxy_providers = initialize_proxy_providers()
@@ -5637,3 +5673,48 @@ async def session_delete_handler(session_id: str, request: Optional[web.Request]
if cache_data:
response["cache"] = cache_data
return web.json_response(response)
async def tunnel_handler(request: web.Request) -> web.StreamResponse:
"""Open a ``--local-proxy`` tunnel, or reattach to the caller's tunnel named by ``?id=``."""
owner = request_secret_key(request)
tunnel_id = request.query.get("id")
tunnel = TUNNELS.get(tunnel_id) if tunnel_id else None
if tunnel_id and (not tunnel or tunnel.owner != owner):
raise APIError(APIErrorCode.NOT_FOUND, "No open tunnel has this id.")
if not tunnel and sum(t.owner == owner for t in TUNNELS.values()) >= MAX_TUNNELS:
raise APIError(APIErrorCode.RATE_LIMITED, f"An API key can have {MAX_TUNNELS} tunnels open at once.")
ws = web.WebSocketResponse(heartbeat=HEARTBEAT, max_msg_size=MAX_MESSAGE)
await ws.prepare(request) # a request that is not a WebSocket upgrade fails here, before a tunnel exists
if not tunnel:
tunnel = Tunnel(owner)
await tunnel.start()
await tunnel.serve(ws)
return ws
async def tunnel_check_handler(request: web.Request) -> web.Response:
"""Look up the exit IP through the caller's tunnel ``?id=``, so the client can confirm the route before a real run."""
import requests
from unshackle.core.utils.ip_info import verify_proxy_exit
proxy = tunnel_proxy(request.query.get("id", ""), request_secret_key(request))
if not proxy:
raise APIError(APIErrorCode.NOT_FOUND, "No open tunnel has this id.")
with requests.Session() as session:
session.proxies = {"all": proxy}
session.trust_env = False
started = time.perf_counter()
try:
info = await asyncio.to_thread(verify_proxy_exit, session)
except ConnectionError as e:
raise APIError(APIErrorCode.INVALID_PROXY, str(e), http_status=424)
if not info:
raise APIError(
APIErrorCode.INVALID_PROXY,
"The server could not look up an IP address through the tunnel.",
http_status=424,
)
ms = round((time.perf_counter() - started) * 1000)
return web.json_response({"ip": info.get("ip"), "country": info.get("country"), "ms": ms})
+142 -20
View File
@@ -22,6 +22,8 @@ from unshackle.core.api.handlers import (
clear_finished_download_jobs_handler,
clear_temp_handler,
dashboard_cdm_logs_handler,
dashboard_drain_start_handler,
dashboard_drain_stop_handler,
dashboard_events_handler,
dashboard_health_handler,
dashboard_jobs_handler,
@@ -66,9 +68,15 @@ from unshackle.core.api.handlers import (
session_segments_handler,
session_titles_handler,
session_tracks_handler,
tunnel_check_handler,
tunnel_handler,
)
from unshackle.core.api.sanitize import sanitize_log
from unshackle.core.drm import cbcs
from unshackle.core.services import Services
from unshackle.core.tunnel import CHECK_ROUTE as TUNNEL_CHECK_ROUTE
from unshackle.core.tunnel import ROUTE as TUNNEL_ROUTE
from unshackle.core.tunnel import close_tunnels
from unshackle.core.update_checker import UpdateChecker
from unshackle.core.utils.redact import redact_path
@@ -89,6 +97,24 @@ async def cors_middleware(
return response
@web.middleware
async def no_5xx_middleware(
request: web.Request, handler: Callable[[web.Request], Awaitable[web.StreamResponse]]
) -> web.StreamResponse:
"""Answer an uncaught exception or any unsent 5xx (a CDM subapplication's own) as a 422; see errors.py."""
try:
response = await handler(request)
except Exception as e:
if isinstance(e, web.HTTPException) and e.status < 500:
raise
# the exception text can hold a caller's key, so the answer gets a generic message
log.exception(f"Unhandled error in {sanitize_log(request.path)}")
response = build_error_response(APIError(APIErrorCode.INTERNAL_ERROR, "Internal server error"))
if response.status >= 500 and not response.prepared:
response.set_status(APIError.default_http_status(APIErrorCode.INTERNAL_ERROR))
return response
log = logging.getLogger("api")
# Route handler signature: takes the request, returns a response (streamed or complete).
@@ -147,6 +173,7 @@ def heartbeat(handler: Handler) -> Handler:
log.exception(f"Error in {request.path}")
result = handle_api_exception(e, debug_mode=debug_mode)
response["late_status"] = result.status
response["error"] = result.get("error")
body = result.body if isinstance(result, web.Response) else None
await response.write(bytes(body) if isinstance(body, (bytes, bytearray)) else b"{}")
await response.write_eof()
@@ -285,7 +312,7 @@ async def services(request: web.Request) -> web.Response:
type: string
nullable: true
description: Full service documentation
'500':
'422':
description: Server error
content:
application/json:
@@ -484,7 +511,7 @@ async def search(request: web.Request) -> web.Response:
description: Profile to use for credentials and cookies (default - None)
proxy:
type: string
description: Full proxy URI, a Control D resolver as controld://<resolver>@dns.controld.com, or a country code when the API key has server_proxy (default - None)
description: Full proxy URI, a Control D resolver as controld://<resolver>@dns.controld.com, or a country code when the API key has server_proxy, or tunnel:<id> for a tunnel the same API key opened on /api/tunnel (default - None)
no_proxy:
type: boolean
description: Force disable all proxy use (default - false)
@@ -692,7 +719,7 @@ async def list_titles(request: web.Request) -> web.Response:
timestamp:
type: string
format: date-time
'500':
'422':
description: Server error
content:
application/json:
@@ -1223,7 +1250,7 @@ async def download_jobs(request: web.Request) -> web.Response:
type: number
'400':
description: Invalid query parameters
'500':
'422':
description: Server error
"""
# Extract query parameters
@@ -1255,7 +1282,7 @@ async def download_job_detail(request: web.Request) -> web.Response:
description: Download job details
'404':
description: Job not found
'500':
'422':
description: Server error
"""
job_id = request.match_info["job_id"]
@@ -1295,7 +1322,7 @@ async def download_job_events(request: web.Request) -> web.StreamResponse:
type: string
'404':
description: Job not found
'500':
'422':
description: Server error
"""
job_id = request.match_info["job_id"]
@@ -1324,7 +1351,7 @@ async def cancel_download_job(request: web.Request) -> web.Response:
description: The server cannot cancel the job
'404':
description: Job not found
'500':
'422':
description: Server error
"""
job_id = request.match_info["job_id"]
@@ -1349,7 +1376,7 @@ async def clear_finished_download_jobs(request: web.Request) -> web.Response:
removed:
type: integer
description: Number of jobs removed
'500':
'422':
description: Server error
"""
return await clear_finished_download_jobs_handler(request)
@@ -1386,7 +1413,7 @@ async def retry_download_job(request: web.Request) -> web.Response:
description: Job not found
'409':
description: Job is not in a terminal state
'500':
'422':
description: Server error
"""
job_id = request.match_info["job_id"]
@@ -1423,7 +1450,7 @@ async def prioritize_download_job(request: web.Request) -> web.Response:
description: Job not found
'409':
description: Job is not queued
'500':
'422':
description: Server error
"""
job_id = request.match_info["job_id"]
@@ -1511,7 +1538,7 @@ async def profiles(request: web.Request) -> web.Response:
type: array
items:
type: string
'500':
'422':
description: Server error
"""
return await profiles_handler(request)
@@ -1579,7 +1606,7 @@ async def server_config(request: web.Request) -> web.Response:
items:
type: string
description: Available service tags (allowlist-filtered)
'500':
'422':
description: Server error
"""
return await server_config_handler(request)
@@ -1650,7 +1677,7 @@ async def download_history(request: web.Request) -> web.Response:
type: integer
'400':
description: Invalid query parameters
'500':
'422':
description: Server error
"""
query_params = {"limit": request.query.get("limit"), "service": request.query.get("service")}
@@ -1675,7 +1702,7 @@ async def delete_history(request: web.Request) -> web.Response:
description: History entry removed
'404':
description: History entry not found
'500':
'422':
description: Server error
"""
return await delete_history_handler(request.match_info["job_id"], request)
@@ -1700,7 +1727,7 @@ async def maintenance_clear_cache(request: web.Request) -> web.Response:
type: boolean
freed_bytes:
type: integer
'500':
'422':
description: Server error
"""
return await clear_cache_handler(request)
@@ -1725,7 +1752,7 @@ async def maintenance_clear_temp(request: web.Request) -> web.Response:
type: boolean
freed_bytes:
type: integer
'500':
'422':
description: Server error
"""
return await clear_temp_handler(request)
@@ -1777,7 +1804,7 @@ async def maintenance_refresh_services(request: web.Request) -> web.Response:
description: Services whose new code failed to import
items:
type: string
'500':
'422':
description: Server error
"""
return await refresh_services_handler(request)
@@ -1812,7 +1839,7 @@ async def env_check(request: web.Request) -> web.Response:
nullable: true
required:
type: boolean
'500':
'422':
description: Server error
"""
return await env_check_handler(request)
@@ -1847,6 +1874,7 @@ async def session_create(request: web.Request) -> web.Response:
type: string
proxy:
type: string
description: As on /api/search, including tunnel:<id> for a --local-proxy client
no_proxy:
type: boolean
profile:
@@ -1996,6 +2024,57 @@ async def session_create(request: web.Request) -> web.Response:
)
@api_handler
async def tunnel(request: web.Request) -> web.StreamResponse:
"""
Open a --local-proxy tunnel.
---
summary: Open tunnel
description: >-
A WebSocket that a --remote client holds open. The server sends the requests of a remote session
created with proxy tunnel:<id> through it, and the client makes them from its own connection.
The first message is {"id": <tunnel id>}.
parameters:
- name: id
in: query
required: false
description: Reattach to this tunnel after a drop
schema:
type: string
responses:
'101':
description: WebSocket open
'404':
description: No open tunnel with this id for this API key
'429':
description: The API key has the maximum number of tunnels open
"""
return await tunnel_handler(request)
@api_handler
async def tunnel_check(request: web.Request) -> web.Response:
"""
Look up the exit IP through a tunnel.
---
summary: Check tunnel
parameters:
- name: id
in: query
required: true
schema:
type: string
responses:
'200':
description: The exit IP and country through the tunnel, and the lookup time in ms
'404':
description: No open tunnel with this id for this API key
'424':
description: No IP lookup got through the tunnel, or the requests left from the server itself
"""
return await tunnel_check_handler(request)
@api_handler
@heartbeat
async def session_titles(request: web.Request) -> web.Response:
@@ -2581,8 +2660,8 @@ async def dashboard_status(request: web.Request) -> web.Response:
---
summary: Dashboard server status
description: >
Version, uptime, bind address, mode, request counters per API key, and counts of
services, remote sessions and jobs. Every /api/dashboard/ route needs
Version, uptime, bind address, mode, request counters per API key, error counts by
service and error code, and counts of services, remote sessions and jobs. Every /api/dashboard/ route needs
`serve.dashboard.key` in X-Secret-Key; a user key never passes, and the routes do not
exist without a dashboard key.
tags: [Dashboard]
@@ -2595,6 +2674,44 @@ async def dashboard_status(request: web.Request) -> web.Response:
return await dashboard_status_handler(request)
@api_handler
async def dashboard_drain_start(request: web.Request) -> web.Response:
"""
Dashboard: start a drain before a restart.
---
summary: Dashboard drain start
description: >
Refuses every new remote session, download job and job retry with `SERVER_DRAINING` (409)
until the server restarts or the operator cancels the drain. Open remote sessions continue, and
each one gets a warning in its service log. Poll `/api/dashboard/status` until `sessions`
and the active jobs reach 0, then restart. A second call changes nothing.
tags: [Dashboard]
responses:
'200':
description: The status body, with `draining` true
'401':
description: Dashboard key missing or invalid
"""
return await dashboard_drain_start_handler(request)
@api_handler
async def dashboard_drain_stop(request: web.Request) -> web.Response:
"""
Dashboard: cancel a drain.
---
summary: Dashboard drain cancel
description: The server accepts new remote sessions and download jobs again.
tags: [Dashboard]
responses:
'200':
description: The status body, with `draining` false
'401':
description: Dashboard key missing or invalid
"""
return await dashboard_drain_stop_handler(request)
@api_handler
async def dashboard_sessions(request: web.Request) -> web.Response:
"""
@@ -3259,6 +3376,8 @@ ROUTES: list[tuple[str, str, Handler, bool]] = [
("GET", "/api/remote/logins", remote_logins, False),
("POST", "/api/remote/logins/{login_id}/input", remote_login_input, False),
("POST", "/api/session/create", session_create, True),
("GET", TUNNEL_ROUTE, tunnel, True),
("GET", TUNNEL_CHECK_ROUTE, tunnel_check, True),
("GET", "/api/session/{session_id}/titles", session_titles, True),
("POST", "/api/session/{session_id}/tracks", session_tracks, True),
("POST", "/api/session/{session_id}/segments", session_segments, True),
@@ -3275,6 +3394,8 @@ ROUTES: list[tuple[str, str, Handler, bool]] = [
DASHBOARD_ROUTES: list[tuple[str, str, Handler, bool]] = [
("GET", DASHBOARD_PREFIX + "status", dashboard_status, True),
("POST", DASHBOARD_PREFIX + "drain", dashboard_drain_start, True),
("DELETE", DASHBOARD_PREFIX + "drain", dashboard_drain_stop, True),
("GET", DASHBOARD_PREFIX + "sessions", dashboard_sessions, True),
("GET", DASHBOARD_PREFIX + "jobs", dashboard_jobs, True),
("GET", DASHBOARD_PREFIX + "logs", dashboard_logs, True),
@@ -3303,6 +3424,7 @@ def setup_routes(
if remote_only and not remote:
continue
add[method](path, handler)
app.on_shutdown.append(close_tunnels)
def setup_swagger(app: web.Application, dashboard: bool = False) -> None:
+34
View File
@@ -24,6 +24,11 @@ from unshackle.core.tracks import Track
log = logging.getLogger("api.session")
DRAIN_WARNING = (
"This server restarts for an update when the open downloads finish. "
"This download continues. Start new downloads after the restart."
)
@dataclass
class SessionEntry:
@@ -129,6 +134,7 @@ class SessionStore:
self._sessions: Dict[str, SessionEntry] = {}
self._lock = asyncio.Lock()
self._cleanup_task: Optional[asyncio.Task] = None
self.draining = False
@property
def ttl(self) -> int:
@@ -322,6 +328,21 @@ class SessionStore:
"""Snapshot of every live session."""
return list(self._sessions.values())
def start_drain(self) -> None:
"""Refuse new sessions and jobs until the restart, and warn each open session through its service log."""
if self.draining:
return
self.draining = True
log.warning(f"Draining: new sessions and jobs are refused, {len(self._sessions)} session(s) open")
for entry in self._sessions.values():
if entry.log_buffer is not None:
entry.log_buffer.append(logging.WARNING, DRAIN_WARNING)
def stop_drain(self) -> None:
if self.draining:
self.draining = False
log.warning("Drain cancelled: new sessions and jobs are accepted again")
@property
def session_count(self) -> int:
"""Number of active sessions."""
@@ -331,6 +352,19 @@ class SessionStore:
session_store: Optional[SessionStore] = None
def refuse_while_draining() -> None:
"""Raise SERVER_DRAINING for a new session or job once the operator has started a drain."""
from unshackle.core.api.errors import APIError, APIErrorCode
if get_session_store().draining:
raise APIError(
APIErrorCode.SERVER_DRAINING,
"This server restarts for an update and does not start new sessions or downloads now. "
"Try again in a few minutes.",
retryable=True,
)
def get_session_store() -> SessionStore:
"""Get or make the global remote session store singleton."""
global session_store
+36 -8
View File
@@ -7,7 +7,7 @@ import hashlib
import json
import logging
import time
from collections import Counter, deque
from collections import Counter, defaultdict, deque
from dataclasses import dataclass, field
from functools import lru_cache
from typing import Any, Deque, Dict, List, Optional, Tuple
@@ -143,6 +143,7 @@ class ServerStats:
requests_total: int = 0
requests_rejected: int = 0
keys: Dict[str, KeyStats] = field(default_factory=dict)
errors: Dict[str, Counter] = field(default_factory=lambda: defaultdict(Counter)) # service -> error_code -> count
def key_stats(self, key: Optional[str]) -> KeyStats:
"""The counters for an API key, created on first sight."""
@@ -194,6 +195,7 @@ class ServerStats:
"requests_total": self.requests_total,
"requests_rejected": self.requests_rejected,
"requests_by_key": dict(self.requests_by_key),
"errors": {service: dict(codes) for service, codes in self.errors.items()},
}
@@ -203,16 +205,18 @@ stats = ServerStats()
@web.middleware
async def stats_middleware(request: web.Request, handler: Any) -> web.StreamResponse:
from unshackle.core.api.handlers import request_secret_key
from unshackle.core.api.sanitize import sanitize_log
stats.requests_total += 1
started = time.perf_counter()
response = await handler(request)
status = response.get("late_status", response.status)
bytes_out = getattr(response, "content_length", None) or response.body_length
key = None if status == 401 else configured_key(request_secret_key(request))
if status == 401:
stats.requests_rejected += 1
else:
entry = stats.key_stats(configured_key(request_secret_key(request)))
entry = stats.key_stats(key)
entry.requests += 1
entry.last_seen = time.time()
entry.bytes_out += bytes_out
@@ -235,9 +239,33 @@ async def stats_middleware(request: web.Request, handler: Any) -> web.StreamResp
"bytes_out": bytes_out,
},
)
error = response.get("error")
if error:
service = await _request_service(request, session_id)
stats.errors[service][error["code"]] += 1
logging.getLogger("api").warning(
f"Error ref {error['ref']}: {error['code']} {status} {request.method} {sanitize_log(request.path)} "
f"service={service} user={sanitize_log(mask_key(key))}: {sanitize_log(error['message'])}"
)
return response
async def _request_service(request: web.Request, session_id: Optional[str]) -> str:
"""The installed service tag a request is for, from its remote session or its JSON body; ``-`` if none.
Only an installed tag counts, so a caller cannot grow ``stats.errors`` with made-up names.
"""
from unshackle.core.api.session_store import get_session_store
from unshackle.core.services import Services
entry = get_session_store().peek(session_id) if session_id else None
service = entry.service_tag if entry is not None else (await _consumed_json(request)).get("service")
if not isinstance(service, str):
return "-"
tag = Services.get_tag(service)
return tag if tag in Services.get_tags() else "-"
class SeqRing:
"""Keep the last N records in memory, each with a ``seq`` one above the record before it."""
@@ -347,20 +375,20 @@ def _reply_json(response: Any) -> Dict[str, Any]:
return data if isinstance(data, dict) else {}
async def _request_session_id(request: web.Request) -> Any:
"""The ``session_id`` of the JSON request body, when the handler consumed it: aiohttp caches that read.
async def _consumed_json(request: web.Request) -> Dict[str, Any]:
"""The JSON object of the request body when the handler consumed it (aiohttp caches that read), else ``{}``.
A body nobody read belongs to a call refused before the handler, a 401 or a 429. It stays
unread: reading it would spend time and memory on an unauthenticated caller, and wait on a
body that never finishes arriving.
"""
if not request.content.at_eof():
return None
return {}
try:
body = await request.json()
except Exception:
return None
return body.get("session_id") if isinstance(body, dict) else None
return {}
return body if isinstance(body, dict) else {}
async def _cdm_record(request: web.Request, drm: str, op: str, outcome: Any) -> Dict[str, Any]:
@@ -388,7 +416,7 @@ async def _cdm_record(request: web.Request, drm: str, op: str, outcome: Any) ->
elif op == "close":
session_id = request.match_info.get("session_id")
else:
session_id = await _request_session_id(request)
session_id = (await _consumed_json(request)).get("session_id")
return {
"drm": drm,
"op": op,
+34 -13
View File
@@ -102,6 +102,7 @@ class Resolver:
self.lock = threading.Lock()
def resolve(self, host: str) -> str:
"""Get the IPv4 address Control D gives `host`. A failure raises `ConnectionError` without the resolver ID."""
host = host.lower()
with self.lock:
cached = self.cache.get(host)
@@ -109,13 +110,20 @@ class Resolver:
return cached[0]
query = base64.urlsafe_b64encode(build_query(host)).rstrip(b"=").decode()
response = self.session.get(
self.url,
params={"dns": query},
headers={"accept": "application/dns-message"},
timeout=DOH_TIMEOUT,
)
response.raise_for_status()
try:
response = self.session.get(
self.url,
params={"dns": query},
headers={"accept": "application/dns-message"},
timeout=DOH_TIMEOUT,
)
response.raise_for_status()
except requests.HTTPError as e:
raise ConnectionError(f"Control D DNS answered HTTP {e.response.status_code}") from None
except requests.Timeout:
raise ConnectionError("Control D DNS timed out") from None
except requests.RequestException as e:
raise ConnectionError(f"Control D DNS could not be reached ({type(e).__name__})") from None
records = parse_a_records(response.content)
if not records:
@@ -158,6 +166,17 @@ class Handler(socketserver.StreamRequestHandler):
except OSError:
pass
def refuse(self, host: str, port: int, reason: str) -> None:
"""Answer 502 with the reason in the status line, which a client shows as `Tunnel connection failed: ...`.
Some HTTP clients drop the reason phrase, so this also logs each new reason once as a warning.
"""
reason = " ".join(reason.split())
if (host, reason) not in self.server.refused:
self.server.refused.add((host, reason))
log.warning(f"Could not reach {host}:{port}: {reason}")
self.reply(f"502 Bad Gateway ({reason})".encode("latin-1", "replace"))
def forward_response_head(self, upstream: socket.socket) -> None:
"""Send the response head on with `Connection: close`, so the client never reuses this connection."""
data = b""
@@ -212,16 +231,16 @@ class Handler(socketserver.StreamRequestHandler):
except ValueError:
pass
else:
# an IP literal bypasses Control D's DNS and would leak this machine's own IP
log.debug(f"Refused {host}:{port}, an address that Control D cannot redirect")
return self.reply(b"502 Bad Gateway")
return self.refuse(host, port, "an IP address skips Control D's DNS and would expose this machine's IP")
try:
ip = self.server.resolver.resolve(host)
upstream = socket.create_connection((ip, port), timeout=HEAD_TIMEOUT)
except Exception as e:
log.debug(f"Could not reach {host}:{port}: {e}")
return self.reply(b"502 Bad Gateway")
return self.refuse(host, port, str(e) or type(e).__name__)
try:
upstream = socket.create_connection((ip, port), timeout=HEAD_TIMEOUT)
except OSError as e:
return self.refuse(host, port, f"Control D sent {host} to {ip}, which did not answer: {e.strerror or e}")
self.connection.settimeout(IDLE_TIMEOUT)
upstream.settimeout(IDLE_TIMEOUT)
@@ -253,6 +272,7 @@ class LocalProxy(socketserver.ThreadingTCPServer):
"""An HTTP forward proxy on loopback that resolves every host through `resolver`."""
super().__init__(("127.0.0.1", 0), Handler)
self.resolver = resolver
self.refused: set[tuple[str, str]] = set()
@property
def uri(self) -> str:
@@ -460,6 +480,7 @@ class ControlD(Proxy):
if server:
with server.resolver.lock:
server.resolver.cache.clear()
server.refused.clear()
REGIONS.pop(profile, None)
REGIONS[profile] = query
LOCATIONS[profile] = location
+35
View File
@@ -8,6 +8,9 @@ import re
from typing import Any, List, Optional
from urllib.parse import urlparse
import click
from click.core import ParameterSource
from unshackle.core.proxies.proxy import Proxy
from unshackle.core.utils.redact import mask_proxy
@@ -189,6 +192,38 @@ def describe_proxy(provider: Any, uri: str, query: str, allow_debug: bool = True
return f"Using {name} Proxy: {mask_proxy(uri, isinstance(provider, Basic), allow_debug)}"
def require_alone_local_proxy(ctx: click.Context) -> None:
"""Check ``--local-proxy``: it needs a remote run and no other proxy option.
A proxy option that comes from the ``dl`` config defaults, not the command line, is cleared instead.
"""
if not ctx.params.get("remote"):
raise click.UsageError(
"--local-proxy needs --remote: it sends a remote server's requests through this machine."
)
given = []
for key in ("proxy", "no_proxy", "proxy_download", "no_proxy_download"):
if not ctx.params.get(key):
continue
if ctx.get_parameter_source(key) in (ParameterSource.DEFAULT, ParameterSource.DEFAULT_MAP):
ctx.params[key] = False if key.startswith("no_") else None
else:
given.append("--" + key.replace("_", "-"))
if given:
raise click.UsageError(f"--local-proxy does not work with {', '.join(given)}.")
def describe_explicit_proxy(uri: str) -> str:
"""The log line for a proxy URI typed in full: a proxy on this machine shows only as its port."""
if is_loopback(uri):
try:
port = urlparse(uri if "//" in uri else f"//{uri}").port
except ValueError:
port = None
return f"Using a local proxy on port {port}" if port else "Using a local proxy"
return f"Using explicit Proxy: {mask_proxy(uri)}"
def resolve_proxy(proxy: str, proxy_providers: List[Any]) -> Optional[str]:
"""Change a proxy parameter to a real proxy URI.
+56 -5
View File
@@ -50,6 +50,7 @@ from unshackle.core.titles.music import SONG_FIELDS, Album, Song
from unshackle.core.tracks import Audio, Chapter, Chapters, Subtitle, Tracks, Video
from unshackle.core.tracks.attachment import Attachment
from unshackle.core.tracks.track import Track
from unshackle.core.tunnel import TunnelClient
from unshackle.core.utils import run_proxy
from unshackle.core.utils.collections import ci_get
from unshackle.core.utils.redact import redact_path, redact_text, safe_display_url
@@ -104,6 +105,7 @@ class RemoteClient:
self.auth_headers = list(auth_headers) if auth_headers else list(DEFAULT_AUTH_HEADERS)
self._auth_header_index = 0
self._session: Optional[requests.Session] = None
self.tunnel: Optional[TunnelClient] = None
# wraps every request, so the service's CDM calls get answered while the server runs its code
self.in_flight: Callable[[str], contextlib.AbstractContextManager[Any]] = lambda _: contextlib.nullcontext()
@@ -192,6 +194,12 @@ class RemoteClient:
if optional:
log.debug(f"Optional endpoint {endpoint} unavailable [{error_code}]: {error_msg}")
return {}
if self.tunnel and self.tunnel.lost:
raise RemoteError(
"The tunnel to the server closed, so the server can no longer send its requests through this "
"machine. Run the command again.",
"TUNNEL_CLOSED",
)
raise RemoteError(f"Server error [{error_code}]: {error_msg}", error_code)
return detail
@@ -920,6 +928,18 @@ def resolve_remote_proxy_arg(proxy_arg: str, providers: Optional[list] = None) -
raise click.ClickException(str(e))
def start_tunnel(client: RemoteClient) -> TunnelClient:
"""Open a tunnel that sends the server's requests out through this machine's own connection."""
headers = {k: str(v) for k, v in client.session.headers.items() if k.lower() != "connection"}
tunnel = TunnelClient(client.server_url, headers)
try:
tunnel.start()
except ConnectionError as e:
raise click.ClickException(f"--local-proxy needs a tunnel to the server. {e}")
client.tunnel = tunnel
return tunnel
def server_proxy_arg(params: Dict[str, Any]) -> str:
"""The --proxy value to resolve for a server: the one typed when it resolved to a proxy on this machine."""
proxy = params["proxy"]
@@ -1164,11 +1184,17 @@ class RemoteService:
data.update(self.client_login(profile))
if params.get("no_proxy"):
data["no_proxy"] = True
elif params.get("local_proxy"):
data["proxy"] = self.tunnel()
elif params.get("proxy"):
providers = getattr(self.ctx.obj, "proxy_providers", None)
data["proxy"] = resolve_remote_proxy_arg(server_proxy_arg(params), providers)
self.require_reachable_proxy(data["proxy"])
response = self.client.post("/api/search", data)
try:
response = self.client.post("/api/search", data)
finally:
if self.client.tunnel: # a search is one request, so its tunnel ends with it
self.client.tunnel.stop()
self.save_returned_cache(response.get("cache") or {}, profile)
for result in response.get("results") or []:
yield SearchResult(
@@ -1189,7 +1215,22 @@ class RemoteService:
def require_reachable_proxy(self, proxy: Optional[str]) -> None:
"""Refuse a proxy on this machine when the server is elsewhere: the server cannot reach it."""
if proxy and is_loopback(proxy) and not is_loopback(self.client.server_url):
raise click.ClickException("That proxy is on your machine, so --remote cannot use it.")
raise click.ClickException(
"That proxy is on your machine, so --remote cannot use it. "
"To send the server's requests through this machine's connection, use --local-proxy without --proxy."
)
def tunnel(self) -> str:
"""The ``tunnel:<id>`` proxy to send the server for ``--local-proxy``, opening the tunnel the first time."""
if self._server_accounts is not None:
raise click.ClickException(
f"The server logs in to {self.service_tag} with its own account, so --local-proxy does not work with it."
)
tunnel = self.client.tunnel
if tunnel is None:
tunnel = start_tunnel(self.client)
self.log.info("The server sends its requests through your connection on this machine")
return f"tunnel:{tunnel.id}"
def authenticate(self, cookies: Optional[CookieJar] = None, credential: Optional[Credential] = None) -> None:
self.credential = credential
@@ -1219,7 +1260,10 @@ class RemoteService:
pass
providers = getattr(self.ctx.obj, "proxy_providers", None)
if not no_proxy and not proxy and self._server_accounts and not self._server_accounts.get("global"):
local = bool(self.ctx.parent and self.ctx.parent.params.get("local_proxy"))
if local:
create_data["proxy"] = self.tunnel()
elif not no_proxy and not proxy and self._server_accounts and not self._server_accounts.get("global"):
regions = [str(r).lower() for r in self._server_accounts.get("regions") or []]
if regions and client_region and client_region not in regions:
try:
@@ -1241,7 +1285,7 @@ class RemoteService:
if country:
create_data["proxy_region"] = country.group(1).lower()
self.require_reachable_proxy(create_data.get("proxy"))
local_proxy = None if no_proxy else (proxy or create_data.get("proxy"))
local_proxy = None if no_proxy or local else (proxy or create_data.get("proxy"))
if local_proxy:
self._session.proxies["all"] = local_proxy
run_proxy.pin(self._session)
@@ -1334,7 +1378,12 @@ class RemoteService:
"argv": argv,
}
result = self.client.post("/api/session/create", create_data)
try:
result = self.client.post("/api/session/create", create_data)
except BaseException:
if local and self.client.tunnel: # no session, so close() never runs: free the tunnel now
self.client.tunnel.stop()
raise
self._session_id = result["session_id"]
atexit.register(self.close)
# an older server omits the field, and the tracks response settles it
@@ -1975,6 +2024,8 @@ class RemoteService:
pass
except Exception as e:
self.log.warning(f"Failed to clean up remote session: {e}")
if self.client.tunnel:
self.client.tunnel.stop()
def save_returned_cache(self, cache_data: Dict[str, str], profile: Optional[str] = None) -> None:
"""Save cache files returned by the server to the local cache directory.
+14 -9
View File
@@ -1,4 +1,5 @@
import base64
import contextlib
import html
import logging
import re
@@ -768,6 +769,17 @@ class Track:
shutil.rmtree(save_dir)
resume.clear_sidecar(save_dir)
def abandon() -> None:
"""Clean up a failed or cancelled download and stash its segments when resume is on.
The downloader does not wait for its workers on failure, so on Windows one can still hold a
.!dev file open (WinError 32). A cleanup error must not replace the download's own error.
"""
with contextlib.suppress(OSError):
cleanup()
if keep_segments:
stash_segments(save_dir)
if not DOWNLOAD_LICENCE_ONLY.is_set():
if config.directories.temp.is_file():
raise ValueError(f"Temp Directory '{config.directories.temp}' must be a Directory, not a file")
@@ -927,19 +939,12 @@ class Track:
progress(downloaded="[red]FAILED")
raise
except DownloadCancelled:
try:
cleanup()
if keep_segments:
stash_segments(save_dir)
except OSError:
pass
abandon()
progress(downloaded="[yellow]SKIPPED")
return
except (Exception, KeyboardInterrupt):
if not DOWNLOAD_LICENCE_ONLY.is_set():
cleanup()
if keep_segments:
stash_segments(save_dir)
abandon()
raise
if DOWNLOAD_CANCELLED.is_set():
+412
View File
@@ -0,0 +1,412 @@
"""Reverse tunnel: a remote client's own network as the egress of its server-side service.
The client holds a WebSocket open on ``/api/tunnel`` before it creates its remote session. The server gives the
service a SOCKS5 proxy on loopback, and sends every connection that proxy accepts to the client, which
dials it from its own connection and relays the bytes. Only the client dials. TLS stays end to end between
the server and the service, so the client sees only ciphertext.
"""
from __future__ import annotations
import asyncio
import ipaddress
import itertools
import json
import logging
import secrets
import socket
import struct
import threading
from typing import Any, Optional
import aiohttp
from aiohttp import WSMsgType, web
log = logging.getLogger("tunnel")
OPEN, OK, DATA, CLOSE = range(4)
HEADER = struct.Struct("!IB") # stream id, frame kind
HEARTBEAT = 20
GRACE = 60
RELINK_WAIT = 10
OPEN_TIMEOUT = 15
MAX_TUNNELS = 4
MAX_STREAMS = 256
CHUNK = 65536
BACKLOG = 64
MAX_MESSAGE = 4 * CHUNK
ROUTE = "/api/tunnel"
CHECK_ROUTE = "/api/tunnel/check"
def frame(sid: int, kind: int, data: bytes = b"") -> bytes:
return HEADER.pack(sid, kind) + data
def new_queue() -> asyncio.Queue[Optional[bytes]]:
return asyncio.Queue(maxsize=BACKLOG + 1) # the extra slot always takes the end marker
class Link:
"""One WebSocket that carries many TCP connections, each with its own id."""
def __init__(self, ws: Any) -> None:
self.ws = ws
self.lock = asyncio.Lock()
self.streams: dict[int, asyncio.Queue[Optional[bytes]]] = {}
async def send(self, sid: int, kind: int, data: bytes = b"") -> None:
async with self.lock:
await self.ws.send_bytes(frame(sid, kind, data))
async def pump(self, sid: int, reader: asyncio.StreamReader) -> None:
"""Copy a socket into the WebSocket until either side closes."""
try:
while chunk := await reader.read(CHUNK):
await self.send(sid, DATA, chunk)
except (ConnectionError, RuntimeError):
pass
finally:
await self.close_stream(sid) # the peer does not echo CLOSE, so this also ends our drain
async def drain(self, q: asyncio.Queue[Optional[bytes]], writer: asyncio.StreamWriter) -> None:
"""Copy the bytes that the peer sends for one connection into a socket, then close the socket."""
try:
while (chunk := await q.get()) is not None:
writer.write(chunk)
await writer.drain()
except ConnectionError:
pass
finally:
writer.close()
def end(self, sid: int) -> bool:
"""End one connection's drain; False when the connection was already gone."""
if (q := self.streams.pop(sid, None)) is None:
return False
q.put_nowait(None)
return True
async def close_stream(self, sid: int) -> None:
"""End one connection here and tell the peer."""
if self.end(sid) and not self.ws.closed:
await self.send(sid, CLOSE)
async def feed(self, sid: int, body: bytes) -> None:
"""Queue bytes for one connection; drop a connection whose socket has stopped reading them."""
q = self.streams.get(sid)
if q is None:
return
if q.qsize() < BACKLOG:
q.put_nowait(body)
else:
await self.close_stream(sid)
def end_all(self) -> None:
for sid in list(self.streams):
self.end(sid)
async def relay(
self, sid: int, q: asyncio.Queue[Optional[bytes]], reader: asyncio.StreamReader, writer: asyncio.StreamWriter
) -> None:
await asyncio.gather(self.pump(sid, reader), self.drain(q, writer))
TUNNELS: dict[str, "Tunnel"] = {}
class Tunnel:
"""The server side of one client's tunnel: a loopback SOCKS5 listener and the client's current link."""
def __init__(self, owner: Optional[str]) -> None:
self.id = secrets.token_urlsafe(16)
self.secret = secrets.token_urlsafe(24)
self.owner = owner
self.link: Optional[Link] = None
self.pending: dict[int, asyncio.Future[bool]] = {}
self.sids = itertools.count(1)
self.listener: Optional[asyncio.Server] = None
self.expiry: Optional[asyncio.TimerHandle] = None
self.linked = asyncio.Event()
self.closing: Optional[asyncio.Task[None]] = None
@property
def port(self) -> int:
assert self.listener is not None
return int(self.listener.sockets[0].getsockname()[1])
@property
def proxy_uri(self) -> str:
# without credentials, another API key could use this port as a raw socks5h://127.0.0.1 proxy
return f"socks5h://{self.id}:{self.secret}@127.0.0.1:{self.port}"
async def start(self) -> None:
self.listener = await asyncio.start_server(self.accept, "127.0.0.1", 0)
TUNNELS[self.id] = self
log.info(f"Tunnel {self.id[:6]} opened on 127.0.0.1:{self.port}")
async def close(self) -> None:
if TUNNELS.pop(self.id, None):
log.info(f"Tunnel {self.id[:6]} closed")
if self.expiry:
self.expiry.cancel()
if self.listener:
self.listener.close()
await self.detach()
async def detach(self) -> None:
link, self.link = self.link, None
self.linked.clear()
for fut in self.pending.values():
fut.cancel()
self.pending.clear()
if link:
link.end_all()
await link.ws.close()
async def on_frame(self, link: Link, data: bytes) -> None:
"""Act on a frame from the client. An OPEN from the client is ignored: the server never dials for it."""
if len(data) < HEADER.size:
return
sid, kind = HEADER.unpack_from(data)
if kind == OK and (fut := self.pending.pop(sid, None)) and not fut.done():
fut.set_result(True)
elif kind == DATA:
await link.feed(sid, data[HEADER.size :])
elif kind == CLOSE:
if (fut := self.pending.pop(sid, None)) and not fut.done():
fut.set_result(False)
link.end(sid)
async def wait_link(self) -> Optional[Link]:
"""The current link, waiting ``RELINK_WAIT`` seconds for a dropped one to come back."""
if not self.link:
try:
await asyncio.wait_for(self.linked.wait(), RELINK_WAIT)
except asyncio.TimeoutError:
pass
return self.link
async def accept(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
"""Take one SOCKS5 CONNECT with this tunnel's username and password, and open it on the client."""
def refuse(answer: bytes) -> None:
writer.write(answer)
writer.close()
def reply(code: int) -> bytes:
return bytes((5, code, 0, 1)) + bytes(6)
try:
version, methods = await reader.readexactly(2)
if version != 5 or 2 not in await reader.readexactly(methods):
return refuse(b"\x05\xff")
writer.write(b"\x05\x02")
auth_version = (await reader.readexactly(1))[0]
user = await reader.readexactly((await reader.readexactly(1))[0])
password = await reader.readexactly((await reader.readexactly(1))[0])
if not (
auth_version == 1
and secrets.compare_digest(user, self.id.encode())
and secrets.compare_digest(password, self.secret.encode())
):
return refuse(b"\x01\x01")
writer.write(b"\x01\x00")
_, command, _, address_type = await reader.readexactly(4)
if address_type == 1:
host = socket.inet_ntop(socket.AF_INET, await reader.readexactly(4))
elif address_type == 4:
host = socket.inet_ntop(socket.AF_INET6, await reader.readexactly(16))
elif address_type == 3:
host = (await reader.readexactly((await reader.readexactly(1))[0])).decode()
else:
return refuse(reply(8)) # address type not supported
port = await reader.readexactly(2)
except (asyncio.IncompleteReadError, ConnectionError, UnicodeDecodeError):
writer.close()
return
if command != 1:
return refuse(reply(7)) # command not supported
link = await self.wait_link()
if not link:
return refuse(reply(5))
sid = next(self.sids)
fut = self.pending[sid] = asyncio.get_running_loop().create_future()
q = link.streams[sid] = new_queue()
ok = False
try:
await link.send(sid, OPEN, port + host.encode())
ok = await asyncio.wait_for(fut, OPEN_TIMEOUT)
except (asyncio.TimeoutError, asyncio.CancelledError, ConnectionError, RuntimeError):
pass
if not ok:
self.pending.pop(sid, None)
await link.close_stream(sid) # a client still dialling would relay into a dead stream
return refuse(reply(5)) # connection refused
writer.write(reply(0))
await link.relay(sid, q, reader, writer)
async def serve(self, ws: web.WebSocketResponse) -> None:
"""Make ``ws`` the tunnel's link until it closes, then wait ``GRACE`` seconds for a reconnect."""
if self.expiry:
self.expiry.cancel()
self.expiry = None
await self.detach()
link = self.link = Link(ws)
try:
await ws.send_str(json.dumps({"id": self.id}))
self.linked.set()
async for msg in ws:
if msg.type == WSMsgType.BINARY:
await self.on_frame(link, msg.data)
finally:
if self.link is link:
self.link = None
self.linked.clear()
link.end_all()
if self.id in TUNNELS:
self.expiry = asyncio.get_running_loop().call_later(GRACE, self.expire)
def expire(self) -> None:
self.closing = asyncio.ensure_future(self.close()) # held, so the loop cannot drop the task
def tunnel_proxy(tunnel_id: str, owner: Optional[str]) -> Optional[str]:
"""The loopback proxy URI for the tunnel ``tunnel_id`` that the caller owns, or None."""
tunnel = TUNNELS.get(tunnel_id)
return tunnel.proxy_uri if tunnel and tunnel.owner == owner else None
async def close_tunnels(_: web.Application) -> None:
for tunnel in list(TUNNELS.values()):
await tunnel.close()
def is_public_ip(address: str) -> bool:
try:
return ipaddress.ip_address(address.partition("%")[0]).is_global
except ValueError:
return False
async def connect_public(host: str, port: int) -> tuple[asyncio.StreamReader, asyncio.StreamWriter]:
"""Connect to the first public address of ``host``. The server picks the targets, so a name that resolves
only to the client's own machine or LAN is refused."""
infos = await asyncio.get_running_loop().getaddrinfo(host, port, type=socket.SOCK_STREAM)
addresses = [info[4][0] for info in infos if is_public_ip(str(info[4][0]))]
if not addresses:
raise PermissionError(f"{host} resolves only to local addresses")
error: OSError = OSError(f"could not connect to {host}")
for address in addresses:
try:
return await asyncio.wait_for(asyncio.open_connection(address, port), 10)
except (OSError, asyncio.TimeoutError) as e:
error = OSError(str(e))
raise error
class TunnelClient:
"""The client side: a background thread that keeps the tunnel's WebSocket open and dials for the server."""
def __init__(self, server_url: str, headers: dict[str, str]) -> None:
self.url = server_url.rstrip("/") + ROUTE
self.headers = headers
self.id: Optional[str] = None
self.error: Optional[BaseException] = None
self.lost = False
self.ready = threading.Event()
self.relays: set[asyncio.Task[None]] = set()
self.thread = threading.Thread(target=lambda: asyncio.run(self.run()), name="tunnel", daemon=True)
self.loop: Optional[asyncio.AbstractEventLoop] = None
self.task: Optional[asyncio.Task[None]] = None
def start(self, timeout: float = 20) -> str:
"""Open the tunnel and return its id."""
self.thread.start()
if not self.ready.wait(timeout) or not self.id:
self.stop() # or the thread opens the tunnel later and nobody uses it
raise ConnectionError(f"Could not open a tunnel to the server: {self.error or 'timed out'}")
return self.id
def stop(self) -> None:
"""Close the tunnel and wait for its connections to end, so none is left pending at exit."""
if self.loop and self.task and self.thread.is_alive():
self.loop.call_soon_threadsafe(self.task.cancel)
self.thread.join(5)
async def run(self) -> None:
self.loop, self.task = asyncio.get_running_loop(), asyncio.current_task()
try:
await self.connect()
except asyncio.CancelledError:
pass
finally:
for t in self.relays:
t.cancel()
await asyncio.gather(*self.relays, return_exceptions=True)
async def connect(self) -> None:
backoff = 1
async with aiohttp.ClientSession(headers=self.headers) as http:
while True:
try:
url = f"{self.url}?id={self.id}" if self.id else self.url
async with http.ws_connect(url, heartbeat=HEARTBEAT, max_msg_size=MAX_MESSAGE) as ws:
hello = await ws.receive_json(timeout=15)
self.id = hello["id"]
self.ready.set()
backoff = 1
await self.serve(Link(ws))
except aiohttp.WSServerHandshakeError as e:
self.error = e
# a 404 means the tunnel expired or the server has no tunnel route; a 5xx is a proxy in between
if not self.id or e.status < 500:
if self.id:
self.lost = True
log.warning(f"The server closed the tunnel (HTTP {e.status}), so its requests now fail")
self.ready.set()
return
except (aiohttp.ClientError, asyncio.TimeoutError, ValueError, KeyError, TypeError) as e:
self.error = e
if not self.id:
self.ready.set()
return
log.warning(f"The tunnel to the server dropped, reconnecting in {backoff}s")
await asyncio.sleep(backoff)
backoff = min(backoff * 2, 30)
async def serve(self, link: Link) -> None:
try:
async for msg in link.ws:
if msg.type != WSMsgType.BINARY or len(msg.data) < HEADER.size:
continue
sid, kind = HEADER.unpack_from(msg.data)
body = msg.data[HEADER.size :]
if kind == OPEN:
if len(link.streams) >= MAX_STREAMS or sid in link.streams or len(body) < 3:
await link.send(sid, CLOSE)
continue
link.streams[sid] = new_queue()
host = body[2:].decode(errors="replace")
task = asyncio.ensure_future(self.open(link, sid, host, int.from_bytes(body[:2], "big")))
self.relays.add(task)
task.add_done_callback(self.relays.discard)
elif kind == DATA:
await link.feed(sid, body)
elif kind == CLOSE:
link.end(sid)
finally:
link.end_all()
async def open(self, link: Link, sid: int, host: str, port: int) -> None:
try:
reader, writer = await connect_public(host, port)
except (OSError, ValueError, asyncio.TimeoutError) as e: # the idna codec raises ValueError on a bad hostname
log.debug(f"Tunnel dial to {host}:{port} failed: {e}")
await link.close_stream(sid)
return
if (q := link.streams.get(sid)) is None: # the server gave up while we dialled
writer.close()
return
await link.send(sid, OK)
await link.relay(sid, q, reader, writer)