diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index 4106ce8b..7fc7f0f2 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -207,7 +207,7 @@ order the service used, then renumbers the episodes into the order you asked for | Flag | Description | |---|---| | `--proxy` | Proxy URI, a country or location code resolved from configured providers, or `provider:region` (e.g. `nordvpn:ca`, `gluetun:us`, `protonvpn:de:berlin`, `controld:yul`). | -| `--local-proxy` | With `--remote`, send the server's service requests through your machine's own connection. Works with no other proxy option. Check it first with `unshackle tunnel-test`. See [Proxies & VPN](proxies-and-vpn.md#send-the-servers-requests-through-your-machine). | +| `--local-proxy` | With `--remote`, send the server's service requests through your machine's own connection, or through a `--proxy` that your machine connects to. Check it first with `unshackle tunnel-test`. See [Proxies & VPN](proxies-and-vpn.md#send-the-servers-requests-through-your-machine). | | `--no-proxy` | Force-disable all proxy use. | | `--no-proxy-download` | Bypass the proxy for **all downloads** (manifest, licence, and auth stay proxied). | | `--proxy-download` | Use a different proxy for **all downloads**, in the same forms as `--proxy` (manifest, licence, and auth stay on `--proxy`). | @@ -251,7 +251,7 @@ unshackle logs in to the service before the search. A service that declares a se |---|---| | `-p`, `--profile` | Profile for credentials and cookies. | | `--proxy` | Proxy URI, country or location code, or `provider:region` (e.g. `controld:ca`). | -| `--local-proxy` | With `--remote`, send the server's search requests through your machine's own connection. Works with no other proxy option. | +| `--local-proxy` | With `--remote`, send the server's search requests through your machine's own connection, or through a `--proxy` that your machine connects to. | | `--no-proxy` | Force-disable all proxy use. | | `--remote` | Search on a remote unshackle server. The server logs in to the service. unshackle sends it your credentials, cookies and cache for the profile, and saves the cache that the server sends back. A service that searches with no login gets none of them. | | `--local` | Search with the local service code, also when the config sets `remote: true` for the service. | @@ -294,12 +294,13 @@ machine exits from, opens a tunnel to the server, and makes the server look up i through the tunnel. It stops at the first step that fails. ``` -unshackle tunnel-test [--server NAME] +unshackle tunnel-test [--server NAME] [--proxy VALUE] ``` | Flag | Description | |---|---| | `--server` | Name of the remote server from `remote_services` config, if you configure more than one. | +| `--proxy` | Test the route through this proxy, as `--local-proxy --proxy` uses it. Takes any `--proxy` value. | See [Proxies & VPN](proxies-and-vpn.md#send-the-servers-requests-through-your-machine). diff --git a/docs/guide/proxies-and-vpn.md b/docs/guide/proxies-and-vpn.md index 74bd3072..7a10dd92 100644 --- a/docs/guide/proxies-and-vpn.md +++ b/docs/guide/proxies-and-vpn.md @@ -27,7 +27,7 @@ Five options on the `dl` command control proxying at download time: | Option | Effect | |---|---| | `--proxy` | The proxy to use. Either an explicit URI, or a query that unshackle resolves against your configured providers. | -| `--local-proxy` | With `--remote`, send the server's service requests through your machine's own connection. Works with no other proxy option. See [Send the server's requests through your machine](#send-the-servers-requests-through-your-machine). | +| `--local-proxy` | With `--remote`, send the server's service requests through your machine's own connection, or through a `--proxy` that your machine connects to. See [Send the server's requests through your machine](#send-the-servers-requests-through-your-machine). | | `--no-proxy` | Force **all** proxy use off. No providers are initialised and no proxy query is resolved. | | `--no-proxy-download` | Use the proxy for the manifest, licence, and authentication, but bypass it for the **downloads** themselves. | | `--proxy-download` | Use `--proxy` for the manifest, licence, and authentication, and a different proxy for the **downloads**. Takes the same forms as `--proxy`. | @@ -273,18 +273,31 @@ unshackle dl --remote --local-proxy EXAMPLE 81234567 A VPN app that covers your whole machine then also covers the server's requests, so the service sees your VPN's IP address. `--local-proxy` needs `--remote` (or a service with -`remote: true`), and it works with no other proxy option: unshackle stops if you also give -`--proxy`, `--no-proxy`, `--proxy-download` or `--no-proxy-download`. A proxy option from the `dl` -config defaults does not count: `--local-proxy` turns it off for the run. +`remote: true`). unshackle stops if you also give `--no-proxy`, `--proxy-download` or +`--no-proxy-download`. A proxy option from the `dl` config defaults does not count: +`--local-proxy` turns it off for the run. -A `--proxy` on `localhost` or a loopback address, such as the local port of a VPN app, stops a -`--remote` run with `That proxy is on your machine, so --remote cannot use it`. Connect the VPN -app for the whole machine, then use `--local-proxy` without `--proxy`. +Add `--proxy` to send the server's requests through a proxy that your machine connects to. The +proxy can be on your machine, such as the local port of a VPN app or `hola-proxy`, or any value +that `--proxy` takes, such as `ca` or `nordvpn:us`: + +```shell +unshackle dl --remote --local-proxy --proxy http://127.0.0.1:8080 EXAMPLE 81234567 +``` + +The proxy must be `http://`, `https://`, `socks5://` or `socks5h://`, with a username and +password in the URI if it needs them. If the proxy fails, the request fails: unshackle never +sends it from your own connection instead. Your machine still refuses a target on your own +machine or network, and a host name that your machine resolves to a local address only. + +Without `--local-proxy`, a `--proxy` on `localhost` or a loopback address stops a `--remote` run +with `--remote cannot use a proxy on this machine without --local-proxy`, because the server +cannot reach it. #### Test it first `unshackle tunnel-test` checks the route before a real run. Add `--server ` to test a -server other than the default one: +server other than the default one, and `--proxy ` to test the route through a proxy: ```shell unshackle tunnel-test @@ -292,7 +305,7 @@ unshackle tunnel-test It does three steps, and it stops at the first one that fails: -1. It looks up the IP address that your machine exits from. +1. It looks up the IP address that your machine exits from, through `--proxy` if you give one. 2. It opens a tunnel to the server. 3. It makes the server look up its exit IP address through the tunnel. That address must be the same as in step 1, or in the same country when one lookup used IPv4 and the other IPv6. diff --git a/tests/remote/unit/test_remote_search.py b/tests/remote/unit/test_remote_search.py index accc7910..d5a7a65b 100644 --- a/tests/remote/unit/test_remote_search.py +++ b/tests/remote/unit/test_remote_search.py @@ -73,10 +73,12 @@ def test_search_sends_no_login(server_accounts: Any, anonymous: bool) -> None: assert client.sent == {"endpoint": "/api/search", "service": "TEST", "query": "query"} -def test_local_proxy_sends_the_server_a_tunnel(monkeypatch: pytest.MonkeyPatch) -> None: +@pytest.mark.parametrize("proxy", [None, "http://127.0.0.1:8080"]) +def test_local_proxy_sends_the_server_a_tunnel(monkeypatch: pytest.MonkeyPatch, proxy: str | None) -> None: class FakeTunnel: - def __init__(self, server_url: str, headers: dict) -> None: + def __init__(self, server_url: str, headers: dict, exit_proxy: str | None = None) -> None: self.id = "abc" + self.exit_proxy = exit_proxy def start(self) -> str: return self.id @@ -85,13 +87,14 @@ def test_local_proxy_sends_the_server_a_tunnel(monkeypatch: pytest.MonkeyPatch) self.stopped = True monkeypatch.setattr(remote_service, "TunnelClient", FakeTunnel) - svc, client, _ = make_service({"local_proxy": True}) + svc, client, _ = make_service({"local_proxy": True, "proxy": proxy}) svc.log = logging.getLogger("test") client.session = SimpleNamespace(headers={"X-Secret-Key": "k"}) # type: ignore[attr-defined] list(svc.search()) assert client.sent["proxy"] == "tunnel:abc" + assert client.tunnel.exit_proxy == proxy assert client.tunnel.stopped @@ -101,7 +104,7 @@ def test_a_loopback_proxy_points_to_local_proxy(monkeypatch: pytest.MonkeyPatch) monkeypatch.setattr(remote_service, "resolve_remote_proxy_arg", lambda arg, providers=None: arg) svc, _, _ = make_service({"proxy": "http://127.0.0.1:8888"}, {"global": True}) - with pytest.raises(click.ClickException, match="--local-proxy without --proxy"): + with pytest.raises(click.ClickException, match="without --local-proxy"): list(svc.search()) diff --git a/tests/remote/unit/test_tunnel.py b/tests/remote/unit/test_tunnel.py index ec4acf1f..4217ca3b 100644 --- a/tests/remote/unit/test_tunnel.py +++ b/tests/remote/unit/test_tunnel.py @@ -21,12 +21,14 @@ from multidict import CIMultiDict from unshackle.core import tunnel from unshackle.core.api.routes import tunnel as tunnel_route from unshackle.core.api.routes import tunnel_check +from unshackle.core.proxies.controld import relay from unshackle.core.tunnel import ( CHECK_ROUTE, OPEN, ROUTE, TUNNELS, TunnelClient, + connect_proxied, connect_public, frame, is_ip_literal, @@ -258,13 +260,16 @@ def local_proxy_ctx(params: dict, typed: set) -> click.Context: return ctx -def test_local_proxy_works_with_no_other_proxy_option() -> None: +def test_local_proxy_works_with_a_typed_proxy_and_no_other_proxy_option() -> None: from unshackle.core.proxies.resolve import require_alone_local_proxy - with pytest.raises(click.UsageError, match="--proxy, --proxy-download"): + with pytest.raises(click.UsageError, match="with --proxy-download\\.$"): require_alone_local_proxy( local_proxy_ctx({"remote": True, "proxy": "us", "proxy_download": "us"}, {"proxy", "proxy_download"}) ) + ctx = local_proxy_ctx({"remote": True, "proxy": "http://127.0.0.1:8080"}, {"proxy"}) + require_alone_local_proxy(ctx) + assert ctx.params["proxy"] == "http://127.0.0.1:8080" with pytest.raises(click.UsageError, match="needs --remote"): require_alone_local_proxy(local_proxy_ctx({"remote": False}, set())) @@ -612,3 +617,120 @@ def test_a_failed_start_leaves_no_thread_behind() -> None: assert not client.thread.is_alive() finally: silent.close() + + +class ExitProxy(socketserver.BaseRequestHandler): + """An exit proxy on this machine: takes one HTTP CONNECT or SOCKS5 CONNECT and pipes it to the test target.""" + + server: ExitServer + + def handle(self) -> None: + sock, seen = self.request, self.server.seen + stream = sock.makefile("rb") + if stream.peek(1)[:1] == b"\x05": + stream.read(stream.read(2)[1]) + sock.sendall(b"\x05\x02") + stream.read(1) + user = stream.read(stream.read(1)[0]) + seen.append(user + b":" + stream.read(stream.read(1)[0])) + sock.sendall(b"\x01\x00") + address_type = stream.read(4)[3] + seen.append(stream.read(stream.read(1)[0] if address_type == 3 else 4 if address_type == 1 else 16)) + stream.read(2) + sock.sendall(b"\x05\x00\x00\x01" + bytes(6)) + else: + head = b"" + while not head.endswith(b"\r\n\r\n"): + head += stream.readline() + seen.append(head) + sock.sendall(b"HTTP/1.1 " + self.server.status + b"\r\n\r\n") + if not self.server.status.startswith(b"200"): + return + upstream = socket.create_connection(("127.0.0.1", self.server.target)) + threading.Thread(target=relay, args=(upstream, sock), daemon=True).start() + relay(sock, upstream) + + +class ExitServer(socketserver.ThreadingTCPServer): + daemon_threads = True + seen: list[bytes] + status: bytes + target: int + + +@pytest.fixture +def exit_proxy(target: int) -> Iterator[ExitServer]: + proxy = ExitServer(("127.0.0.1", 0), ExitProxy) + proxy.seen, proxy.status, proxy.target = [], b"200 Connection established", target + threading.Thread(target=proxy.serve_forever, daemon=True).start() + yield proxy + proxy.shutdown() + + +@pytest.mark.parametrize("scheme", ["http", "socks5h"]) +def test_the_tunnel_dials_through_an_exit_proxy_with_its_login( + server: str, exit_proxy: ExitServer, scheme: str +) -> None: + port = exit_proxy.server_address[1] + client = TunnelClient(server, {"X-Secret-Key": "alice"}, f"{scheme}://us%40er:p%3Ass@127.0.0.1:{port}") + proxy = tunnel_proxy(client.start(), "alice") + resp = requests.get("http://hello.example/", proxies={"all": proxy}, timeout=10) + client.stop() + + assert resp.text == "hello through the tunnel" + if scheme == "http": + assert exit_proxy.seen[0].startswith(b"CONNECT hello.example:80 HTTP/1.1\r\n") + assert b"Proxy-Authorization: Basic dXNAZXI6cDpzcw==\r\n" in exit_proxy.seen[0] + else: + assert exit_proxy.seen == [b"us@er:p:ss", b"hello.example"] + + +@pytest.mark.parametrize( + ("host", "address"), + [ + ("127.0.0.1", None), + ("::1", None), + ("10.1", None), + ("localhost", None), + ("printer.localhost.", None), + ("bad.example\r\nX-Injected: 1", None), + ("bad example", None), + ("rebind.example", "192.168.1.1"), + ("metadata.example", "169.254.169.254"), + ], +) +def test_the_exit_proxy_never_carries_a_local_or_malformed_target( + monkeypatch: pytest.MonkeyPatch, exit_proxy: ExitServer, host: str, address: str | None +) -> None: + async def resolve(self: object, host: str, port: int, **_: object) -> list: + return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", (address, port))] + + if address: + monkeypatch.setattr(asyncio.BaseEventLoop, "getaddrinfo", resolve) + with pytest.raises(PermissionError): + asyncio.run(connect_proxied(f"http://127.0.0.1:{exit_proxy.server_address[1]}", host, 80)) + assert exit_proxy.seen == [] + + +def test_a_refused_connect_fails_and_never_dials_direct( + server: str, target: int, exit_proxy: ExitServer, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(tunnel, "is_public_ip", lambda address: True) # so a direct dial to the target would work + monkeypatch.setattr(tunnel, "dialled_itself", lambda writer: False) + exit_proxy.status = b"407 Proxy Authentication Required" + hits = Hello.hits + client = TunnelClient(server, {"X-Secret-Key": "alice"}, f"http://127.0.0.1:{exit_proxy.server_address[1]}") + proxy = tunnel_proxy(client.start(), "alice") + with pytest.raises(requests.exceptions.ConnectionError): + requests.get(f"http://127.0.0.1:{target}/", proxies={"all": proxy}, timeout=10) + client.stop() + assert len(exit_proxy.seen) == 1 and Hello.hits == hits + with pytest.raises(OSError, match="407"): + asyncio.run(connect_proxied(f"http://127.0.0.1:{exit_proxy.server_address[1]}", "hello.example", 80)) + + +def test_a_proxy_the_tunnel_cannot_dial_through_is_refused_at_start() -> None: + from unshackle.core.remote_service import start_tunnel + + with pytest.raises(click.ClickException, match="cannot use a socks4 proxy"): + start_tunnel(None, "socks4://127.0.0.1:1080") # type: ignore[arg-type] diff --git a/unshackle/commands/dl.py b/unshackle/commands/dl.py index a141c4bb..46c63f54 100644 --- a/unshackle/commands/dl.py +++ b/unshackle/commands/dl.py @@ -936,7 +936,7 @@ class dl: "--local-proxy", is_flag=True, default=False, - help="With --remote, the server sends its service requests out through this machine's own connection (and any VPN on it). Works with no other proxy option.", + help="With --remote, the server sends its service requests out through this machine's own connection (and any VPN on it), or through a --proxy dialled from this machine.", ) @click.option( "--tag", @@ -1570,14 +1570,18 @@ class dl: proxy, no_proxy = ctx.params.get("proxy"), bool(ctx.params.get("no_proxy")) # A server on this machine can reach a local proxy, so only a server elsewhere is guarded - self.remote_needs_public_proxy = self.is_remote and not is_loopback(server_url(ctx.params.get("server"))) + self.remote_needs_public_proxy = ( + self.is_remote + and not ctx.params.get("local_proxy") + and not is_loopback(server_url(ctx.params.get("server"))) + ) self.proxy_providers = [] if no_proxy: ctx.params["proxy"] = None else: if self.remote_needs_public_proxy and proxy and proxy.lower().startswith("gluetun:"): - raise click.UsageError("Gluetun runs on your machine, so --remote cannot use it.") + raise click.UsageError("Gluetun runs on your machine, so --remote needs --local-proxy to use it.") if proxy_providers is not None: self.proxy_providers = list(proxy_providers) else: diff --git a/unshackle/commands/search.py b/unshackle/commands/search.py index 1701f2f4..a9d36251 100644 --- a/unshackle/commands/search.py +++ b/unshackle/commands/search.py @@ -48,7 +48,7 @@ from unshackle.core.utils.collections import ci_get, merge_dict "--local-proxy", is_flag=True, default=False, - help="With --remote, the server sends its service requests out through this machine's own connection (and any VPN on it). Works with no other proxy option.", + help="With --remote, the server sends its service requests out through this machine's own connection (and any VPN on it), or through a --proxy dialled from this machine.", ) @click.option("--no-proxy", is_flag=True, default=False, help="Force disable all proxy use.") @click.option( diff --git a/unshackle/commands/tunnel_test.py b/unshackle/commands/tunnel_test.py index e6d31fdb..56e1d889 100644 --- a/unshackle/commands/tunnel_test.py +++ b/unshackle/commands/tunnel_test.py @@ -2,9 +2,12 @@ import logging from typing import Optional import click +import requests -from unshackle.core.remote_service import remote_client, start_tunnel +from unshackle.core.proxies.resolve import load_proxy_providers +from unshackle.core.remote_service import remote_client, resolve_proxy_arg, start_tunnel from unshackle.core.tunnel import CHECK_ROUTE +from unshackle.core.utils import run_proxy from unshackle.core.utils.ip_info import get_ip_info from unshackle.core.utils.redact import safe_display_url @@ -13,21 +16,30 @@ from unshackle.core.utils.redact import safe_display_url @click.option( "--server", type=str, default=None, help="Name of the remote server from remote_services config (if multiple)." ) -def tunnel_test(server: Optional[str]) -> None: +@click.option( + "--proxy", type=str, default=None, help="Check the tunnel through this proxy, as --local-proxy --proxy does." +) +def tunnel_test(server: Optional[str], proxy: Optional[str]) -> None: """Check --local-proxy before a real --remote run. - The command finds the exit IP of this machine, opens a tunnel to the server, and makes the server - look up its exit IP through the tunnel. The two must match. + The command finds the exit IP of this machine (through --proxy when given), opens a tunnel to the server, + and makes the server look up its exit IP through the tunnel. The two must match. """ log = logging.getLogger("tunnel-test") - local = get_ip_info() + proxy = resolve_proxy_arg(proxy, load_proxy_providers()) if proxy else None + session = None + if proxy: + session = requests.Session() + session.proxies["all"] = proxy + run_proxy.pin(session) + local = get_ip_info(session) if not local: raise click.ClickException("Step 1 failed: an IP lookup from this machine did not work.") log.info(f"Step 1: this machine exits at {local['ip']} ({str(local.get('country', '?')).upper()})") client = remote_client(server) - tunnel = start_tunnel(client) + tunnel = start_tunnel(client, proxy) log.info(f"Step 2: opened a tunnel to {safe_display_url(client.server_url)}") try: remote = client.get(f"{CHECK_ROUTE}?id={tunnel.id}") diff --git a/unshackle/core/proxies/resolve.py b/unshackle/core/proxies/resolve.py index 0cbd588d..a1efb2c5 100644 --- a/unshackle/core/proxies/resolve.py +++ b/unshackle/core/proxies/resolve.py @@ -207,9 +207,10 @@ def describe_proxy(provider: Any, uri: str, query: str, allow_debug: bool = True def require_alone_local_proxy(ctx: click.Context) -> None: - """Check ``--local-proxy``: it needs a remote run and no other proxy option. + """Check ``--local-proxy``: it needs a remote run, and takes no proxy option but a typed ``--proxy``. - A proxy option that comes from the ``dl`` config defaults, not the command line, is cleared instead. + A typed ``--proxy`` becomes the tunnel's exit on this machine. A proxy option that comes from the ``dl`` + config defaults, not the command line, is cleared instead. """ if not ctx.params.get("remote"): raise click.UsageError( @@ -221,7 +222,7 @@ def require_alone_local_proxy(ctx: click.Context) -> None: continue if ctx.get_parameter_source(key) in (ParameterSource.DEFAULT, ParameterSource.DEFAULT_MAP): ctx.params[key] = False if key.startswith("no_") else None - else: + elif key != "proxy": given.append("--" + key.replace("_", "-")) if given: raise click.UsageError(f"--local-proxy does not work with {', '.join(given)}.") diff --git a/unshackle/core/remote_service.py b/unshackle/core/remote_service.py index 1bf8d984..f514482e 100644 --- a/unshackle/core/remote_service.py +++ b/unshackle/core/remote_service.py @@ -26,7 +26,7 @@ from http.cookiejar import CookieJar from pathlib import Path from threading import Event, Lock, Thread, local from typing import Any, Callable, Dict, Iterable, Iterator, Optional, Protocol, TypeVar, Union -from urllib.parse import urlencode +from urllib.parse import urlencode, urlsplit from uuid import UUID import click @@ -54,7 +54,7 @@ from unshackle.core.titles.music import SONG_FIELDS, Album, Song from unshackle.core.tracks import Audio, Chapter, Chapters, Subtitle, Tracks, Video from unshackle.core.tracks.attachment import Attachment from unshackle.core.tracks.track import Track -from unshackle.core.tunnel import TunnelClient, parse_retry_after +from unshackle.core.tunnel import PROXY_PORTS, TunnelClient, parse_retry_after from unshackle.core.utils import run_proxy from unshackle.core.utils.collections import ci_get from unshackle.core.utils.redact import redact_path, redact_text, safe_display_url @@ -819,15 +819,14 @@ def load_cookies_for_transport(service_tag: str, profile: Optional[str]) -> Opti return None -def resolve_proxy_arg(proxy_arg: Optional[str]) -> Optional[str]: +def resolve_proxy_arg(proxy_arg: Optional[str], providers: Optional[list] = None) -> Optional[str]: if not proxy_arg: return None from unshackle.core.proxies.resolve import initialize_proxy_providers, resolve_proxy try: - providers = initialize_proxy_providers() - return resolve_proxy(proxy_arg, providers) + return resolve_proxy(proxy_arg, initialize_proxy_providers() if providers is None else providers) except ValueError as e: raise click.ClickException(str(e)) @@ -860,13 +859,16 @@ def resolve_remote_proxy_arg(proxy_arg: str, providers: Optional[list] = None) - raise click.ClickException(str(e)) -def start_tunnel(client: RemoteClient) -> TunnelClient: - """Open a tunnel that sends the server's requests out through this machine's own connection.""" +def start_tunnel(client: RemoteClient, proxy: Optional[str] = None) -> TunnelClient: + """Open a tunnel that sends the server's requests out through this machine's own connection, or through + ``proxy`` from this machine.""" + if proxy and (scheme := urlsplit(proxy).scheme.lower()) not in PROXY_PORTS: + raise click.ClickException(f"--local-proxy cannot use a {scheme} proxy") headers = {k: str(v) for k, v in client.session.headers.items() if k.lower() != "connection"} slot_deadline = time.monotonic() + RETRY_AFTER_MAX rate_waits = 0 while True: - tunnel = TunnelClient(client.server_url, headers) + tunnel = TunnelClient(client.server_url, headers, proxy) try: tunnel.start() break @@ -1190,7 +1192,7 @@ class RemoteService: def require_reachable_proxy(self, proxy: Optional[str]) -> None: """Refuse a proxy on this machine when the server is elsewhere: the server cannot reach it.""" if proxy and is_loopback(proxy) and not is_loopback(self.client.server_url): - raise click.ClickException("--remote cannot use a proxy on this machine; use --local-proxy without --proxy") + raise click.ClickException("--remote cannot use a proxy on this machine without --local-proxy") def tunnel(self) -> str: """The ``tunnel:`` proxy to send the server for ``--local-proxy``, opening the tunnel the first time.""" @@ -1200,8 +1202,11 @@ class RemoteService: ) tunnel = self.client.tunnel if tunnel is None: - tunnel = start_tunnel(self.client) - self.log.info("The server sends its requests through your connection on this machine") + proxy = self.ctx.parent.params.get("proxy") if self.ctx.parent else None + tunnel = start_tunnel(self.client, proxy) + self.log.info( + f"The server sends its requests through your {'proxy' if proxy else 'connection'} on this machine" + ) return f"tunnel:{tunnel.id}" def authenticate(self, cookies: Optional[CookieJar] = None, credential: Optional[Credential] = None) -> None: @@ -1256,7 +1261,7 @@ class RemoteService: if country: create_data["proxy_region"] = country.group(1).lower() self.require_reachable_proxy(create_data.get("proxy")) - local_proxy = None if no_proxy or local else (proxy or create_data.get("proxy")) + local_proxy = None if no_proxy else proxy if local else proxy or create_data.get("proxy") if local_proxy: self._session.proxies["all"] = local_proxy run_proxy.pin(self._session) diff --git a/unshackle/core/tunnel.py b/unshackle/core/tunnel.py index 46fb9e0e..7a61a233 100644 --- a/unshackle/core/tunnel.py +++ b/unshackle/core/tunnel.py @@ -9,22 +9,32 @@ the server and the service, so the client sees only ciphertext. from __future__ import annotations import asyncio +import base64 +import functools import ipaddress import itertools import json import logging +import re import secrets import socket +import ssl import struct import threading from typing import Any, Optional +from urllib.parse import unquote, urlsplit import aiohttp +import certifi from aiohttp import WSCloseCode, WSMsgType, web from multidict import CIMultiDict log = logging.getLogger("tunnel") +PROXY_PORTS = {"http": 80, "https": 443, "socks5": 1080, "socks5h": 1080} +PROXY_TIMEOUT = 10 +HOSTNAME = re.compile(r"[A-Za-z0-9_-]{1,63}(\.[A-Za-z0-9_-]{1,63})*\.?") + OPEN, OK, DATA, CLOSE = range(4) HEADER = struct.Struct("!IB") # stream id, frame kind HEARTBEAT = 20 @@ -358,15 +368,7 @@ async def connect_public(host: str, port: int) -> tuple[asyncio.StreamReader, as synthetic one (such as fake-IP DNS in a TUN) and map it back to the real host, which the client cannot check. An IP address from the server must be public.""" infos = await asyncio.get_running_loop().getaddrinfo(host, port, type=socket.SOCK_STREAM) - resolved = [str(info[4][0]) for info in infos] - if is_ip_literal(host): - addresses = [a for a in resolved if is_public_ip(a)] - if not addresses: - raise PermissionError(f"{host} is not a public address") - else: - addresses = [a for a in resolved if not is_local_ip(a)] - if not addresses: - raise PermissionError(f"{host} resolves only to local addresses ({', '.join(resolved)})") + addresses = allowed_addresses(host, [str(info[4][0]) for info in infos]) error: OSError = OSError(f"could not connect to {host}") for address in addresses: try: @@ -381,6 +383,111 @@ async def connect_public(host: str, port: int) -> tuple[asyncio.StreamReader, as raise error +def allowed_addresses(host: str, resolved: list[str]) -> list[str]: + """The addresses of ``host`` the client may dial for the server: public ones for an IP address, non-local + ones for a name.""" + if is_ip_literal(host): + if addresses := [a for a in resolved if is_public_ip(a)]: + return addresses + raise PermissionError(f"{host} is not a public address") + if addresses := [a for a in resolved if not is_local_ip(a)]: + return addresses + raise PermissionError(f"{host} resolves only to local addresses ({', '.join(resolved)})") + + +async def check_proxied_target(host: str, port: int) -> None: + """Refuse a target the exit proxy must not reach for the server: a malformed name, a non-public IP address, + or a name that resolves here only to local addresses. The proxy resolves the name itself, so a name this + machine cannot resolve passes.""" + if not (HOSTNAME.fullmatch(host) or parse_ip(host)): + raise PermissionError("a malformed host name") + name = host.lower().rstrip(".") + if name == "localhost" or name.endswith(".localhost"): + raise PermissionError(f"{host} is this machine") + try: + infos = await asyncio.get_running_loop().getaddrinfo(host, port, type=socket.SOCK_STREAM) + except (OSError, UnicodeError): + if is_ip_literal(host): + raise PermissionError(f"{host} is not a public address") + return + allowed_addresses(host, [str(info[4][0]) for info in infos]) + + +async def connect_proxied(proxy: str, host: str, port: int) -> tuple[asyncio.StreamReader, asyncio.StreamWriter]: + """Connect to ``host`` through an http, https or socks5 proxy on the client's side. A failure raises, and + never falls back to a direct connection.""" + await check_proxied_target(host, port) + uri = urlsplit(proxy) + scheme = uri.scheme.lower() + user, password = unquote(uri.username or ""), unquote(uri.password or "") + tls = tls_context() if scheme == "https" else None + reader, writer = await asyncio.wait_for( + asyncio.open_connection( + uri.hostname, uri.port or PROXY_PORTS[scheme], ssl=tls, server_hostname=uri.hostname if tls else None + ), + PROXY_TIMEOUT, + ) + handshake = socks5_handshake if scheme.startswith("socks5") else http_connect + try: + await asyncio.wait_for(handshake(reader, writer, host, port, user, password), PROXY_TIMEOUT) + except BaseException as e: + writer.close() + if isinstance(e, asyncio.IncompleteReadError): + raise OSError("the proxy closed the connection") from e + raise + return reader, writer + + +@functools.cache +def tls_context() -> ssl.SSLContext: + return ssl.create_default_context(cafile=certifi.where()) + + +async def http_connect( + reader: asyncio.StreamReader, writer: asyncio.StreamWriter, host: str, port: int, user: str, password: str +) -> None: + target = f"[{host}]:{port}" if ":" in host else f"{host}:{port}" + auth = f"Proxy-Authorization: Basic {base64.b64encode(f'{user}:{password}'.encode()).decode()}\r\n" if user else "" + writer.write(f"CONNECT {target} HTTP/1.1\r\nHost: {target}\r\n{auth}\r\n".encode()) + status = await reader.readline() + parts = status.split(maxsplit=2) + if len(parts) < 2 or not parts[0].startswith(b"HTTP/") or not (parts[1].isdigit() and parts[1][:1] == b"2"): + raise OSError(f"the proxy answered {status[:100].decode('latin-1').strip() or 'nothing'}") + for _ in range(100): + line = await reader.readline() + if not line: + raise OSError("the proxy closed the connection") + if line in (b"\r\n", b"\n"): + return + raise OSError("the proxy sent too many headers") + + +async def socks5_handshake( + reader: asyncio.StreamReader, writer: asyncio.StreamWriter, host: str, port: int, user: str, password: str +) -> None: + writer.write(b"\x05\x02\x00\x02" if user else b"\x05\x01\x00") + version, method = await reader.readexactly(2) + if version != 5: + raise OSError("the proxy is not a SOCKS5 proxy") + if method == 2 and user: + name, secret = user.encode(), password.encode() + writer.write(b"\x01" + bytes([len(name)]) + name + bytes([len(secret)]) + secret) + if (await reader.readexactly(2))[1] != 0: + raise OSError("the SOCKS5 proxy refused the login") + elif method != 0: + raise OSError("the SOCKS5 proxy accepts no login method this client has") + ip = parse_ip(host) + address = ( + (b"\x01" if ip.version == 4 else b"\x04") + ip.packed if ip else b"\x03" + bytes([len(host)]) + host.encode() + ) + writer.write(b"\x05\x01\x00" + address + port.to_bytes(2, "big")) + _, code, _, address_type = await reader.readexactly(4) + if code != 0: + raise OSError(f"the SOCKS5 proxy refused the connection (code {code})") + length = {1: 4, 4: 16}.get(address_type) or (await reader.readexactly(1))[0] + await reader.readexactly(length + 2) + + def parse_retry_after(value: Optional[str]) -> Optional[int]: """Whole seconds, at least 1, from a delay-seconds Retry-After; None for an absent, HTTP-date or malformed one.""" if not value or len(value) > 9 or not (value.isascii() and value.isdigit()): @@ -405,9 +512,11 @@ def refusal(e: aiohttp.WSServerHandshakeError) -> str: class TunnelClient: """The client side: a background thread that keeps the tunnel's WebSocket open and dials for the server.""" - def __init__(self, server_url: str, headers: dict[str, str]) -> None: + def __init__(self, server_url: str, headers: dict[str, str], proxy: Optional[str] = None) -> None: self.url = server_url.rstrip("/") + ROUTE self.headers = headers + self.proxy = proxy + self.proxy_failed = False self.id: Optional[str] = None self.error: Optional[BaseException] = None self.retry_after: Optional[int] = None @@ -502,7 +611,9 @@ class TunnelClient: async def open(self, link: Link, sid: int, host: str, port: int) -> None: try: - reader, writer = await connect_public(host, port) + reader, writer = await ( + connect_proxied(self.proxy, host, port) if self.proxy else connect_public(host, port) + ) except PermissionError as e: if host not in self.refused: # the server retries, so warn once per host self.refused.add(host) @@ -510,6 +621,9 @@ class TunnelClient: await link.close_stream(sid) return except (OSError, ValueError, asyncio.TimeoutError) as e: # the idna codec raises ValueError on a bad hostname + if self.proxy and not self.proxy_failed: + self.proxy_failed = True + log.warning(f"Tunnel dial through the proxy failed: {str(e) or 'timed out'}") log.debug(f"Tunnel dial to {host}:{port} failed: {e}") await link.close_stream(sid) return