Compare commits

...
5 Commits
Author SHA1 Message Date
imSp4rky 96c3840b82 fix(session): keep cookie paths so same-name cookies stay on their own path
RnetCookieAdapter stores cookies by domain, path and name and sends path, secure and a leading-dot domain to rnet. It also takes server-rotated values, and worker children and track sessions get the cookies with their paths.
2026-10-31 22:57:22 -06:00
imSp4rky 86f26c31d5 refactor(tracks): name and match resolutions by one standard-height rule
standard_resolution() replaces the aspect-ratio logic in naming and is shared by -q matching, export names and progress/hybrid labels, so 1920x1088, 1916x1076, portrait and DCI sizes get standard names.
2026-10-31 22:05:26 -06:00
imSp4rky d5a8ed2a39 fix(titles): name a dv profile 7 or 8 track with an unnamed pq base DV.HDR 2026-10-31 21:52:03 -06:00
imSp4rky 15e46d8bcc fix(titles): name a PQ video track without HDR10 metadata HDR 2026-10-31 21:42:11 -06:00
imSp4rky f9c1c3d390 feat(tracks): add subtitle.dedupe_codecs to keep one codec per subtitle variant 2026-10-31 16:35:01 -06:00
23 changed files with 587 additions and 260 deletions
+3
View File
@@ -135,6 +135,9 @@ Services that require a login read either **cookies** or **credentials**.
rows a text editor broke. unshackle refuses a JSON export. Export the cookies
again in Netscape format.
unshackle sends each cookie only to the host and path that its row names, as a
browser does. A domain that starts with a dot also covers its subdomains.
- **Credentials**: store a username and password per service in your config:
```yaml title="unshackle.yaml"
+11 -2
View File
@@ -105,8 +105,8 @@ Every variable below is valid in both output and folder templates. unshackle tak
| `title_type` | Media kind of the title | `movie`, `series`, `music` |
| `year` | Release year | `2024` |
| `source` | Service tag / class name (empty with `--no-source`) | `EXAMPLE` |
| `quality` | Resolution with scan suffix | `1080p`, `2160p`, `576i` |
| `resolution` | Resolution number only | `1080` |
| `quality` | Standard resolution with scan suffix | `1080p`, `2160p`, `576i` |
| `resolution` | Standard resolution number only | `1080` |
| `video` | Video codec | `H.264`, `H.265` |
| `hdr` | Dynamic-range label | `HDR`, `HDR10P`, `DV`, `DV.HDR`, `DV.HDR10P`, `HLG` |
| `hfr` | High frame rate marker (frame rate > 30) | `HFR` |
@@ -115,6 +115,15 @@ Every variable below is valid in both output and folder templates. unshackle tak
| `repack` | `REPACK` when the `--repack` flag is used | `REPACK` |
| `lang_tag` | Result of your `language_tags` rules, if configured | `MULTi` |
!!! note "`quality` and `resolution` use the standard height"
unshackle names a video by the nearest standard height (4320, 2160, 1440, 1080, 720, 576, 540,
480, 360 or 240) when it is within 10%. It measures the short side and the height that a 16:9
picture of the same width has, and uses the larger one. So a cropped 1920x800, a padded
1920x1088 and a 1080x1920 portrait video are all `1080p`. A short side within 16 lines of
2160, 1440 or 1080 gives that height, so an ultrawide 2560x1080 is `1080p`. A size with no
standard height near it, such as 1600x900, keeps its own height. `dl -q` and the
`dl --export` file name use the same rule.
!!! note "How `language_tags` rules match"
unshackle tries the rules in order, and they compare languages with the same [matching rules](downloading.md#exact-vs-fuzzy-matching) as `-l`: a rule condition of `en` matches `en-US`, `en-GB`, and any other `en-*` form, but `es-419` matches only `es-419`. Within a single rule, **all** conditions must match (AND logic) for it to apply. The first rule that fully matches wins. If none match, unshackle removes `{lang_tag?}` from the filename cleanly.
+2
View File
@@ -441,6 +441,7 @@ subtitle:
| `sidecar_format` | `srt` | a format name (e.g. `srt`, `vtt`) or `original` | Format used for sidecar files. |
| `group_by` | `type` | `type`, `language` | Whether all tracks of one type stay together, or each language stays next to its own variants. |
| `language_priority` | *(unset)* | a list of languages | Languages to sort to the top. The rest keep the default order. Nothing is removed. |
| `dedupe_codecs` | *(unset)* | a list of codecs, e.g. `[VTT, STPP, WVTT]` | Keep one copy of a subtitle that the service sends in several codecs: the earliest listed codec wins. |
### Which settings to change
@@ -454,6 +455,7 @@ only when you want something the defaults do not do:
| Music and lyric lines stripped along with the sound effects | `sdh_method: filter-subs` |
| Lyric lines kept | `sdh_method: subby`, or install SubtitleEdit |
| No stripped copies made at all | `strip_sdh: false` |
| One copy of each subtitle when the service sends the same one in several codecs | `dedupe_codecs: [VTT, STPP, WVTT]` |
| One backend for every conversion, whatever you install | `conversion_method: <backend>` |
Two keys read as if they control markup, and do not:
+1
View File
@@ -194,6 +194,7 @@ in or writes them as sidecar files. See [Subtitles](../../guide/subtitles.md) fo
| `type_priority` | list | *(unset)* | Ordered ranking of subtitle types (`forced`, `normal`, `sdh`; CC counts as SDH). Types you leave out fall to the end. |
| `group_by` | str | `"type"` | Which key groups the subtitle tracks. `type` puts all forced tracks together, then all normal, then all SDH. `language` keeps each language next to its own variants. |
| `language_priority` | list | *(unset)* | Languages to sort to the top, in the order you give. The rest keep the default order: original language first, then alphabetical. Nothing is removed. |
| `dedupe_codecs` | list | *(unset)* | Subtitle codecs in order of preference, by value or name, for example `[VTT, STPP, WVTT]`. When tracks of the same language, forced/SDH/CC type and name come in several listed codecs, unshackle drops the ones in a later codec. It always keeps codecs you leave out. Unset keeps every track. |
```yaml
subtitle:
+3 -2
View File
@@ -134,8 +134,9 @@ Put the more specific rules first, because unshackle stops at the first match.
Matching is exact, and `hdr` is a composite label for Dolby Vision titles: a DV title with
an HDR10 base layer gives `DV.HDR`, and with an HDR10+ base layer `DV.HDR10P`. Plain `DV`
does **not** match those. Name every variant you want, as the example above does. Note
also that HDR10 gives `HDR`, not `HDR10`. The full set of values is `HDR`, `HDR10P`, `DV`,
`DV.HDR`, `DV.HDR10P`, and `HLG`, as listed in the
also that HDR10 gives `HDR`, not `HDR10`, and so does a PQ video track with no HDR10
metadata. The full set of values is `HDR`, `HDR10P`, `DV`, `DV.HDR`, `DV.HDR10P`, and
`HLG`, as listed in the
[variable table](../../guide/output-and-naming.md#template-variables).
### Operators
-1
View File
@@ -76,7 +76,6 @@ def test_codec_names_renames_video_and_each_hdr_part(monkeypatch: pytest.MonkeyP
height=2160,
format="HEVC",
scan_type="Progressive",
other_display_aspect_ratio=["16:9"],
hdr_format_commercial="HDR10+",
hdr_format="Dolby Vision, Version 1.0, Profile 8.1",
transfer_characteristics=None,
+35 -5
View File
@@ -1,4 +1,4 @@
"""Tests for the HDR tag in the filename context when MediaInfo misses HDR10+."""
"""Tests for the HDR tag in the filename context when MediaInfo misses HDR10+ or the HDR format."""
from __future__ import annotations
@@ -22,7 +22,9 @@ def clean_config(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(config, "codec_names", {})
def hdr_tag(commercial: str, range_: Video.Range) -> str:
def hdr_tag(
commercial: str, range_: Video.Range, transfer: str = "PQ", hdr_format: str = "SMPTE ST 2086", profile: str = ""
) -> str:
movie = Movie(id_="movie-0001", service=DummyService, name="The Film", year=2024, language="en")
movie.tracks = Tracks(Video(url="https://example.test/v.mpd", language="en", codec=Video.Codec.AV1, range_=range_))
video = types.SimpleNamespace(
@@ -30,10 +32,10 @@ def hdr_tag(commercial: str, range_: Video.Range) -> str:
height=1080,
format="AV1",
scan_type="Progressive",
other_display_aspect_ratio=["16:9"],
hdr_format_commercial=commercial,
hdr_format="SMPTE ST 2086",
transfer_characteristics="PQ",
hdr_format=hdr_format,
hdr_format_profile=profile,
transfer_characteristics=transfer,
transfer_characteristics_original=None,
frame_rate="23.976",
)
@@ -51,3 +53,31 @@ def test_an_hdr10_track_keeps_the_hdr_tag() -> None:
def test_mediainfo_hdr10p_is_unchanged() -> None:
assert hdr_tag("HDR10 / HDR10+", Video.Range.HDR10) == "HDR10P"
def test_pq_without_an_hdr_format_gets_the_hdr_tag() -> None:
assert hdr_tag("", Video.Range.HDR10) == "HDR"
def test_hlg_without_an_hdr_format_keeps_the_hlg_tag() -> None:
assert hdr_tag("", Video.Range.HLG, transfer="HLG") == "HLG"
def test_sdr_gets_no_hdr_tag() -> None:
assert hdr_tag("", Video.Range.SDR, transfer="BT.709") == ""
def test_a_profile_8_dv_track_with_an_unnamed_pq_base_gets_dv_hdr() -> None:
assert hdr_tag("Dolby Vision", Video.Range.DV, hdr_format="Dolby Vision", profile="dvhe.08") == "DV.HDR"
def test_a_profile_5_dv_track_stays_dv() -> None:
assert hdr_tag("Dolby Vision", Video.Range.DV, hdr_format="Dolby Vision", profile="dvhe.05") == "DV"
def test_a_profile_8_4_dv_track_with_an_hlg_base_stays_dv() -> None:
assert hdr_tag("Dolby Vision", Video.Range.DV, transfer="HLG", hdr_format="Dolby Vision", profile="dvhe.08") == "DV"
def test_a_dv_track_with_a_named_hdr10_base_keeps_dv_hdr() -> None:
assert hdr_tag("HDR10", Video.Range.DV, hdr_format="Dolby Vision", profile="dvhe.08") == "DV.HDR"
+4 -4
View File
@@ -418,7 +418,7 @@ def test_a_conversion_run_removes_its_rpus_so_the_next_base_makes_its_own(
monkeypatch.setattr(Hybrid, "level_6", level_6)
monkeypatch.setattr(hybrid_mod.console, "print", lambda *a, **k: None)
Hybrid([SimpleNamespace(id="b", path=base, height=2160, range=Video.Range.HDR10P)], "TEST")
Hybrid([SimpleNamespace(id="b", path=base, width=None, height=2160, range=Video.Range.HDR10P)], "TEST")
assert seen == {"stale_l6": False, "active_fraction": 0.9}
assert not (tmp_path / "RPU.bin").exists() and not (tmp_path / "RPU_L6.bin").exists()
@@ -446,8 +446,8 @@ def test_a_failed_resolution_removes_its_crop_edits_and_keeps_the_dv_files(
monkeypatch.setattr(Hybrid, "injecting", failed_inject)
monkeypatch.setattr(hybrid_mod.console, "print", lambda *a, **k: None)
videos = [
SimpleNamespace(id="b", path=tmp_path / "base.mkv", height=2160, range=Video.Range.HDR10),
SimpleNamespace(id="d", path=tmp_path / "dv.mkv", height=1080, range=Video.Range.DV),
SimpleNamespace(id="b", path=tmp_path / "base.mkv", width=None, height=2160, range=Video.Range.HDR10),
SimpleNamespace(id="d", path=tmp_path / "dv.mkv", width=None, height=1080, range=Video.Range.DV),
]
with pytest.raises(ValueError):
@@ -559,7 +559,7 @@ def test_hybrid_asks_for_hdr10plus_only_for_a_conversion_without_dv(
def test_the_hybrid_step_gets_the_base_alone_when_there_is_no_dv(tmp_path: Path) -> None:
base = SimpleNamespace(id="base", path=tmp_path / "base.mp4", height=2160, range=Video.Range.HDR10P)
base = SimpleNamespace(id="base", path=tmp_path / "base.mp4", width=None, height=2160, range=Video.Range.HDR10P)
tracks = hybrid_source_tracks(base, None) # type: ignore[arg-type]
+119
View File
@@ -0,0 +1,119 @@
"""Tests for the standard resolution in the filename context, the export name and -q matching."""
from __future__ import annotations
import types
from typing import Optional
import pytest
from unshackle.core.config import config
from unshackle.core.export_name import resolution
from unshackle.core.titles.movie import Movie
from unshackle.core.tracks import Video
from unshackle.core.tracks.tracks import at_resolution
from unshackle.core.utilities import standard_resolution
class DummyService:
pass
@pytest.fixture(autouse=True)
def clean_config(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(config, "tag_rules", [])
monkeypatch.setattr(config, "language_tags", {})
monkeypatch.setattr(config, "codec_names", {})
def quality(width: Optional[int], height: Optional[int], scan_type: str = "Progressive") -> str:
movie = Movie(id_="movie-0001", service=DummyService, name="The Film", year=2024, language="en")
video = types.SimpleNamespace(
width=width,
height=height,
format="AVC",
scan_type=scan_type,
hdr_format_commercial=None,
hdr_format=None,
transfer_characteristics="BT.709",
transfer_characteristics_original=None,
frame_rate="25.000",
)
media_info = types.SimpleNamespace(video_tracks=[video], audio_tracks=[])
return str(movie.build_base_template_context(media_info)["quality"])
@pytest.mark.parametrize(
("width", "height", "expected"),
[
(3840, 2160, 2160),
(1920, 1080, 1080),
(1280, 720, 720),
(2560, 1440, 1440),
(1920, 1088, 1080), # padded to a multiple of 16
(960, 544, 540),
(640, 368, 360),
(1916, 1076, 1080), # a few lines short
(3836, 2156, 2160),
(1920, 800, 1080), # scope crop
(1920, 1036, 1080), # flat crop
(3840, 1600, 2160),
(1280, 544, 720),
(4096, 2160, 2160), # DCI
(4096, 1716, 2160),
(2048, 858, 1080),
(1440, 1080, 1080), # 4:3 pillarbox
(960, 720, 720),
(720, 576, 576), # anamorphic SD
(704, 576, 576),
(720, 480, 480),
(854, 480, 480),
(1080, 1920, 1080), # portrait
(1080, 1350, 1080),
(720, 1280, 720),
(2560, 1080, 1080), # ultrawide
(3440, 1440, 1440),
(1600, 900, 900), # no standard height near it
],
)
def test_standard_resolution(width: int, height: int, expected: int) -> None:
assert standard_resolution(width, height) == expected
def test_a_picture_with_one_side_uses_it() -> None:
assert standard_resolution(None, 1088) == 1080
def test_a_picture_with_no_size_has_no_resolution() -> None:
assert standard_resolution(None, None) is None
def test_the_filename_quality_uses_the_standard_resolution() -> None:
assert quality(1920, 1088) == "1080p"
assert quality(1920, 800) == "1080p"
def test_an_interlaced_picture_keeps_the_i_suffix() -> None:
assert quality(1920, 1088, scan_type="Interlaced") == "1080i"
def test_a_video_track_with_no_size_gives_an_empty_quality() -> None:
assert quality(None, None) == ""
def video(width: int, height: int) -> Video:
return Video(url="https://example.test/v.mpd", language="en", codec=Video.Codec.AVC, width=width, height=height)
def test_the_export_name_uses_the_standard_resolution() -> None:
assert resolution(video(1080, 1920)) == 1080
assert resolution(video(4096, 2160)) == 2160
@pytest.mark.parametrize(("width", "height"), [(1920, 1080), (1920, 800), (1916, 1076), (1920, 1088)])
def test_q_1080_matches_odd_1080p_encodes(width: int, height: int) -> None:
assert at_resolution(video(width, height), 1080)
def test_q_1080_does_not_match_720p() -> None:
assert not at_resolution(video(1280, 720), 1080)
+94 -2
View File
@@ -37,6 +37,10 @@ class Recorder(BaseHTTPRequestHandler):
self.send_header("Set-Cookie", "sid=abc123; Path=/")
elif self.path.startswith("/set/"):
self.send_header("Set-Cookie", f"{self.path[len('/set/') :]}; Path=/")
elif self.path.startswith("/scoped/set/"):
self.send_header("Set-Cookie", f"{self.path[len('/scoped/set/') :]}; Path=/scoped")
elif self.path.startswith("/scoped/expire/"):
self.send_header("Set-Cookie", f"{self.path[len('/scoped/expire/') :]}=; Max-Age=0; Path=/scoped")
self.end_headers()
self.wfile.write(body)
@@ -57,7 +61,7 @@ def server() -> Iterator[str]:
httpd.server_close()
def make_cookie(name: str, value: str, domain: str) -> Cookie:
def make_cookie(name: str, value: str, domain: str, path: str = "/") -> Cookie:
return Cookie(
version=0,
name=name,
@@ -67,7 +71,7 @@ def make_cookie(name: str, value: str, domain: str) -> Cookie:
domain=domain,
domain_specified=True,
domain_initial_dot=domain.startswith("."),
path="/",
path=path,
path_specified=True,
secure=False,
expires=None,
@@ -333,3 +337,91 @@ def test_the_rebuilt_client_is_published_only_when_it_is_ready(server: str, monk
assert seen and all(client is old for client in seen)
assert session._client is not old
def path_session(*cookies: Cookie) -> RnetSession:
session = RnetSession()
session.cookies.update(list(cookies))
return session
def test_same_name_cookies_reach_only_their_own_path(server: str) -> None:
"""Two cookies with one name on two paths must each reach only their own path."""
session = path_session(make_cookie("h", "A", "127.0.0.1", "/a"), make_cookie("h", "B", "127.0.0.1", "/b"))
for path, expected in (("/a/seg", "h=A"), ("/b/seg", "h=B"), ("/", "")):
session.get(f"{server}{path}")
assert Recorder.cookies_sent[-1] == expected
session.headers["X-Rebuild"] = "1"
del session.headers["X-Rebuild"]
session.get(f"{server}/a/seg")
assert Recorder.cookies_sent[-1] == "h=A"
def test_delete_drops_every_path_copy(server: str) -> None:
"""``del cookies[name]`` must reach a cookie stored on a path, not only the root-path one."""
session = path_session(make_cookie("h", "A", "127.0.0.1", "/a"), make_cookie("h", "B", "127.0.0.1", "/b"))
session.get(f"{server}/a/seg")
del session.cookies["h"]
session.get(f"{server}/a/seg")
session.get(f"{server}/b/seg")
assert Recorder.cookies_sent[-2:] == ["", ""]
assert list(session.cookies.jar) == []
def test_clear_by_path_spares_the_other_path(server: str) -> None:
session = path_session(make_cookie("h", "A", "127.0.0.1", "/a"), make_cookie("h", "B", "127.0.0.1", "/b"))
session.get(f"{server}/a/seg")
session.cookies.clear(domain="127.0.0.1", path="/a", name="h")
session.get(f"{server}/a/seg")
session.get(f"{server}/b/seg")
assert Recorder.cookies_sent[-2:] == ["", "h=B"]
assert session.cookies.get("h", path="/b") == "B"
assert session.cookies.get("h", path="/a") is None
def test_a_rotated_path_cookie_survives_a_rebuild(server: str) -> None:
"""A value the server set for a stored path cookie must win after ``rebuild_client`` and on save."""
session = path_session(make_cookie("h", "STALE", "127.0.0.1", "/scoped"))
session.headers["X-Auth"] = "secret"
session.get(f"{server}/scoped/set/h=FRESH")
session.headers.pop("X-Auth", None)
session.get(f"{server}/scoped/seg")
assert Recorder.cookies_sent[-1] == "h=FRESH"
assert [(c.path, c.value) for c in session.cookies.jar] == [("/scoped", "FRESH")]
def test_a_server_expiry_drops_the_stored_cookie(server: str) -> None:
session = path_session(make_cookie("h", "STALE", "127.0.0.1", "/scoped"))
session.headers["X-Auth"] = "secret"
session.get(f"{server}/scoped/expire/h")
session.headers.pop("X-Auth", None)
session.get(f"{server}/scoped/seg")
assert Recorder.cookies_sent[-1] == ""
assert "h" not in session.cookies
def test_only_a_leading_dot_widens_a_cookie_to_subdomains() -> None:
"""A Netscape row with a leading dot covers subdomains; any other domain stays host-only."""
assert RnetCookieAdapter.to_rnet(make_cookie("s", "1", ".example.com")).domain == "example.com"
assert RnetCookieAdapter.to_rnet(make_cookie("s", "1", "www.example.com")).domain is None
def test_the_latest_write_answers_a_lookup_by_name() -> None:
session = path_session(make_cookie("tok", "file", ".example.com"))
session.cookies.update({"tok": "dict"})
assert session.cookies.get("tok") == "dict"
assert session.cookies["tok"] == "dict"
session.cookies.set("tok", "set", domain=".example.com")
assert session.cookies.get("tok") == "set"
assert session.cookies.get("tok", domain="") == "dict"
assert [c.value for c in session.cookies.jar] == ["set"]
-1
View File
@@ -104,7 +104,6 @@ def test_a_quality_rule_matches_the_built_context(monkeypatch: pytest.MonkeyPatc
height=2160,
format="HEVC",
scan_type="Progressive",
other_display_aspect_ratio=["16:9"],
hdr_format_commercial=None,
hdr_format=None,
transfer_characteristics=None,
+25 -6
View File
@@ -47,7 +47,7 @@ def test_rnet_spec_round_trips_cookie_domains() -> None:
rebuilt = rebuild_session(pickle.loads(pickle.dumps(spec)), 1)
assert rebuilt is not None
assert rebuilt.cookies.get_dict_by_domain() == ns.cookies.get_dict_by_domain()
assert [vars(c) for c in rebuilt.cookies.snapshot()] == [vars(c) for c in ns.cookies.snapshot()]
assert rebuilt.cookies.get_dict() == ns.cookies.get_dict()
assert rebuilt.cookies.get("cdn_token", domain="cdn.example.com") == "abc"
assert rebuilt.cookies.get("api_token", domain="api.example.com") == "def"
@@ -71,17 +71,36 @@ def test_rnet_spec_cookies_carry_domain_information() -> None:
spec = build_session_spec(ns)
assert spec is not None
cookies = spec["cookies"]
assert set(cookies) == {"cdn.example.com", "api.example.com", ""}
assert all(isinstance(jar, dict) for jar in cookies.values())
assert cookies["cdn.example.com"] == {"cdn_token": "abc"}
assert {(c.domain, c.path, c.name, c.value) for c in spec["cookies"]} == {
("cdn.example.com", "/", "cdn_token", "abc"),
("api.example.com", "/", "api_token", "def"),
("", "/", "loose", "ghi"),
}
def test_rnet_spec_keeps_same_name_cookies_on_their_paths() -> None:
"""Same-name cookies on two paths must reach a child apart, or a stale one goes to every path."""
ns = make_session("Chrome131")
ns.cookies.set("hdntl", "one", domain=".cdn.example.com", path="/t/1")
ns.cookies.set("hdntl", "two", domain=".cdn.example.com", path="/t/2")
spec = build_session_spec(ns)
assert spec is not None
rebuilt = rebuild_session(pickle.loads(pickle.dumps(spec)), 1)
assert rebuilt is not None
assert rebuilt.cookies.get("hdntl", path="/t/1") == "one"
assert rebuilt.cookies.get("hdntl", path="/t/2") == "two"
def test_rnet_spec_is_picklable() -> None:
spec = build_session_spec(rnet_session_with_cookies())
assert spec is not None
assert pickle.loads(pickle.dumps(spec)) == spec
loaded = pickle.loads(pickle.dumps(spec))
# stdlib Cookie has no __eq__, so compare the cookies by their fields
assert [vars(c) for c in loaded.pop("cookies")] == [vars(c) for c in spec.pop("cookies")]
assert loaded == spec
def test_rnet_spec_without_impersonate_is_not_rebuildable() -> None:
+28 -3
View File
@@ -3,11 +3,19 @@ import pytest
from unshackle.core.tracks import Subtitle, Tracks
def sub(lang: str, forced: bool = False, sdh: bool = False, cc: bool = False, orig: bool = False) -> Subtitle:
def sub(
lang: str,
forced: bool = False,
sdh: bool = False,
cc: bool = False,
orig: bool = False,
codec: Subtitle.Codec = Subtitle.Codec.SubRip,
tag: str = "",
) -> Subtitle:
track = Subtitle(
url=f"https://example.test/{lang}-{forced}-{sdh}-{cc}",
url=f"https://example.test/{lang}-{forced}-{sdh}-{cc}-{codec.value}{tag}",
language=lang,
codec=Subtitle.Codec.SubRip,
codec=codec,
forced=forced,
sdh=sdh,
cc=cc,
@@ -266,3 +274,20 @@ def test_all_sorts_the_original_track_then_its_other_regional_forms() -> None:
)
t.sort_audio(by_language=["all"])
assert [str(a.language) for a in t.audio] == ["fr-CA", "fr-FR", "en"]
def test_dedupe_subtitle_codecs_keeps_the_first_listed_codec_of_each_variant() -> None:
codec = Subtitle.Codec
vtt = sub("en", codec=codec.WebVTT)
wvtt = sub("en", codec=codec.fVTT)
stpp = sub("en", codec=codec.fTTML)
stpp_cc = sub("en", cc=True, codec=codec.fTTML)
srt = sub("en") # unlisted codec: always kept
de_wvtt = sub("de", codec=codec.fVTT)
vtt_second = sub("en", codec=codec.WebVTT, tag="-second") # same winning codec: kept, it may differ in content
tracks = Tracks()
tracks.subtitles = [wvtt, stpp, vtt, stpp_cc, srt, de_wvtt, vtt_second]
tracks.dedupe_subtitle_codecs([codec.WebVTT, codec.fTTML, codec.fVTT])
assert tracks.subtitles == [vtt, stpp_cc, srt, de_wvtt, vtt_second]
+12 -28
View File
@@ -120,6 +120,7 @@ from unshackle.core.utilities import (
partition_exclusions,
plain_subtitle_tags,
resolve_sort_langs,
standard_resolution,
suggest_font_packages,
time_elapsed_since,
)
@@ -2453,6 +2454,10 @@ class dl:
level="INFO", operation="get_tracks", service=self.service, context=tracks_info
)
if dedupe_codecs := config.subtitle.get("dedupe_codecs"):
codecs = map(SUBTITLE_CODEC.convert, dedupe_codecs)
title.tracks.dedupe_subtitle_codecs([c for c in codecs if isinstance(c, Subtitle.Codec)])
# a plain track covers an SDH track only when it is the same variant, as -sl selects them,
# so en-US SDH is stripped next to a plain en-GB: -sl en-US would otherwise keep only the SDH
if config.subtitle.get("strip_sdh", True):
@@ -2719,9 +2724,7 @@ class dl:
v for v in non_hybrid_tracks if v.range in non_hybrid_ranges
]
for resolution in quality:
if any(v.height == resolution for v in deliverable_pool):
continue
if any(v.width and int(v.width * 9 / 16) == resolution for v in deliverable_pool):
if any(at_resolution(v, resolution) for v in deliverable_pool):
continue
missing_resolutions.append(resolution)
@@ -2737,12 +2740,7 @@ class dl:
if non_hybrid_ranges and non_hybrid_tracks:
non_hybrid_selected = [
v
for v in non_hybrid_tracks
if any(
v.height == res or (v.width and int(v.width * (9 / 16)) == res)
for res in quality
)
v for v in non_hybrid_tracks if any(at_resolution(v, res) for res in quality)
]
title.tracks.videos = Tracks.merge_video_selections(
hybrid_selected, non_hybrid_selected
@@ -2754,9 +2752,7 @@ class dl:
title.tracks.by_resolutions(quality)
for resolution in quality:
if any(v.height == resolution for v in title.tracks.videos):
continue
if any(v.width and int(v.width * 9 / 16) == resolution for v in title.tracks.videos):
if any(at_resolution(v, resolution) for v in title.tracks.videos):
continue
missing_resolutions.append(resolution)
@@ -2814,11 +2810,7 @@ class dl:
candidates = [
t
for t in non_hybrid_tracks
if (
not resolution
or t.height == resolution
or (t.width and int(t.width * (9 / 16)) == resolution)
)
if (not resolution or at_resolution(t, resolution))
and (not color_range or t.range == color_range)
and (not codec or t.codec == codec)
and (vlang is None or str(t.language) == vlang)
@@ -2847,11 +2839,7 @@ class dl:
candidates = [
t
for t in title.tracks.videos
if (
not resolution
or t.height == resolution
or (t.width and int(t.width * (9 / 16)) == resolution)
)
if (not resolution or at_resolution(t, resolution))
and (not color_range or t.range == color_range)
and (not codec or t.codec == codec)
and (vlang is None or str(t.language) == vlang)
@@ -2889,11 +2877,7 @@ class dl:
candidates = [
t
for t in fallback_pool
if (
not resolution
or t.height == resolution
or (t.width and int(t.width * (9 / 16)) == resolution)
)
if (not resolution or at_resolution(t, resolution))
and (not color_range or t.range == color_range)
and (not codec or t.codec == codec)
and (vlang is None or str(t.language) == vlang)
@@ -3649,7 +3633,7 @@ class dl:
head = next((v for v in video_tracks if v), None)
if head:
if len(quality) > 1:
task_description += f" {head.height}p"
task_description += f" {standard_resolution(head.width, head.height)}p"
if len(range_) > 1:
task_description += f" {head.range.name}"
if len(vcodec) > 1:
+4 -3
View File
@@ -10,6 +10,7 @@ import time
from typing import Any, Callable, Optional
from unshackle.core.constants import AnyTrack
from unshackle.core.utilities import standard_resolution
JOB_PROGRESS_TERMINAL_STATES = {"Downloaded", "Decrypted", "[yellow]SKIPPED"}
PROGRESS_EMIT_INTERVAL = 0.25
@@ -26,9 +27,9 @@ def track_progress_label(track: AnyTrack) -> str:
track_type = type(track).__name__
if track_type == "Video":
parts = ["video"]
height = getattr(track, "height", None)
if height:
parts.append(f"{height}p")
resolution = standard_resolution(getattr(track, "width", None), getattr(track, "height", None))
if resolution:
parts.append(f"{resolution}p")
track_range = getattr(track, "range", None)
if track_range is not None:
parts.append(track_range.value)
+2 -8
View File
@@ -1247,7 +1247,7 @@ def build_session_spec(session: Optional[Any]) -> Optional[dict[str, Any]]:
"kind": "rnet",
"impersonate": name,
"headers": dict(session.headers),
"cookies": session.cookies.get_dict_by_domain(),
"cookies": session.cookies.snapshot(), # stdlib Cookies pickle with their paths
"origin_cookies": session.export_origin_cookies(),
"proxy": session.proxies.get("all") or session.proxies.get("https") or session.proxies.get("http"),
}
@@ -1290,13 +1290,7 @@ def rebuild_session(spec: dict[str, Any], max_workers: int) -> Optional[Any]:
ns = make_session(spec["impersonate"])
if spec.get("headers"):
ns.headers.update(spec["headers"])
for domain, jar in spec.get("cookies", {}).items():
if domain:
for name, value in jar.items():
ns.cookies.set(name, value, domain=domain)
else:
# no domain to scope to; update() reproduces the parent's localhost-scoped state
ns.cookies.update(jar)
ns.cookies.update(spec.get("cookies") or [])
if spec.get("proxy"):
ns.proxies.update({"all": spec["proxy"]})
ns.import_origin_cookies(spec.get("origin_cookies") or {})
+3 -9
View File
@@ -15,9 +15,8 @@ from typing import Any, Optional
from unshackle.core.constants import AnyTrack
from unshackle.core.titles import Episode, Movie, Song, Title_T
from unshackle.core.tracks import Audio, Video
from unshackle.core.utilities import sanitize_filename
from unshackle.core.utilities import sanitize_filename, standard_resolution
STANDARD_HEIGHTS = (4320, 2160, 1440, 1080, 720, 576, 480, 360)
MANIFEST_ORDER = (Video.Descriptor.DASH, Video.Descriptor.HLS, Video.Descriptor.ISM, Video.Descriptor.URL)
MAX_SCOPE_PARTS = 3
MAX_TITLE_LENGTH = 100
@@ -107,13 +106,8 @@ def scope_part(titles: Sequence[Title_T], seasons: Mapping[int, Collection[int]]
def resolution(track: Video) -> Optional[int]:
"""The resolution a release name uses: a letterboxed 1920x800 is 1080p, from its width."""
if not track.height:
return None
height = track.height
if track.width and track.width > height:
height = max(height, track.width * 9 // 16)
return next((std for std in STANDARD_HEIGHTS if abs(height - std) <= 16), height)
"""The resolution a release name uses, the same as the {resolution} template variable."""
return standard_resolution(track.width, track.height) if track.height else None
def audio_token(track: Audio) -> str:
+167 -138
View File
@@ -10,9 +10,10 @@ import re
import sys
import time
from collections.abc import Iterator, MutableMapping
from copy import copy
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from http.cookiejar import CookieJar
from http.cookiejar import Cookie, CookieJar
from typing import Any, Optional, Union
from urllib.parse import urlencode, urlparse, urlunparse
@@ -288,210 +289,231 @@ class RnetSessionHeaders(CaseInsensitiveDict):
class RnetCookieAdapter(MutableMapping):
"""Cookie adapter that bridges requests-style cookie access to rnet."""
"""Cookie adapter that bridges requests-style cookie access to rnet.
One store keyed by (domain, path, name) holds every cookie set through this class, in write
order. Same-name cookies on different paths stay apart on the wire, and the latest write
answers a lookup by name.
"""
def __init__(self, client: Any, session: Optional[RnetSession] = None) -> None:
self._client = client
self._session = session
self._cookies: dict[str, dict[str, str]] = {}
self._flat: dict[str, str] = {}
self._original_cookies: list[Any] = []
self._store: dict[tuple[str, str, str], Cookie] = {}
self._from_jar: set[tuple[str, str, str]] = set()
def set_cookie_on_client(self, url: str, name: str, value: str) -> None:
"""Set a cookie on the rnet client, or buffer locally if the client is not yet created."""
@staticmethod
def client_url(cookie: Cookie) -> str:
"""The URL rnet stores *cookie* under; a domain-less cookie goes to localhost."""
host = (cookie.domain or "").lstrip(".")
return f"https://{host}" if host else "https://localhost"
@staticmethod
def to_rnet(cookie: Cookie) -> rnet.Cookie:
"""The rnet form of *cookie*.
A leading dot marks a cookie for subdomains too, as in a Netscape cookie file; without
one the cookie stays on its own host. Expiry stays out, so no layer drops a cookie the
cookie loader deliberately kept.
"""
domain = cookie.domain or ""
return rnet.Cookie(
cookie.name,
cookie.value or "",
domain=domain if domain.startswith(".") else None,
path=cookie.path or "/",
secure=bool(cookie.secure),
)
def push(self, cookie: Cookie) -> None:
"""Set *cookie* on the rnet client; before the client exists the store buffers it."""
if self._client is not None:
try:
self._client.set_cookie(url, rnet.Cookie(name, value))
self._client.set_cookie(self.client_url(cookie), self.to_rnet(cookie))
except Exception:
pass
def add(self, cookie: Cookie, from_jar: bool = False) -> None:
"""Store *cookie* as the latest write for its (domain, path, name) and send it to the client."""
key = (cookie.domain or "", cookie.path or "/", cookie.name)
self._store.pop(key, None)
self._store[key] = cookie
if from_jar:
self._from_jar.add(key)
self.push(cookie)
def snapshot(self) -> list[Cookie]:
"""Every stored cookie, oldest write first. Download threads write the store, so read a copy."""
return list(self._store.values())
def flush_to_client(self) -> None:
"""Push all buffered cookies to the rnet client after unshackle makes it."""
if self._client is None:
return
for domain, cookies in self._cookies.items():
url = f"https://{domain.lstrip('.')}" if domain else "https://localhost"
for name, value in cookies.items():
try:
self._client.set_cookie(url, rnet.Cookie(name, value))
except Exception:
pass
for cookie in self.snapshot():
self.push(cookie)
def client_urls(self, domain: Optional[str] = None) -> list[str]:
def client_urls(self, domain: Optional[str] = None, name: Optional[str] = None) -> list[str]:
"""URLs the rnet cookie jar could hold a cookie under.
rnet keys its jar by URL and cannot list the jar whole, so a removal has to name every
URL this class has set a cookie on and every origin the HTTP session has requested.
rnet keys its jar by URL, removes a cookie only at its exact path, and cannot list the
jar. So a removal names every host this class has set a cookie on, every origin the
HTTP session has requested, and every stored path of the cookie *name*.
"""
urls = {f"https://{d.lstrip('.')}" for d in self._cookies if d}
urls.add("https://localhost")
cookies = self.snapshot()
hosts = {self.client_url(c) for c in cookies} | {"https://localhost"}
if self._session is not None:
urls |= set(self._session._origins)
hosts |= set(self._session._origins)
if domain is not None:
host = domain.lstrip(".")
urls = {url for url in urls if (urlparse(url).hostname or "") == host}
urls.add(f"https://{host}")
return sorted(urls)
hosts = {url for url in hosts if (urlparse(url).hostname or "") == host} | {f"https://{host}"}
paths = {c.path for c in cookies if c.path and c.path != "/" and (name is None or c.name == name)}
return sorted(hosts | {url + path for url in hosts for path in paths})
def remove_cookie_on_client(self, name: str, domain: Optional[str] = None) -> None:
"""Drop a cookie from the rnet cookie jar.
"""Drop a cookie from the rnet cookie jar; call it before the store forgets the cookie's paths.
The client is built with cookie_store=True, so a cookie the server set lives in that
jar alone. Dropping it from the local dicts leaves it going out on the wire.
jar alone. Dropping it from the store leaves it going out on the wire.
"""
if self._client is None:
return
for url in self.client_urls(domain):
for url in self.client_urls(domain, name):
try:
self._client.remove_cookie(url, name)
except Exception:
pass
def absorb(self, response: Any) -> None:
"""Take a value the server set for a stored cookie, so a client rebuild or a cookie save keeps it.
A server cookie replaces each stored cookie of the same name and path that covers the
response host, and an expired one drops it.
"""
server_cookies = response.cookies if self._store else None
if not server_cookies:
return
parsed = urlparse(response.url)
host = parsed.hostname or ""
default_path = (parsed.path[: parsed.path.rfind("/")] or "/") if parsed.path.startswith("/") else "/"
now = datetime.now(timezone.utc)
for server_cookie in server_cookies:
path = server_cookie.path if (server_cookie.path or "").startswith("/") else default_path
expires, max_age = server_cookie.expires, server_cookie.max_age
if expires is not None and expires.tzinfo is None:
expires = expires.replace(tzinfo=timezone.utc)
expired = (max_age is not None and max_age.total_seconds() <= 0) or (expires is not None and expires <= now)
for cookie in self.snapshot():
domain = cookie.domain or ""
if cookie.name != server_cookie.name or cookie.path != path or not domain:
continue
if host != domain.lstrip(".") and not (domain.startswith(".") and host.endswith(domain)):
continue
if expired:
self.clear(domain=domain, path=path, name=cookie.name)
elif cookie.value != server_cookie.value:
fresh = copy(cookie)
fresh.value = server_cookie.value
self.add(fresh)
@property
def jar(self) -> CookieJar:
"""Return a CookieJar with original Cookie objects (requests compat).
"""Return a CookieJar of the cookies that arrived as a CookieJar (requests compat).
``save_cookies`` in dl.py uses this to write cookies back to disk.
``save_cookies`` in dl.py uses this to write cookies back to disk, so it carries a value
the server rotated for one of those cookies but not the cookies set by name.
"""
jar = CookieJar()
for cookie in self._original_cookies:
jar.set_cookie(cookie)
for key, cookie in list(self._store.items()):
if key in self._from_jar:
jar.set_cookie(cookie)
return jar
def update(self, other: Any = None, /, **kwargs: Any) -> None:
if other is None:
other = {}
if isinstance(other, CookieJar):
if isinstance(other, (CookieJar, list)):
for cookie in other:
domain = cookie.domain or ""
name = cookie.name
value = cookie.value or ""
self._flat[name] = value
self._cookies.setdefault(domain, {})[name] = value
self._original_cookies.append(cookie)
url = f"https://{domain.lstrip('.')}" if domain else "https://localhost"
self.set_cookie_on_client(url, name, value)
elif isinstance(other, dict):
for name, value in other.items():
self._flat[name] = value
self.set_cookie_on_client("https://localhost", name, str(value))
self._flat.update(other)
self.add(cookie, from_jar=True)
elif hasattr(other, "items"):
for name, value in other.items():
self._flat[name] = str(value)
self.set_cookie_on_client("https://localhost", name, str(value))
self.add(create_cookie(name, str(value)))
for name, value in kwargs.items():
self._flat[name] = value
self.set_cookie_on_client("https://localhost", name, value)
self.add(create_cookie(name, str(value)))
def get( # type: ignore[override]
self, name: str, default: Optional[str] = None, domain: Optional[str] = None, path: Optional[str] = None
) -> Optional[str]:
if domain and domain in self._cookies:
return self._cookies[domain].get(name, default)
return self._flat.get(name, default)
"""The latest value of *name*."""
for cookie in reversed(self.snapshot()):
if cookie.name == name and domain in (None, cookie.domain) and path in (None, cookie.path):
return cookie.value
return default
def set(self, name: str, value: str, domain: str = "localhost") -> None:
self._flat[name] = value
self._cookies.setdefault(domain, {})[name] = value
url = f"https://{domain.lstrip('.')}"
self.set_cookie_on_client(url, name, value)
def set(self, name: str, value: str, domain: str = "localhost", path: str = "/", **kwargs: Any) -> None:
self.add(create_cookie(name, value, domain=domain, path=path, **kwargs))
def flat(self) -> dict[str, str]:
"""Each cookie name with its latest value."""
return {cookie.name: cookie.value or "" for cookie in self.snapshot()}
def __getitem__(self, name: str) -> str:
return self._flat[name]
return self.flat()[name]
def __setitem__(self, name: str, value: str) -> None:
self.set(name, value)
def __delitem__(self, name: str) -> None:
self._flat.pop(name, None)
for domain_cookies in self._cookies.values():
domain_cookies.pop(name, None)
self._original_cookies = [cookie for cookie in self._original_cookies if cookie.name != name]
self.remove_cookie_on_client(name)
self.clear(name=name)
def __contains__(self, name: object) -> bool:
return name in self._flat
return any(cookie.name == name for cookie in self.snapshot())
def __iter__(self) -> Iterator:
return iter(self._flat)
return iter(self.flat())
def __len__(self) -> int:
return len(self._flat)
return len(self.flat())
def __bool__(self) -> bool:
return bool(self._flat)
return bool(self._store)
def get_dict(self, domain: Optional[str] = None, path: Optional[str] = None) -> dict[str, str]:
"""Return cookies as a plain dict (requests RequestsCookieJar compat).
With a *domain*, this method returns only the cookies for that domain.
It accepts *path* for API compatibility but ignores it (flat storage).
"""
if domain is not None:
return dict(self._cookies.get(domain, {}))
return dict(self._flat)
def get_dict_by_domain(self) -> dict[str, dict[str, str]]:
"""Return cookies grouped by domain, domain-less ones under ``""``.
rnet scopes each cookie to the host of the URL it was set on, so a copy made with
``get_dict`` can only ever replay against one host. This class records cookies that
arrived as a plain dict flat only, and reports them under the empty domain, to match
the localhost fallback in :meth:`flush_to_client`.
"""
grouped = {domain: dict(cookies) for domain, cookies in self._cookies.items() if cookies}
scoped = {name for cookies in grouped.values() for name in cookies}
leftovers = {name: value for name, value in self._flat.items() if name not in scoped}
if leftovers:
grouped[""] = {**grouped.get("", {}), **leftovers}
return grouped
"""Return cookies as a plain dict (requests RequestsCookieJar compat), filtered by exact domain and path."""
return {c.name: c.value or "" for c in self.snapshot() if domain in (None, c.domain) and path in (None, c.path)}
def clear(self, domain: Optional[str] = None, path: Optional[str] = None, name: Optional[str] = None) -> None:
"""Remove cookies (requests RequestsCookieJar compat).
- ``clear()`` removes all cookies.
- ``clear(domain=..., path=..., name=...)`` removes a specific cookie.
- ``clear(domain=..., path=..., name=...)`` removes the cookies that match every given filter.
"""
if name is not None:
self._flat.pop(name, None)
if domain is not None and domain in self._cookies:
self._cookies[domain].pop(name, None)
else:
for domain_cookies in self._cookies.values():
domain_cookies.pop(name, None)
self._original_cookies = [
cookie
for cookie in self._original_cookies
if cookie.name != name or (domain is not None and cookie.domain != domain)
]
self.remove_cookie_on_client(name, domain)
elif domain is not None:
removed = self._cookies.pop(domain, {})
for k in removed:
# Only remove from flat if no other domain has same key
still_exists = any(k in dc for dc in self._cookies.values())
if not still_exists:
self._flat.pop(k, None)
self._original_cookies = [cookie for cookie in self._original_cookies if cookie.domain != domain]
for k in removed:
self.remove_cookie_on_client(k, domain)
else:
self._flat.clear()
self._cookies.clear()
self._original_cookies.clear()
if domain is None and path is None and name is None:
self._store.clear()
self._from_jar.clear()
if self._client is not None:
self._client.clear_cookies()
return
def hit(cookie: Cookie) -> bool:
return domain in (None, cookie.domain) and path in (None, cookie.path) and name in (None, cookie.name)
names = {name} if name is not None else {c.name for c in self.snapshot() if hit(c)}
for each in names:
self.remove_cookie_on_client(each, domain)
self._store = {key: c for key, c in list(self._store.items()) if not hit(c)}
self._from_jar &= self._store.keys()
# removal on the client is by host and name, so put back what a path filter spared
for cookie in self.snapshot():
if cookie.name in names:
self.push(cookie)
def items(self) -> list[tuple[str, str]]: # type: ignore[override]
return list(self._flat.items())
return list(self.flat().items())
def keys(self) -> list[str]: # type: ignore[override]
return list(self._flat.keys())
return list(self.flat().keys())
def values(self) -> list[str]: # type: ignore[override]
return list(self._flat.values())
return list(self.flat().values())
class RnetProxyDict(dict):
@@ -671,9 +693,10 @@ class RnetSession:
that replaces the impersonate header set instead of overlaying it, which breaks the
fingerprint.
A new client also starts with an empty cookie jar. This method replays the cookies
unshackle set itself, then the cookies of every origin this HTTP session has used.
That order lets a value the server rotated win over the buffered one. The new client
A new client also starts with an empty cookie jar. This method replays the stored
cookies, which hold any value the server rotated for them, then the root-path cookies
of every origin this HTTP session has used. A cookie the server set on a deeper path
that the store never held is lost, because rnet cannot list its jar. The new client
goes on the HTTP session last, so a request from another thread keeps the old client
until the new one holds the headers and the cookies.
"""
@@ -739,8 +762,8 @@ class RnetSession:
def scoped_cookies(self) -> CookieJar:
"""Every cookie this HTTP session sends, each scoped to its domain.
The live client's cookies of each requested origin come first, then the cookies set
through :attr:`cookies` for any other host.
The live client's root-path cookies of each requested origin come first, then the
cookies stored in :attr:`cookies` with their own paths.
"""
jar = CookieJar()
hosts: dict[str, set[str]] = {}
@@ -749,11 +772,11 @@ class RnetSession:
for name, value in self.parse_cookie_header(header):
jar.set_cookie(create_cookie(name, value, domain=host))
hosts.setdefault(name, set()).add(host)
for domain, values in self.cookies.get_dict_by_domain().items():
bare = domain.lstrip(".")
for name, value in values.items():
if not any(h == bare or h.endswith("." + bare) for h in hosts.get(name, ())):
jar.set_cookie(create_cookie(name, value, domain=domain))
for cookie in self.cookies.snapshot():
bare = (cookie.domain or "").lstrip(".")
live = any(h == bare or h.endswith("." + bare) for h in hosts.get(cookie.name, ()))
if not (live and cookie.path in ("", "/")):
jar.set_cookie(cookie)
return jar
def import_origin_cookies(self, exported: dict[str, str]) -> None:
@@ -813,6 +836,14 @@ class RnetSession:
sleep_time = backoff_value + random.uniform(-jitter, jitter)
return min(sleep_time, self.max_backoff)
def absorb_cookies(self, response: RnetResponse) -> RnetResponse:
"""Record a value the server set for a stored cookie; a cookie fault must never fail the request."""
try:
self.cookies.absorb(response)
except Exception: # nosec B110 - the store keeps its old value, the rnet jar still has the new one
pass
return response
def request(self, method: str, url: str, **kwargs: Any) -> RnetResponse:
"""Send a request, retrying on the status forcelist and on the caught exception types.
@@ -865,16 +896,14 @@ class RnetSession:
kwargs["headers"] = dict(kwargs["headers"])
if method_upper not in self.allowed_methods:
raw_resp = client.request(rnet_method, url, **kwargs)
return RnetResponse(raw_resp)
return self.absorb_cookies(RnetResponse(client.request(rnet_method, url, **kwargs)))
last_exception: Optional[Exception] = None
response: Optional[RnetResponse] = None
for attempt in range(max_retries + 1):
try:
raw_resp = client.request(rnet_method, url, **kwargs)
response = RnetResponse(raw_resp)
response = self.absorb_cookies(RnetResponse(client.request(rnet_method, url, **kwargs)))
if config.debug_requests:
parsed_url = urlparse(url)
+14 -30
View File
@@ -14,7 +14,7 @@ from unshackle.core.constants import (
VIDEO_CODEC_MAP,
)
from unshackle.core.tracks import Tracks, Video
from unshackle.core.utilities import spoken_language
from unshackle.core.utilities import spoken_language, standard_resolution
def rename(name: Optional[str]) -> str:
@@ -119,40 +119,14 @@ class Title:
context["edition"] = " ".join(first_track.edition)
if primary_video_track:
width = getattr(primary_video_track, "width", primary_video_track.height)
resolution = min(width, primary_video_track.height)
try:
dar = getattr(primary_video_track, "other_display_aspect_ratio", None) or []
if dar and dar[0]:
aspect_ratio = [int(float(plane)) for plane in str(dar[0]).split(":")]
if len(aspect_ratio) == 1:
aspect_ratio.append(1)
ratio = aspect_ratio[0] / aspect_ratio[1]
if ratio not in (16 / 9, 4 / 3, 9 / 16, 3 / 4):
if abs(width - 3840) <= 50:
width = 3840
elif abs(width - 2560) <= 50:
width = 2560
elif abs(width - 1920) <= 50 or abs(width - 1620) <= 50:
width = 1920
elif abs(width - 1280) <= 50 or abs(width - 1080) <= 50:
width = 1280
resolution = int(max(width, primary_video_track.height) * (9 / 16))
track_height = primary_video_track.height
if abs(resolution - track_height) <= 10 or track_height in (2160, 1440, 1080, 720, 480):
resolution = track_height
# garbage DAR strings or a missing height fall back to the plain min(width, height)
except (ValueError, TypeError, ZeroDivisionError):
pass
resolution = standard_resolution(primary_video_track.width, primary_video_track.height)
scan_suffix = "i" if str(getattr(primary_video_track, "scan_type", "")).lower() == "interlaced" else "p"
context.update(
{
"quality": f"{resolution}{scan_suffix}",
"resolution": str(resolution),
"quality": f"{resolution}{scan_suffix}" if resolution else "",
"resolution": str(resolution) if resolution else "",
"video": rename(VIDEO_CODEC_MAP.get(primary_video_track.format, primary_video_track.format)),
}
)
@@ -163,6 +137,14 @@ class Title:
if (primary_video_track.hdr_format or "").startswith("Dolby Vision"):
context["hdr"] = "DV"
base_layer = DYNAMIC_RANGE_MAP.get(hdr_format)
profile = getattr(primary_video_track, "hdr_format_profile", None) or ""
if (
base_layer in (None, "DV")
and trc
and "PQ" in trc
and profile.startswith(("dvhe.07", "dvhe.08"))
):
base_layer = "HDR"
if base_layer and base_layer != "DV":
context["hdr"] += f".{base_layer}"
elif (primary_video_track.hdr_format or "").startswith("HDR Vivid"):
@@ -173,6 +155,8 @@ class Title:
context["hdr"] = DYNAMIC_RANGE_MAP["HDR10+"]
elif trc and "HLG" in trc:
context["hdr"] = "HLG"
elif trc and "PQ" in trc:
context["hdr"] = "HDR"
else:
context["hdr"] = ""
context["hdr"] = ".".join(filter(None, map(rename, context["hdr"].split("."))))
+6 -2
View File
@@ -14,7 +14,7 @@ from rich.rule import Rule
from unshackle.core.binaries import FFMPEG, FFProbe, HDR10PlusTool
from unshackle.core.config import config
from unshackle.core.console import console
from unshackle.core.utilities import log_event
from unshackle.core.utilities import log_event, standard_resolution
from unshackle.core.utils import dovi
from unshackle.core.utils.subprocess import run_step
@@ -136,7 +136,11 @@ class Hybrid:
hdr10_track = next((v for v in videos if v.range == Video.Range.HDR10), None)
hdr10p_track = next((v for v in videos if v.range == Video.Range.HDR10P), None)
track_for_res = hdr10_track or hdr10p_track
self.resolution = f"{track_for_res.height}p" if track_for_res and track_for_res.height else "Unknown"
self.resolution = (
f"{standard_resolution(track_for_res.width, track_for_res.height)}p"
if track_for_res and track_for_res.height
else "Unknown"
)
console.print(Padding(Rule(f"[rule.text]HDR10+DV Hybrid ({self.resolution})"), (1, 2)))
+6 -13
View File
@@ -59,22 +59,15 @@ def direct_session(session: AnySession, proxy: Optional[str] = None) -> Session:
pass
cookies = getattr(session, "cookies", None)
if cookies is not None:
jar = getattr(cookies, "jar", None)
snapshot = getattr(cookies, "snapshot", None)
try:
new.cookies.update(jar if jar is not None else cookies)
if snapshot is None:
new.cookies.update(cookies)
else:
for cookie in snapshot():
new.cookies.set_cookie(copy(cookie)) # set_cookie edits a quoted value in place
except Exception:
pass
# RnetCookieAdapter.jar holds only cookies that arrived as a CookieJar; the ones set by name or
# received from a server live in its domain map, so copy those across too
by_domain = getattr(cookies, "get_dict_by_domain", None)
if by_domain is not None:
try:
for domain, named in by_domain().items():
for name, value in named.items():
if name not in new.cookies:
new.cookies.set(name, value, **({"domain": domain} if domain else {}))
except Exception:
pass
new.mount(
"https://",
HTTPAdapter(
+25 -3
View File
@@ -34,6 +34,7 @@ from unshackle.core.utilities import (
matching_languages,
original_tag,
sanitize_filename,
standard_resolution,
valid_language,
)
from unshackle.core.utils.collections import as_list, flatten
@@ -44,8 +45,12 @@ MP4BOX_OPEN_FAILED = re.compile(r"error while opening|Error opening file|Invalid
def at_resolution(video: Video, res: int) -> bool:
"""Whether a track has this height, or this height from a 16:9 width (a letterboxed encode)."""
return video.height == res or bool(video.width and int(video.width * 9 / 16) == res)
"""Whether a track has this height, this height from a 16:9 width, or this standard resolution (a padded encode)."""
return (
video.height == res
or bool(video.width and int(video.width * 9 / 16) == res)
or standard_resolution(video.width, video.height) == res
)
def file_track_ids(path: Path) -> dict[str, int]:
@@ -308,6 +313,23 @@ class Tracks:
kept.append(track)
setattr(self, kind, kept)
def dedupe_subtitle_codecs(self, priority: Sequence[Subtitle.Codec]) -> None:
"""
Drop a subtitle when the same language, flag set and name exists in a codec earlier in priority.
A subtitle in an unlisted codec is always kept.
"""
rank = {codec: i for i, codec in reversed(list(enumerate(priority)))}
def variant(sub: Subtitle) -> tuple:
signature = self.signature(sub)
return signature[:2] + signature[3:] # the signature without its codec
best: dict[tuple, int] = {}
for sub in self.subtitles:
if sub.codec in rank:
best[variant(sub)] = min(best.get(variant(sub), rank[sub.codec]), rank[sub.codec])
self.subtitles = [s for s in self.subtitles if s.codec not in rank or rank[s.codec] == best[variant(s)]]
def sort_videos(
self, by_language: Optional[Sequence[Union[str, Language]]] = None, exact_match: bool = False
) -> None:
@@ -556,7 +578,7 @@ class Tracks:
for group in groups.values():
matches = [x for x in group if x.height == resolution]
if not matches:
matches = [x for x in group if x.width and int(x.width * (9 / 16)) == resolution]
matches = [x for x in group if at_resolution(x, resolution)]
keep.update(id(x) for x in matches[: per_resolution or None])
self.videos = [x for x in self.videos if id(x) in keep]
+23
View File
@@ -122,6 +122,29 @@ def import_module_by_path(path: Path) -> ModuleType:
return module
RESOLUTION_TIERS = (4320, 2160, 1440, 1080, 720, 576, 540, 480, 360, 240)
def standard_resolution(width: Optional[int], height: Optional[int]) -> Optional[int]:
"""The resolution a release name uses: the nearest standard height within 10%, from the short side or a 16:9 width.
A cropped 1920x800, a padded 1920x1088 and a 1080x1920 portrait picture are all 1080. A size with no
standard height near it keeps its own, and a picture with no size gives None.
"""
sides = [side for side in (width, height) if side]
if not sides:
return None
short, long = min(sides), max(sides)
# naming policy: a short side near 2160, 1440 or 1080 names the picture, so an ultrawide 2560x1080 is 1080
# as other media tools name it, even though a 1440 scope crop has the same size
by_height = next((tier for tier in (2160, 1440, 1080) if abs(short - tier) <= 16), None)
if by_height:
return by_height
equivalent = max(short, long * 9 / 16)
tier = min(RESOLUTION_TIERS, key=lambda t: abs(t - equivalent))
return tier if abs(tier - equivalent) <= tier / 10 else round(equivalent)
def sanitize_filename(filename: str, spacer: str = ".", unicode: Optional[bool] = None) -> str:
"""
Sanitise a string to be filename safe.