diff --git a/docs/dev/rest-api/endpoints.md b/docs/dev/rest-api/endpoints.md index 84f6e952..49be857f 100644 --- a/docs/dev/rest-api/endpoints.md +++ b/docs/dev/rest-api/endpoints.md @@ -493,7 +493,7 @@ Make a download job. It requires `service` and `title_id`. Every other field is | `output_dir` | string | `null` | Output directory, relative to the server's downloads directory. The server rejects a path that resolves outside it with `400`. | | `no_cache` / `reset_cache` | boolean | `false` | Title cache controls. | -**Validation.** Invalid values return `400 INVALID_PARAMETERS`. `vcodec`, `acodec` and `sub_format` take the same spellings as the CLI options of the same name (`unshackle dl --help` prints each list). The API and the CLI read one list, so they cannot disagree. `range` must be one of SDR/HDR10/HDR10P/DV/HLG/HYBRID, and `HDR10+` is also valid. The bitrate, download worker, and download counts must be positive integers. You may set at most one of the `*_only` flags. You cannot combine `no_subs` with `subs_only`, or `no_audio` with `audio_only`. The language fields (`lang`, `v_lang`, `a_lang`, `s_lang`, `forced_s_lang` and the `require_*` fields) must be arrays of strings: a single string such as `"en"` returns `400`. The server checks every other field with the type of its `dl` option. A `null` or an empty string gets the default from the table. +**Validation.** Invalid values return `400 INVALID_PARAMETERS`. `vcodec`, `acodec` and `sub_format` take the same spellings as the CLI options of the same name (`unshackle dl --help` prints each list). The API and the CLI read one list, so they cannot disagree. `range` must be one of SDR/HDR10/HDR10P/VIVID/DV/HLG/HYBRID, and `HDR10+` and `HDR Vivid` are also valid. The bitrate, download worker, and download counts must be positive integers. You may set at most one of the `*_only` flags. You cannot combine `no_subs` with `subs_only`, or `no_audio` with `audio_only`. The language fields (`lang`, `v_lang`, `a_lang`, `s_lang`, `forced_s_lang` and the `require_*` fields) must be arrays of strings: a single string such as `"en"` returns `400`. The server checks every other field with the type of its `dl` option. A `null` or an empty string gets the default from the table. **Service options.** A service's own CLI options travel in the same body. Send them in a nested `service_params` object, or as flat top-level keys. When the body has `service_params`, the server diff --git a/docs/dev/rest-api/quickstart.md b/docs/dev/rest-api/quickstart.md index c67fb107..7181df9f 100644 --- a/docs/dev/rest-api/quickstart.md +++ b/docs/dev/rest-api/quickstart.md @@ -236,7 +236,7 @@ The request body accepts the full download parameter set. A few of the most usef | `quality` | `[]` | Resolution(s), e.g. `[1080]`; empty means best available | | `vcodec` | `[]` | Video codec, e.g. `"H265"` | | `acodec` | `[]` | Audio codec, e.g. `"EC3"` | -| `range` | `["SDR"]` | Dynamic range: `SDR`, `HDR10`, `HDR10+`, `DV`, `HLG`, `HYBRID` | +| `range` | `["SDR"]` | Dynamic range: `SDR`, `HDR10`, `HDR10+`, `VIVID`, `DV`, `HLG`, `HYBRID` | | `wanted` | `[]` | Episodes to fetch, e.g. `["S01E01"]`, or music tracks, e.g. `["1-5"]`. Empty means all | | `lang` | `["orig"]` | Video/audio language(s) | | `s_lang` | `["all"]` | Subtitle language(s) | diff --git a/docs/dev/service-architecture.md b/docs/dev/service-architecture.md index 1bdcfdca..a67a581f 100644 --- a/docs/dev/service-architecture.md +++ b/docs/dev/service-architecture.md @@ -277,8 +277,8 @@ do not give a `name`, unshackle derives a human-readable name from the language. closed-caption descriptors. The relevant enums: - **`Video.Codec`**: `AVC` (H.264), `HEVC` (H.265), `VC1`, `VP8`, `VP9`, `AV1`, `LCEVC`. -- **`Video.Range`**: `SDR`, `HLG`, `HDR10`, `HDR10P` (HDR10+), `DV` (Dolby Vision), and - `HYBRID`. `HYBRID` is special: it tells the pipeline to select both an HDR10(+) base and +- **`Video.Range`**: `SDR`, `HLG`, `HDR10`, `HDR10P` (HDR10+), `VIVID` (HDR Vivid), `DV` (Dolby + Vision), and `HYBRID`. `HYBRID` is special: it tells the pipeline to select both an HDR10(+) base and a DV track and merge the DV RPU onto the HDR10 track with `dovi_tool` (see [Dolby Vision hybrid](#dolby-vision-and-hybrid)). @@ -290,7 +290,7 @@ closed-caption descriptors. The relevant enums: `Audio` adds `bitrate`, `channels`, `joc` (Joint Object Coding, the Atmos indicator), and `descriptive`. `Audio.Codec` covers `AAC`, `AC3` (`DD`), `EC3` (`DD+`), `AC4`, -`OPUS`, `OGG` (Vorbis), `DTS`, `DTSX`, `DTSHD`, `MPEGH`, `ALAC`, `FLAC`, and `MP3`. The `atmos` property is `True` when +`OPUS`, `OGG` (Vorbis), `DTS`, `DTSX`, `DTSHD`, `MPEGH`, `AV3A` (Audio Vivid), `ALAC`, `FLAC`, and `MP3`. The `atmos` property is `True` when `joc` is set (or when `extra` flags it). ### `Subtitle` diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index ed3957af..ab69d634 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -79,7 +79,7 @@ unshackle dl [OPTIONS] SERVICE [SERVICE ARGS...] | `-ab`, `--abitrate` | highest | Exact audio bitrate in kbps. | | `-vb-range`, `--vbitrate-range` | - | Video bitrate range in kbps, e.g. `6000-7000`; picks highest within. Mutually exclusive with `--vbitrate`. | | `-ab-range`, `--abitrate-range` | - | Audio bitrate range in kbps, e.g. `128-256`. Mutually exclusive with `--abitrate`. | -| `-r`, `--range` | `SDR` | Colour range(s): `SDR`, `HLG`, `HDR10`, `HDR10P` (HDR10+), `DV`, `HYBRID`. | +| `-r`, `--range` | `SDR` | Colour range(s): `SDR`, `HLG`, `HDR10`, `HDR10P` (HDR10+), `VIVID` (HDR Vivid), `DV`, `HYBRID`. | | `--generate-dv` | off | With `-r HYBRID`, make Dolby Vision from the HDR10+ metadata or the decoded video when a title has no Dolby Vision track. With `-r HDR10` or `-r HDR10P`, make it for that base and ignore a Dolby Vision track. See [Dolby Vision without a DV track](downloading.md#dolby-vision-without-a-dv-track). | | `-c`, `--channels` | - | Audio channels; matches sub-layouts (5.1 ≈ 6.0). | | `-naa`, `--noatmos` | off | Exclude Dolby Atmos audio tracks. | diff --git a/docs/guide/downloading.md b/docs/guide/downloading.md index dcdc0c2f..1eba5d61 100644 --- a/docs/guide/downloading.md +++ b/docs/guide/downloading.md @@ -142,6 +142,7 @@ unshackle dl -v hevc,avc EXAMPLE 81234567 | `HLG` | Hybrid Log-Gamma | | `HDR10` | HDR10 | | `HDR10P` | HDR10+ (`-r HDR10+` also works) | +| `VIVID` | HDR Vivid (`-r "HDR Vivid"` also works) | | `DV` | Dolby Vision | | `HYBRID` | Fetch both an HDR10/HDR10+ base and a DV track, then merge them | @@ -320,6 +321,7 @@ bitrate for the top renditions, overriding the manifest value: | `DTSX` | `DTS-X` | DTS:X Profile 2 (DTS-UHD) | | `DTSHD` | `DTS-HD` | DTS-HD High Resolution, Master Audio and Express | | `MPEGH` | `MPEG-H` | MPEG-H 3D Audio | +| `AV3A` | `Audio Vivid` | Audio Vivid (AVS3-P3) | | `ALAC` | `ALAC` | Apple Lossless | | `FLAC` | `FLAC` | FLAC | | `MP3` | `MP3` | MPEG-1 Audio Layer III | diff --git a/docs/guide/output-and-naming.md b/docs/guide/output-and-naming.md index 906b7490..51cd48ca 100644 --- a/docs/guide/output-and-naming.md +++ b/docs/guide/output-and-naming.md @@ -108,7 +108,7 @@ Every variable below is valid in both output and folder templates. unshackle tak | `quality` | Standard resolution with scan suffix | `1080p`, `2160p`, `576i` | | `resolution` | Standard resolution number only | `1080` | | `video` | Video codec | `H.264`, `H.265` | -| `hdr` | Dynamic-range label | `HDR`, `HDR10P`, `DV`, `DV.HDR`, `DV.HDR10P`, `HLG` | +| `hdr` | Dynamic-range label | `HDR`, `HDR10P`, `DV`, `DV.HDR`, `DV.HDR10P`, `HDRVivid`, `HLG` | | `hfr` | High frame rate marker (frame rate > 30) | `HFR` | | `3d` | Stereoscopic 3D marker (MV-HEVC video) | `3D` | | `edition` | Edition label from the track, if any | `Directors Cut` | diff --git a/docs/index.md b/docs/index.md index c7676889..30046b85 100644 --- a/docs/index.md +++ b/docs/index.md @@ -37,7 +37,7 @@ at a title and it will: | Media types | Movies, TV series/seasons/episodes, and music albums/tracks | | Manifests | DASH, HLS, and Smooth Streaming (ISM) | | DRM | Widevine and PlayReady, with ClearKey support | -| Video | AVC, HEVC, VC-1, VP8, VP9, AV1; SDR, HLG, HDR10, HDR10+, Dolby Vision, and Dolby Vision **hybrid** merging | +| Video | AVC, HEVC, VC-1, VP8, VP9, AV1; SDR, HLG, HDR10, HDR10+, HDR Vivid, Dolby Vision, and Dolby Vision **hybrid** merging | | Audio | AAC, AC-3, E-AC-3, AC-4, Opus, Vorbis, DTS, DTS-X, ALAC, FLAC, MP3; Dolby Atmos handling; per-channel-layout selection | | Subtitles | SRT, WebVTT, ASS/SSA, TTML and more, with optional SDH stripping and format conversion | | Key vaults | SQLite, MySQL, and HTTP/API vaults to store and share content keys | diff --git a/docs/reference/configuration/download.md b/docs/reference/configuration/download.md index c1883b18..91012e2d 100644 --- a/docs/reference/configuration/download.md +++ b/docs/reference/configuration/download.md @@ -38,7 +38,7 @@ Common config keys. This is a useful subset, and every `dl` flag works: | `quality` | list | `[]` (best) | Resolution(s), e.g. `[1080]`. | | `vcodec` | list | `[]` (any) | Video codec(s), e.g. `[H265]`. | | `acodec` | list | `[]` (any) | Audio codec(s), e.g. `[EC3]`. | -| `range` | list | `["SDR"]` | Colour range(s): `SDR`, `HLG`, `HDR10`, `HDR10P`, `DV`, `HYBRID`. | +| `range` | list | `["SDR"]` | Colour range(s): `SDR`, `HLG`, `HDR10`, `HDR10P`, `VIVID`, `DV`, `HYBRID`. | | `channels` | float | *(unset)* | Audio channels, e.g. `6` for 5.1. | | `sub_format` | str | *(unset)* | Convert subtitles to this format (`srt`, `vtt`, `original`, ...). | | `sub_font` | str | *(unset)* | Font for subtitles converted to ASS; implies `sub_format: ass`. Ignored when `sub_format` is another format. | @@ -433,6 +433,13 @@ probes the two highest and the two lowest bitrates first: | Some carry HDR10+ | The title has HDR10 and HDR10+ encodes side by side, so unshackle probes every track. | | None carry HDR10+ | Every track stays `HDR10`. | +The probe also finds HDR Vivid in HEVC HDR10 and HLG tracks. A track becomes `VIVID`, so +`-r VIVID` selects it, when its first frame carries CUVA HDR Vivid metadata, or when its MPEG-TS +program map table (PMT) has the `cuvv` registration. An HLG track never becomes `HDR10P`. An audio +track with no codec gets its codec from the PMT of an MPEG-TS first segment: the `AVSA` +registration or `stream_type` `0xD5` gives `Audio Vivid`. The probe has no HDR10+ result for an +MPEG-TS track. + A probe that fails, or a fragment whose metadata is encrypted, keeps the label from the service. unshackle does not probe ISM tracks. Each probe makes one to three small requests: the 64 KiB fragment, plus the playlist of an HLS track or the header of a one-file track. The probes run in diff --git a/docs/reference/configuration/output.md b/docs/reference/configuration/output.md index 818468a1..c1ae062c 100644 --- a/docs/reference/configuration/output.md +++ b/docs/reference/configuration/output.md @@ -144,8 +144,8 @@ Put the more specific rules first, because unshackle stops at the first match. an HDR10 base layer gives `DV.HDR`, and with an HDR10+ base layer `DV.HDR10P`. Plain `DV` does **not** match those. Name every variant you want, as the example above does. Note also that HDR10 gives `HDR`, not `HDR10`, and so does a PQ video track with no HDR10 - metadata. The full set of values is `HDR`, `HDR10P`, `DV`, `DV.HDR`, `DV.HDR10P`, and - `HLG`, as listed in the + metadata. The full set of values is `HDR`, `HDR10P`, `DV`, `DV.HDR`, `DV.HDR10P`, + `HDRVivid`, and `HLG`, as listed in the [variable table](../../guide/output-and-naming.md#template-variables). ### Operators diff --git a/tests/core/test_dash_descriptive_audio.py b/tests/core/test_dash_descriptive_audio.py index dfd05f1b..86ac2dab 100644 --- a/tests/core/test_dash_descriptive_audio.py +++ b/tests/core/test_dash_descriptive_audio.py @@ -12,7 +12,7 @@ def adaptation_set(scheme: str, value: str) -> ET.Element: @pytest.mark.parametrize( "scheme,value", [ - # "description" is the value the DASH role scheme defines. Stan writes it, and reading only + # "description" is the value the DASH role scheme defines. A service writes it, and reading only # "descriptive" left every audio-description track unflagged. ("urn:mpeg:dash:role:2011", "description"), ("urn:mpeg:dash:role:2011", "descriptive"), diff --git a/tests/core/test_export_name.py b/tests/core/test_export_name.py index aeea0523..6c5b7819 100644 --- a/tests/core/test_export_name.py +++ b/tests/core/test_export_name.py @@ -11,7 +11,7 @@ import pytest from unshackle.commands.dl import dl from unshackle.core.config import config -from unshackle.core.export_name import ExportNamer, export_name, move_export, season_episodes +from unshackle.core.export_name import ExportNamer, export_name, move_export, quality_parts, season_episodes from unshackle.core.titles import Episode, Movie, Song from unshackle.core.tracks import Audio, Subtitle, Video @@ -251,3 +251,7 @@ def test_summary_when_nothing_was_exported(tmp_path: Path, caplog: pytest.LogCap with caplog.at_level(logging.INFO, logger="download"): instance.log_export_summary() assert "Nothing was exported" in caplog.messages + + +def test_a_range_value_with_a_space_has_none_in_the_name() -> None: + assert "hdrvivid" in quality_parts([video(codec=Video.Codec.HEVC, range_=Video.Range.VIVID)]) diff --git a/tests/core/test_hdr10plus_probe.py b/tests/core/test_hdr10plus_probe.py index 438fe0b7..31919c28 100644 --- a/tests/core/test_hdr10plus_probe.py +++ b/tests/core/test_hdr10plus_probe.py @@ -5,11 +5,14 @@ from typing import Any import pytest +from tests.core.test_export_name import video from unshackle.core.config import Config, config +from unshackle.core.import_service import video_range +from unshackle.core.manifests.snapshot import match_track from unshackle.core.remote_service import RemoteService -from unshackle.core.tracks import Video +from unshackle.core.tracks import Audio, Video from unshackle.core.utils import hdr10plus -from unshackle.core.utils.hdr10plus import HDR10PLUS_T35, fragment_has_hdr10plus +from unshackle.core.utils.hdr10plus import HDR10PLUS_T35, TS_PACKET, fragment_has_hdr10plus def box(kind: bytes, body: bytes) -> bytes: @@ -76,8 +79,10 @@ def probe(monkeypatch: pytest.MonkeyPatch, videos: list[Any]) -> list[str]: return video.id[-1].encode() monkeypatch.setattr(hdr10plus, "first_fragment", fake_fragment) - monkeypatch.setattr(hdr10plus, "fragment_has_hdr10plus", lambda data: data == b"+") - hdr10plus.probe_hdr10plus(videos, None, log=logging.getLogger("test"), workers=4) # type: ignore[arg-type] + monkeypatch.setattr( + hdr10plus, "fragment_has_hdr10plus", lambda data, t35=HDR10PLUS_T35: t35 == HDR10PLUS_T35 and data == b"+" + ) + hdr10plus.probe_dynamic_hdr(videos, None, log=logging.getLogger("test"), workers=4) # type: ignore[arg-type] return probed @@ -132,7 +137,7 @@ def test_on_start_runs_only_when_there_is_a_track_to_probe(monkeypatch: pytest.M started: list[int] = [] sdr = SimpleNamespace(id="s", range=Video.Range.SDR, codec=Video.Codec.AVC, bitrate=1, session=None) for videos in ([sdr], [hevc("v", 2160)]): - hdr10plus.probe_hdr10plus( + hdr10plus.probe_dynamic_hdr( videos, # type: ignore[arg-type] None, # type: ignore[arg-type] log=logging.getLogger("test"), @@ -165,14 +170,16 @@ def test_first_segment_of_an_hls_playlist_keeps_its_byte_offset() -> None: def probe_calls(monkeypatch: pytest.MonkeyPatch, service: Any, **kwargs: Any) -> list[int]: calls: list[int] = [] monkeypatch.setattr( - hdr10plus, "probe_hdr10plus", lambda videos, session, *, log, workers, on_start: calls.append(workers) + hdr10plus, "probe_dynamic_hdr", lambda videos, session, *, log, workers, on_start: calls.append(workers) ) hdr10plus.probed_tracks(service, None, **kwargs) return calls def test_probed_tracks_follows_the_config_switch_and_the_workers_count(monkeypatch: pytest.MonkeyPatch) -> None: - service = SimpleNamespace(get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)]), session=None) + service = SimpleNamespace( + get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)], audio=[]), session=None + ) assert probe_calls(monkeypatch, service, workers=3) == [3] monkeypatch.setattr(config, "hdr10plus_probe", False) assert probe_calls(monkeypatch, service, workers=3) == [] @@ -180,13 +187,15 @@ def test_probed_tracks_follows_the_config_switch_and_the_workers_count(monkeypat def test_probed_tracks_leaves_a_remote_service_to_its_server(monkeypatch: pytest.MonkeyPatch) -> None: remote = RemoteService.__new__(RemoteService) - remote.get_tracks = lambda title: SimpleNamespace(videos=[hevc("v", 2160)]) # type: ignore[method-assign] + remote.get_tracks = lambda title: SimpleNamespace(videos=[hevc("v", 2160)], audio=[]) # type: ignore[method-assign] assert probe_calls(monkeypatch, remote, workers=3) == [] @pytest.mark.parametrize("workers", [-1, "8"]) def test_a_bad_workers_count_is_an_error(monkeypatch: pytest.MonkeyPatch, workers: Any) -> None: - service = SimpleNamespace(get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)]), session=None) + service = SimpleNamespace( + get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)], audio=[]), session=None + ) with pytest.raises(ValueError, match="workers"): probe_calls(monkeypatch, service, workers=workers) @@ -212,7 +221,9 @@ def test_a_fragment_inside_the_first_window_needs_one_request(monkeypatch: pytes def test_a_quoted_dl_workers_value_works_like_the_cli(monkeypatch: pytest.MonkeyPatch) -> None: - service = SimpleNamespace(get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)]), session=None) + service = SimpleNamespace( + get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)], audio=[]), session=None + ) monkeypatch.setitem(config.dl, "workers", "8") assert probe_calls(monkeypatch, service) == [8] @@ -221,3 +232,114 @@ def test_a_dv_track_with_an_hdr10_base_lists_as_dv_plus_hdr10() -> None: video = Video(url="https://x/v.mp4", language="en", codec=Video.Codec.HEVC, range_=Video.Range.DV, hdr10_base=True) assert "[H.265, DV+HDR10]" in str(video) assert Video.from_dict(video.to_dict()).hdr10_base + + +def ts_packet(pid: int, section: bytes) -> bytes: + body = b"\x00" + section + return bytes([0x47, 0x40 | (pid >> 8), pid & 0xFF, 0x10]) + body + b"\xff" * (184 - len(body)) + + +def ts(*streams: tuple[int, bytes]) -> bytes: + """A PAT and a PMT that list each (stream type, registration) on its own PID.""" + pat = b"\x00\xb0\x0d\x00\x01\xc1\x00\x00\x00\x01\xf0\x00" + b"\x00" * 4 + entries = b"" + for index, (stream_type, registration) in enumerate(streams): + descriptors = b"\x05\x04" + registration if registration else b"" + entries += bytes([stream_type, 0xE1, index, 0xF0, len(descriptors)]) + descriptors + body = b"\x00\x01\xc1\x00\x00\xe1\x00\xf0\x00" + entries + b"\x00" * 4 + pmt = b"\x02" + bytes([0xB0 | (len(body) >> 8), len(body) & 0xFF]) + body + return ts_packet(0, pat) + ts_packet(0x1000, pmt) + + +def test_ts_streams_reads_the_pmt_stream_types_and_registrations() -> None: + assert hdr10plus.ts_streams(ts((0x24, b"cuvv"), (0x0F, b""))) == [(0x24, b"cuvv"), (0x0F, b"")] + assert hdr10plus.ts_streams(ts_packet(0, b"\x00" * 16)) is None + assert hdr10plus.is_ts(ts((0xD5, b""))) and not hdr10plus.is_ts(fragment(nal(1, b""))) + + +def serve_segments(monkeypatch: pytest.MonkeyPatch, segments: dict[str, bytes]) -> None: + """Serve each track's first segment through the real first_fragment, keyed by track id.""" + monkeypatch.setattr(hdr10plus, "first_segment", lambda track, session: (track.id, 0)) + monkeypatch.setattr(hdr10plus, "ranged_get", lambda url, session, byte_range: segments[url]) + + +def test_a_cuvv_registered_ts_track_becomes_vivid(monkeypatch: pytest.MonkeyPatch) -> None: + serve_segments( + monkeypatch, + {"vivid": ts((0x24, b"cuvv"), (0x0F, b"")), "plain": ts((0x24, b"")), "dv": ts((0x24, b"cuvv"))}, + ) + videos = [hevc("vivid", 2160), hevc("plain", 1080), hevc("dv", 2160, range_=Video.Range.DV)] + hdr10plus.probe_dynamic_hdr(videos, None, log=logging.getLogger("test"), workers=4) # type: ignore[arg-type] + assert [v.range for v in videos] == [Video.Range.VIVID, Video.Range.HDR10, Video.Range.DV] + assert not videos[2].hdr10plus + + +def test_cuva_sei_makes_an_fmp4_hdr10_or_hlg_track_vivid(monkeypatch: pytest.MonkeyPatch) -> None: + cuva = fragment(nal(32, b"vps"), sei(4, hdr10plus.CUVA_T35 + b"\x00" * 8), nal(1, b"\xff" * 64)) + plus = fragment(nal(32, b"vps"), sei(4, HDR10PLUS_T35 + b"\x00" * 8), nal(1, b"\xff" * 64)) + serve_segments(monkeypatch, {"pq": cuva, "hlg": cuva, "hlg+": plus}) + videos = [ + hevc(i, 2160 - n, range_=r) + for n, (i, r) in enumerate([("pq", Video.Range.HDR10), ("hlg", Video.Range.HLG), ("hlg+", Video.Range.HLG)]) + ] + hdr10plus.probe_dynamic_hdr(videos, None, log=logging.getLogger("test"), workers=4) # type: ignore[arg-type] + assert [v.range for v in videos] == [Video.Range.VIVID, Video.Range.VIVID, Video.Range.HLG] + + +def test_ts_streams_skips_program_zero_and_survives_bad_lengths() -> None: + pat = b"\x00\xb0\x11\x00\x01\xc1\x00\x00" + b"\x00\x00\xe0\x10" + b"\x00\x01\xf0\x00" + b"\x00" * 4 + data = ts_packet(0, pat) + ts((0x24, b"cuvv"))[TS_PACKET:] + assert hdr10plus.ts_streams(data) == [(0x24, b"cuvv")] + pmt = bytearray(ts((0x24, b"cuvv"))[TS_PACKET:]) + pmt[5 + 12 + 4] = 0xFF + pmt[28:] = bytes(len(pmt) - 28) + assert hdr10plus.ts_streams(ts((0x24, b""))[:TS_PACKET] + bytes(pmt)) == [(0x24, b"cuvv")] + + +def test_audio_without_a_codec_takes_it_from_the_ts_stream_type(monkeypatch: pytest.MonkeyPatch) -> None: + fetched: list[str] = [] + + def fake_fragment(track: Any, session: Any) -> bytes: + fetched.append(track.id) + return ts((0xD5, b"")) + + monkeypatch.setattr(hdr10plus, "first_fragment", fake_fragment) + vivid = SimpleNamespace(id="a1", codec=None, session=None) + named = SimpleNamespace(id="a2", codec=Audio.Codec.AAC, session=None) + hdr10plus.probe_audio_codecs([vivid, named], None, log=logging.getLogger("test"), workers=4) # type: ignore[arg-type,list-item] + assert (vivid.codec, named.codec, fetched) == (Audio.Codec.AV3A, Audio.Codec.AAC, ["a1"]) + assert "[Audio Vivid]" in str(Audio(url="https://x/a.m3u8", language="zh", codec=Audio.Codec.AV3A)) + + +def test_an_avsa_registration_names_audio_vivid(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(hdr10plus, "first_fragment", lambda track, session: ts((0x06, b"AVSA"))) + track = SimpleNamespace(id="a", codec=None, session=None) + hdr10plus.probe_audio_codecs([track], None, log=logging.getLogger("test"), workers=1) # type: ignore[arg-type,list-item] + assert track.codec == Audio.Codec.AV3A + + +def test_vivid_imports_by_value_and_matches_its_unprobed_reparse() -> None: + assert video_range("HDR Vivid") == video_range("vivid") == Video.Range.VIVID + for local, remote in ((Video.Range.HDR10, Video.Range.VIVID), (Video.Range.HLG, Video.Range.VIVID)): + assert match_track(video("r", range_=remote), [video("l", range_=local)]) is not None + assert match_track(video("r", range_=Video.Range.VIVID), [video("l", range_=Video.Range.HDR10P)]) is None + + +def test_audio_vivid_has_no_music_container(tmp_path: Any) -> None: + path = tmp_path / "a.ts" + path.write_bytes(b"\x47") + track = Audio(url="https://x/a.m3u8", language="zh", codec=Audio.Codec.AV3A) + track.path = path + with pytest.raises(ValueError, match="Audio Vivid"): + track.to_music_container() + + +def test_probed_tracks_probes_codecless_audio_without_video(monkeypatch: pytest.MonkeyPatch) -> None: + calls: list[int] = [] + monkeypatch.setattr( + hdr10plus, "probe_audio_codecs", lambda audio, session, *, log, workers: calls.append(len(audio)) + ) + service = SimpleNamespace(get_tracks=lambda title: SimpleNamespace(videos=[], audio=[SimpleNamespace(codec=None)])) + service.session = None + hdr10plus.probed_tracks(service, None, workers=2) # type: ignore[arg-type] + assert calls == [1] diff --git a/tests/remote/unit/test_remote_relay.py b/tests/remote/unit/test_remote_relay.py index 351a9f9d..e6c2a699 100644 --- a/tests/remote/unit/test_remote_relay.py +++ b/tests/remote/unit/test_remote_relay.py @@ -215,7 +215,7 @@ def test_a_quiet_remote_login_shows_its_last_server_lines_as_a_notice(monkeypatc shown = [] monkeypatch.setattr(remote_service, "show_notice", shown.append) monkeypatch.setattr(remote_service.time, "sleep", lambda s: None) - logs = iter([["+ IP Region: CA"], ["+ Open URL: https://disneyplus.com/start\n+ Enter pair code: 9835-2503"], []]) + logs = iter([["+ IP Region: CA"], ["+ Open URL: https://example.test/start\n+ Enter pair code: 9835-2503"], []]) statuses = iter(["authenticating", "authenticating", "authenticating", "authenticated"]) class Client: @@ -236,7 +236,7 @@ def test_a_quiet_remote_login_shows_its_last_server_lines_as_a_notice(monkeypatc remote.poll_auth_completion(poll_interval=0) # silent while the remote was still logging; the pair code once it went quiet; cleared when done - assert shown == ["+ IP Region: CA\n+ Open URL: https://disneyplus.com/start\n+ Enter pair code: 9835-2503", None] + assert shown == ["+ IP Region: CA\n+ Open URL: https://example.test/start\n+ Enter pair code: 9835-2503", None] assert remote.auth_notice is None diff --git a/tests/tracks/test_track_selection_fixes.py b/tests/tracks/test_track_selection_fixes.py index 8020fce7..a3eb2c09 100644 --- a/tests/tracks/test_track_selection_fixes.py +++ b/tests/tracks/test_track_selection_fixes.py @@ -103,6 +103,20 @@ def test_sort_videos_resolution_outranks_bitrate() -> None: assert [t.id for t in tracks.videos] == ["lo-bitrate-2160", "hi-bitrate-720"] +def test_sort_videos_orders_crops_of_one_resolution_by_bitrate() -> None: + tracks = Tracks() + tracks.videos = [ + make_video("1610-lo", height=1610, bitrate=4_000_000, width=3840), + make_video("1606-hi", height=1606, bitrate=15_000_000, width=3840), + make_video("1610-hi", height=1610, bitrate=16_000_000, width=3840), + make_video("804", height=804, bitrate=20_000_000, width=1920), + ] + + tracks.sort_videos() + + assert [t.id for t in tracks.videos] == ["1610-hi", "1606-hi", "1610-lo", "804"] + + def test_sort_videos_unknown_bitrate_still_wins_on_resolution() -> None: tracks = Tracks() tracks.videos = [ diff --git a/unshackle/core/api/routes.py b/unshackle/core/api/routes.py index a2e854d4..e0c49903 100644 --- a/unshackle/core/api/routes.py +++ b/unshackle/core/api/routes.py @@ -952,7 +952,7 @@ async def download(request: web.Request) -> web.Response: - type: array items: type: string - description: Audio codec(s) to download (e.g., "AAC" or ["AAC", "EC3"]) - accepts AAC, AC3, DD, EC3, DD+, EAC3, DDP, AC4, AC-4, OPUS, OGG, VORB, VORBIS, DTS, DTSX, DTS-X, DTSHD, DTS-HD, MPEGH, MPEG-H, ALAC, FLAC, MP3 (default - []) + description: Audio codec(s) to download (e.g., "AAC" or ["AAC", "EC3"]) - accepts AAC, AC3, DD, EC3, DD+, EAC3, DDP, AC4, AC-4, OPUS, OGG, VORB, VORBIS, DTS, DTSX, DTS-X, DTSHD, DTS-HD, MPEGH, MPEG-H, AV3A, Audio Vivid, ALAC, FLAC, MP3 (default - []) vbitrate: type: integer description: Video bitrate in kbps (default - None) @@ -964,7 +964,7 @@ async def download(request: web.Request) -> web.Response: items: type: string description: >- - Video colour range (SDR, HDR10, HDR10+, HLG, DV, HYBRID) (default - ["SDR"]). + Video colour range (SDR, HDR10, HDR10+, VIVID, HLG, DV, HYBRID) (default - ["SDR"]). An API key with a server_cdm_max_height for the service cannot set HYBRID. channels: type: number diff --git a/unshackle/core/constants.py b/unshackle/core/constants.py index 1ad4699f..767305cc 100644 --- a/unshackle/core/constants.py +++ b/unshackle/core/constants.py @@ -30,6 +30,7 @@ DYNAMIC_RANGE_MAP = { "Dolby Vision": "DV", "HDR10 / HDR10+": "HDR10P", "HDR10 / HDR10": "HDR", + "HDR Vivid": "HDRVivid", } AUDIO_CODEC_MAP = { "E-AC-3": "DDP", diff --git a/unshackle/core/export_name.py b/unshackle/core/export_name.py index 57e81ff7..0b6efee7 100644 --- a/unshackle/core/export_name.py +++ b/unshackle/core/export_name.py @@ -144,7 +144,7 @@ def quality_parts(tracks: Iterable[AnyTrack]) -> list[str]: groups = [ [f"{h}p" for h in heights], [c.value.lower().replace(".", "") for c in codecs], - [r.value.lower() for r in ranges], + [r.value.lower().replace(" ", "") for r in ranges], [d.name.lower() for d in manifests], list(dict.fromkeys(audio_token(a) for a in audio_sorted)), ] diff --git a/unshackle/core/import_service.py b/unshackle/core/import_service.py index 3921eefd..18c5ccac 100644 --- a/unshackle/core/import_service.py +++ b/unshackle/core/import_service.py @@ -190,10 +190,7 @@ def video_range(value: Any) -> Optional[Video.Range]: name = str(value or "").upper() if name in Video.Range.__members__: return Video.Range[name] - try: - return Video.Range(name) - except ValueError: - return None + return next((r for r in Video.Range if r.value.upper() == name), None) def int_or_none(value: Any) -> Optional[int]: diff --git a/unshackle/core/manifests/snapshot.py b/unshackle/core/manifests/snapshot.py index dca65781..f394bffe 100644 --- a/unshackle/core/manifests/snapshot.py +++ b/unshackle/core/manifests/snapshot.py @@ -19,6 +19,7 @@ from unshackle.core.manifests.dash import DASH, MissingLanguageError from unshackle.core.manifests.ism import ISM from unshackle.core.tracks import Audio, Subtitle, Tracks, Video from unshackle.core.tracks.track import Track +from unshackle.core.tracks.video import PROBED_RANGES log = logging.getLogger("snapshot") @@ -273,9 +274,9 @@ def same_bitrate(local: Optional[int], remote: Optional[int]) -> bool: def match_track(remote_track: Track, local_tracks: list) -> Optional[Track]: """Find the locally-parsed track that matches a remote track by ID or by attributes. - HDR10 matches HDR10P, because the server's HDR10+ probe relabels a track that the local re-parse calls HDR10. + A range matches the ones ``PROBED_RANGES`` lets the server's probe relabel it to, because the local + re-parse does not probe. """ - hdr10_ranges = {Video.Range.HDR10, Video.Range.HDR10P} remote_id = str(remote_track.id) for lt in local_tracks: if str(lt.id) == remote_id: @@ -292,7 +293,11 @@ def match_track(remote_track: Track, local_tracks: list) -> Optional[Track]: if ( lt.width == remote_track.width and lt.height == remote_track.height - and (lt.range == remote_track.range or {lt.range, remote_track.range} == hdr10_ranges) + and ( + lt.range == remote_track.range + or remote_track.range in PROBED_RANGES.get(lt.range, ()) + or lt.range in PROBED_RANGES.get(remote_track.range, ()) + ) and same_bitrate(lt.bitrate, remote_track.bitrate) ): return lt diff --git a/unshackle/core/titles/title.py b/unshackle/core/titles/title.py index d12751e7..dd736831 100644 --- a/unshackle/core/titles/title.py +++ b/unshackle/core/titles/title.py @@ -149,7 +149,7 @@ class Title: if base_layer and base_layer != "DV": context["hdr"] += f".{base_layer}" elif (primary_video_track.hdr_format or "").startswith("HDR Vivid"): - context["hdr"] = "HDR" + context["hdr"] = DYNAMIC_RANGE_MAP["HDR Vivid"] else: context["hdr"] = DYNAMIC_RANGE_MAP.get(hdr_format, "") if context["hdr"] == "HDR" and any(v.range == Video.Range.HDR10P for v in self.tracks.videos[:1]): diff --git a/unshackle/core/tracks/audio.py b/unshackle/core/tracks/audio.py index 826efb87..18268fcc 100644 --- a/unshackle/core/tracks/audio.py +++ b/unshackle/core/tracks/audio.py @@ -57,6 +57,7 @@ class Audio(Track): DTSX = "DTS-X" # https://en.wikipedia.org/wiki/DTS,_Inc.#DTS:X DTSHD = "DTS-HD" # https://en.wikipedia.org/wiki/DTS-HD_Master_Audio MPEGH = "MPEG-H" # https://en.wikipedia.org/wiki/MPEG-H_3D_Audio + AV3A = "Audio Vivid" # AVS3-P3 Audio Vivid ALAC = "ALAC" # https://en.wikipedia.org/wiki/Apple_Lossless_Audio_Codec FLAC = "FLAC" # https://en.wikipedia.org/wiki/FLAC MP3 = "MP3" # https://en.wikipedia.org/wiki/MP3 @@ -87,6 +88,8 @@ class Audio(Track): return Audio.Codec.DTSHD if mime in MPEGH_SAMPLE_ENTRIES: return Audio.Codec.MPEGH + if mime == "av3a": + return Audio.Codec.AV3A if mime == "alac": return Audio.Codec.ALAC if mime == "flac": @@ -97,6 +100,15 @@ class Audio(Track): return Audio.Codec.MP3 raise ValueError(f"The MIME '{mime}' is not a supported Audio Codec") + @staticmethod + def from_ts(stream_type: int, registration: bytes = b"") -> Optional[Audio.Codec]: + """The codec an MPEG-TS PMT entry names by its registration descriptor or ``stream_type``, else None.""" + if registration == b"AVSA": + return Audio.Codec.AV3A + return {0x0F: Audio.Codec.AAC, 0x81: Audio.Codec.AC3, 0x87: Audio.Codec.EC3, 0xD5: Audio.Codec.AV3A}.get( + stream_type + ) + @staticmethod def from_ffprobe(codec_name: str, profile: str = "") -> Optional[Audio.Codec]: """Map an FFprobe ``codec_name`` to a codec, or None for one core does not name.""" @@ -332,10 +344,12 @@ class Audio(Track): """Remux the track into the container of the codec FFprobe reads from the file, cut to the sample window. The declared codec is used only when FFprobe cannot read the file. Every failure raises and keeps the - downloaded file. + downloaded file. Audio Vivid raises: FFprobe reads it as MP3, and no music container carries it. """ if not self.path or not self.path.exists(): raise ValueError("Cannot remux a Track that has not been downloaded.") + if self.codec == Audio.Codec.AV3A: + raise ValueError(f"No music container carries Audio Vivid track {self.id}") stream = self.probe_stream() probed = Audio.Codec.from_ffprobe(stream.get("codec_name", ""), stream.get("profile", "")) diff --git a/unshackle/core/tracks/hybrid.py b/unshackle/core/tracks/hybrid.py index 7e41fce4..b635795c 100644 --- a/unshackle/core/tracks/hybrid.py +++ b/unshackle/core/tracks/hybrid.py @@ -342,9 +342,6 @@ class Hybrid: }, ) self.log.info("✓ Injection Completed") - if self.source == ("itunes" or "appletvplus"): - Path.unlink(config.directories.temp / "hdr10.mkv") - Path.unlink(config.directories.temp / "dv.mkv") def extract_stream(self, save_path: Path, type_: str) -> None: output = Path(config.directories.temp / f"{type_}.hevc") diff --git a/unshackle/core/tracks/tracks.py b/unshackle/core/tracks/tracks.py index 0e89bc08..7e9fffcd 100644 --- a/unshackle/core/tracks/tracks.py +++ b/unshackle/core/tracks/tracks.py @@ -354,13 +354,22 @@ class Tracks: def sort_videos( self, by_language: Optional[Sequence[Union[str, Language]]] = None, exact_match: bool = False ) -> None: - """Sort video tracks by resolution, 2D before 3D, then bitrate, and optionally language. + """Sort video tracks by standard resolution, 2D before 3D, then bitrate, and optionally language. + The standard resolution puts a 3840x1606 and a 3840x1610 crop in one 2160p group, ordered by bitrate. A 3D track carries two views, so its higher bitrate would otherwise rank it over the 2D one. """ if not self.videos: return - self.videos.sort(key=lambda x: (x.height or 0, not x.stereo, float(x.bitrate or 0.0)), reverse=True) + self.videos.sort( + key=lambda x: ( + standard_resolution(x.width, x.height) or 0, + not x.stereo, + float(x.bitrate or 0.0), + x.height or 0, + ), + reverse=True, + ) for language in reversed(by_language or []): if str(language) in ("all", "best"): language = self._original_language(self.videos, exact_match) diff --git a/unshackle/core/tracks/video.py b/unshackle/core/tracks/video.py index be2221db..8edb3546 100644 --- a/unshackle/core/tracks/video.py +++ b/unshackle/core/tracks/video.py @@ -91,6 +91,7 @@ class Video(Track): HLG = "HLG" # https://en.wikipedia.org/wiki/Hybrid_log%E2%80%93gamma HDR10 = "HDR10" # https://en.wikipedia.org/wiki/HDR10 HDR10P = "HDR10+" # https://en.wikipedia.org/wiki/HDR10%2B + VIVID = "HDR Vivid" # CUVA HDR Vivid (T/UWA 005), dynamic metadata over a PQ or HLG base DV = "DV" # https://en.wikipedia.org/wiki/Dolby_Vision HYBRID = "HYBRID" # Selects both HDR10 and DV tracks for hybrid processing with DoviTool @@ -857,6 +858,10 @@ class Video(Track): return True +PROBED_RANGES = { + Video.Range.HDR10: (Video.Range.HDR10P, Video.Range.VIVID), + Video.Range.HLG: (Video.Range.VIVID,), +} HDR_TRANSFER = { Video.Range.HDR10: "smpte2084", Video.Range.HDR10P: "smpte2084", diff --git a/unshackle/core/utils/firefox_cookie_extractor.py b/unshackle/core/utils/firefox_cookie_extractor.py index 04787651..e70a7589 100644 --- a/unshackle/core/utils/firefox_cookie_extractor.py +++ b/unshackle/core/utils/firefox_cookie_extractor.py @@ -67,7 +67,7 @@ def get_local_storage_data(profile_path: Path, hosts: List[str], tmp_dir_path: P cursor = conn.cursor() for host in hosts: - # Firefox stores LocalStorage as reversed host (e.g., moc.viki.:https:443) + # Firefox stores LocalStorage as reversed host (e.g., moc.example.:https:443) # Use strict prefix matching to prevent over-broad harvesting reversed_host = host[::-1] query = "SELECT key, value FROM webappsstore2 WHERE originKey LIKE ?" diff --git a/unshackle/core/utils/hdr10plus.py b/unshackle/core/utils/hdr10plus.py index d3badef3..1aa16926 100644 --- a/unshackle/core/utils/hdr10plus.py +++ b/unshackle/core/utils/hdr10plus.py @@ -3,7 +3,7 @@ from __future__ import annotations import logging import struct from concurrent.futures import ThreadPoolExecutor -from typing import TYPE_CHECKING, Callable, Optional, Union +from typing import TYPE_CHECKING, Callable, Optional, TypeVar, Union from urllib.parse import urljoin from requests import Session @@ -15,10 +15,76 @@ from unshackle.core.utils.bitrate import PROBE_TIMEOUT, ranged_get if TYPE_CHECKING: from unshackle.core.service import Service from unshackle.core.titles import Title_T - from unshackle.core.tracks import Tracks, Video + from unshackle.core.tracks import Audio, Track, Tracks, Video PROBE_BYTES = 64 * 1024 HDR10PLUS_T35 = b"\xb5\x00\x3c\x00\x01\x04" +CUVA_T35 = b"\x26\x00\x04\x00\x05" +TS_PACKET = 188 +T = TypeVar("T") +R = TypeVar("R") + + +def map_probes(probe: Callable[[T], R], items: list[T], workers: int) -> list[R]: + """``probe`` over ``items`` in order, with no more than ``workers`` in flight.""" + if not items: + return [] + with ThreadPoolExecutor(max_workers=min(workers, len(items))) as executor: + return list(executor.map(probe, items)) + + +def is_ts(data: bytes) -> bool: + """Whether the bytes start as MPEG-TS packets.""" + return data[:1] == b"\x47" and (len(data) <= TS_PACKET or data[TS_PACKET : TS_PACKET + 1] == b"\x47") + + +def ts_streams(data: bytes) -> Optional[list[tuple[int, bytes]]]: + """The ``stream_type`` and registration format identifier of each PMT entry, or None without a PMT. + + PAT program 0 names the network information table, so it is skipped. Lengths that run past the packet + are cut to it, so bad bytes give fewer entries, never an exception. + """ + # reads only a PAT and PMT that fit in one packet, as packagers write them; reassemble + # sections across packets if a source with a longer PMT appears + + def pid(b: bytes, i: int) -> int: + return ((b[i] & 0x1F) << 8) | b[i + 1] + + def length(b: bytes, i: int) -> int: + return ((b[i] & 0x0F) << 8) | b[i + 1] + + def section(packet: bytes, table_id: int) -> bytes: + """The packet's ``table_id`` section up to its CRC, or b"" when the packet holds no such section.""" + adaptation = (packet[3] >> 4) & 3 + payload = packet[5 + packet[4] if adaptation & 2 else 4 :] if adaptation & 1 else b"" + body = payload[1 + payload[0] :] if payload else b"" + body = body[: max(0, 3 + length(body, 1) - 4)] if len(body) >= 3 else b"" + return body if len(body) >= 12 and body[0] == table_id else b"" + + pmt_pid = None + for pos in range(0, len(data) - TS_PACKET + 1, TS_PACKET): + packet = data[pos : pos + TS_PACKET] + if packet[0] != 0x47 or not packet[1] & 0x40: + continue + if pid(packet, 1) == 0 and pmt_pid is None: + pat = section(packet, 0x00) + pmt_pid = next((pid(pat, k + 2) for k in range(8, len(pat) - 3, 4) if pat[k] or pat[k + 1]), None) + elif pid(packet, 1) == pmt_pid: + pmt = section(packet, 0x02) + if not pmt: + return None + i, streams = 12 + length(pmt, 10), [] + while i + 5 <= len(pmt): + info_end = min(i + 5 + length(pmt, i + 3), len(pmt)) + registration, j = b"", i + 5 + while j + 2 <= info_end: + if pmt[j] == 0x05 and j + 6 <= info_end: + registration = pmt[j + 2 : j + 6] + j += 2 + pmt[j + 1] + streams.append((pmt[i], registration)) + i = info_end + return streams + return None def sei_number(rbsp: bytes, i: int) -> tuple[Optional[int], int]: @@ -32,8 +98,8 @@ def sei_number(rbsp: bytes, i: int) -> tuple[Optional[int], int]: return value + rbsp[i], i + 1 -def sei_has_hdr10plus(nal: bytes) -> bool: - """Whether an HEVC SEI NAL holds an ST 2094-40 message.""" +def sei_has_t35(nal: bytes, t35: bytes = HDR10PLUS_T35) -> bool: + """Whether an HEVC SEI NAL holds an ST 2094-40 message, or the T.35 message that ``t35`` starts.""" rbsp = remove_emulation_prevention(nal[2:]) i = 0 while i < len(rbsp) and rbsp[i] != 0x80: @@ -41,15 +107,17 @@ def sei_has_hdr10plus(nal: bytes) -> bool: payload_size, i = sei_number(rbsp, i) if payload_type is None or payload_size is None: return False - if payload_type == 4 and rbsp[i : i + len(HDR10PLUS_T35)] == HDR10PLUS_T35: + if payload_type == 4 and rbsp[i : i + len(t35)] == t35: return True i += payload_size return False -def fragment_has_hdr10plus(data: bytes) -> Optional[bool]: +def fragment_has_hdr10plus(data: bytes, t35: bytes = HDR10PLUS_T35) -> Optional[bool]: """Whether the first sample of an fMP4 fragment carries HDR10+ SEI, or None when the bytes cannot tell. + ``t35`` picks another T.35 message, such as ``CUVA_T35`` for HDR Vivid. + The walk ends at the first VCL NAL, because a prefix SEI comes before it. CENC and cbcs keep non-VCL NALs and every NAL length in the clear, so this needs no content key. A sample that is encrypted whole can read as False as well as None; both leave the label alone. @@ -69,13 +137,13 @@ def fragment_has_hdr10plus(data: bytes) -> Optional[bool]: if nal_type == 39: if pos + 4 + size > len(data): return None - if sei_has_hdr10plus(data[pos + 4 : pos + 4 + size]): + if sei_has_t35(data[pos + 4 : pos + 4 + size], t35): return True pos += 4 + size return None -def first_segment(track: "Video", session: Union[Session, RnetSession]) -> Optional[tuple[str, int]]: +def first_segment(track: "Track", session: Union[Session, RnetSession]) -> Optional[tuple[str, int]]: """The URL and start offset of the track's first media segment, looked up without a full parse.""" from unshackle.core.manifests import DASH from unshackle.core.tracks.track import Track @@ -115,8 +183,8 @@ def first_segment(track: "Video", session: Union[Session, RnetSession]) -> Optio return None -def first_fragment(track: "Video", session: Union[Session, RnetSession]) -> Optional[bytes]: - """The first PROBE_BYTES of the track's first media fragment.""" +def first_fragment(track: "Track", session: Union[Session, RnetSession]) -> Optional[bytes]: + """The first PROBE_BYTES of the track's first media fragment, or of its first segment when that is MPEG-TS.""" segment = first_segment(track, session) if not segment: return None @@ -127,6 +195,8 @@ def first_fragment(track: "Video", session: Union[Session, RnetSession]) -> Opti data = ranged_get(url, session, f"bytes={start}-{start + PROBE_BYTES - 1}") if not data: return None + if is_ts(data): + return data skip = 0 for kind, _, _, end in iter_boxes(data, 0, len(data)): if kind in (b"styp", b"moof", b"mdat"): @@ -138,7 +208,7 @@ def first_fragment(track: "Video", session: Union[Session, RnetSession]) -> Opti return None -def probe_hdr10plus( +def probe_dynamic_hdr( videos: list["Video"], session: Union[Session, RnetSession], *, @@ -146,16 +216,19 @@ def probe_hdr10plus( workers: int, on_start: Optional[Callable[[], None]] = None, ) -> None: - """Relabel HEVC HDR10 tracks whose first sample carries HDR10+ SEI as HDR10P, and flag such DV tracks. + """Relabel HEVC tracks by the dynamic HDR metadata of their first sample, and flag DV tracks with HDR10+. - A DV track keeps its DV range and gets ``hdr10plus`` set, so selection and hybrid pairing are unchanged. - HDR10 and DV are separate ladders. A ladder with one track per resolution is sampled at both ends - first. Two tracks at one resolution can be an HDR10 and an HDR10+ encode side by side, so such a - ladder is probed whole. The rules are in the ``hdr10plus_probe`` section of - docs/reference/configuration/download.md. A failed probe never changes a label. ``workers`` caps the - probes in flight, and ``on_start`` runs once before the first. + HDR10+ SEI makes an HDR10 track HDR10P. CUVA SEI, or the ``cuvv`` registration of an MPEG-TS stream, + makes an HDR10 or HLG track VIVID; ``PROBED_RANGES`` holds which label each range takes. A DV track + keeps its DV range and gets ``hdr10plus`` set, so selection and hybrid pairing are unchanged. HDR10, + HLG and DV are separate ladders. A ladder with one track per resolution is sampled at both ends first. + Two tracks at one resolution can be an HDR10 and an HDR10+ encode side by side, so such a ladder is + probed whole. The rules are in the ``hdr10plus_probe`` section of docs/reference/configuration/download.md. + A failed probe never changes a label. ``workers`` caps the probes in flight, and ``on_start`` runs once + before the first. """ from unshackle.core.tracks import Video + from unshackle.core.tracks.video import PROBED_RANGES ladders = [ sorted( @@ -163,26 +236,32 @@ def probe_hdr10plus( key=lambda v: v.bitrate or 0, reverse=True, ) - for range_ in (Video.Range.HDR10, Video.Range.DV) + for range_ in (*PROBED_RANGES, Video.Range.DV) ] if any(ladders) and on_start: on_start() - def probe(video: "Video") -> Optional[bool]: + def probe(video: "Video") -> Union["Video.Range", bool, None]: + """The range the first segment proves (HDR10P or VIVID), False for plain HDR10, None when it cannot tell.""" + found: Union[Video.Range, bool, None] = None try: data = first_fragment(video, video.session or session) - found = fragment_has_hdr10plus(data) if data else None + if data and is_ts(data): + if any(registration == b"cuvv" for _, registration in ts_streams(data) or []): + found = Video.Range.VIVID + elif data and fragment_has_hdr10plus(data, CUVA_T35): + found = Video.Range.VIVID + elif data: + hdr10plus = fragment_has_hdr10plus(data) + found = Video.Range.HDR10P if hdr10plus else hdr10plus except Exception as e: - log.debug(f"{video.id}: HDR10+ probe failed ({e})") + log.debug(f"{video.id}: dynamic HDR probe failed ({e})") return None - log.debug(f"{video.id}: HDR10+ probe {found}") + log.debug(f"{video.id}: dynamic HDR probe {found}") return found - def probe_all(tracks: list["Video"]) -> dict[int, Optional[bool]]: - if not tracks: - return {} - with ThreadPoolExecutor(max_workers=min(workers, len(tracks))) as executor: - return dict(zip(map(id, tracks), executor.map(probe, tracks))) + def probe_all(tracks: list["Video"]) -> dict[int, Union["Video.Range", bool, None]]: + return dict(zip(map(id, tracks), map_probes(probe, tracks, workers))) plans: list[tuple[list["Video"], list["Video"], list["Video"]]] = [] for candidates in ladders: @@ -193,23 +272,39 @@ def probe_hdr10plus( # one round for the samples of every ladder, then one for the rest of each mixed ladder found = probe_all([v for _, samples, _ in plans for v in samples]) + labels = [{found[id(s)] for s in samples} for _, samples, _ in plans] found.update( - probe_all( - [ - v - for _, samples, rest in plans - if any(found[id(s)] for s in samples) and not all(found[id(s)] for s in samples) - for v in rest - ] - ) + probe_all([v for (_, _, rest), seen in zip(plans, labels) if any(seen) and len(seen) > 1 for v in rest]) ) - for candidates, samples, _ in plans: - hits = candidates if all(found[id(s)] for s in samples) else [v for v in candidates if found.get(id(v))] - for video in hits: + for (candidates, _, _), seen in zip(plans, labels): + whole = next(iter(seen)) if len(seen) == 1 else None + for video in candidates: + label = found.get(id(video), whole) if video.range == Video.Range.DV: - video.hdr10plus = True - else: - video.range = Video.Range.HDR10P + video.hdr10plus = video.hdr10plus or label == Video.Range.HDR10P + elif isinstance(label, Video.Range) and label in PROBED_RANGES.get(video.range, ()): + video.range = label + + +def probe_audio_codecs( + audio: list["Audio"], session: Union[Session, RnetSession], *, log: logging.Logger, workers: int +) -> None: + """Name the codec of each audio track that has none from the PMT of its MPEG-TS first segment.""" + from unshackle.core.tracks import Audio + + def probe(track: "Audio") -> None: + try: + data = first_fragment(track, track.session or session) + streams = (ts_streams(data) if data and is_ts(data) else None) or [] + except Exception as e: + log.debug(f"{track.id}: audio codec probe failed ({e})") + return + codec = next(filter(None, (Audio.Codec.from_ts(*stream) for stream in streams)), None) + log.debug(f"{track.id}: audio codec probe {codec}") + if codec: + track.codec = codec + + map_probes(probe, [a for a in audio if not a.codec], workers) def probed_tracks( @@ -219,26 +314,23 @@ def probed_tracks( workers: Optional[int] = None, on_probe: Optional[Callable[[], None]] = None, ) -> "Tracks": - """The title's tracks from ``service.get_tracks``, with HDR10+ found in tracks labelled HDR10. + """The title's tracks from ``service.get_tracks``, with HDR10+ and HDR Vivid found in HDR10 and HLG tracks. - ``workers`` defaults to ``dl.workers`` from the config. A remote service's tracks come back from a - server that already probed them. + Audio with no codec gets one from the MPEG-TS ``stream_type``. ``workers`` defaults to ``dl.workers`` from + the config. A remote service's tracks come back from a server that already probed them. """ from unshackle.core.config import config from unshackle.core.downloaders import default_max_workers from unshackle.core.remote_service import RemoteService tracks = service.get_tracks(title) - if config.hdr10plus_probe and tracks.videos and not isinstance(service, RemoteService): + codecless_audio = any(not a.codec for a in tracks.audio) + if config.hdr10plus_probe and (tracks.videos or codecless_audio) and not isinstance(service, RemoteService): # int() as click does for --workers, so a quoted dl.workers works on the REST path too workers = workers or int(config.dl.get("workers") or 0) or default_max_workers() if not isinstance(workers, int) or isinstance(workers, bool) or workers < 1: raise ValueError(f"workers must be a whole number of 1 or more, not {workers!r}") - probe_hdr10plus( - tracks.videos, - service.session, - log=logging.getLogger("hdr10plus"), - workers=workers, - on_start=on_probe, - ) + log = logging.getLogger("hdr10plus") + probe_dynamic_hdr(tracks.videos, service.session, log=log, workers=workers, on_start=on_probe) + probe_audio_codecs(tracks.audio, service.session, log=log, workers=workers) return tracks