mirror of
https://github.com/unshackle-dl/unshackle.git
synced 2026-11-15 05:25:12 +00:00
feat(tracks): support HDR Vivid and Audio Vivid
Add Video.Range.VIVID and Audio.Codec.AV3A, found by the first-segment probe (CUVA SEI, MPEG-TS cuvv/AVSA/0xD5); sort videos by standard resolution, then bitrate; drop service names left in core and tests. Closes #179
This commit is contained in:
imSp4rky
@@ -493,7 +493,7 @@ Make a download job. It requires `service` and `title_id`. Every other field is
|
||||
| `output_dir` | string | `null` | Output directory, relative to the server's downloads directory. The server rejects a path that resolves outside it with `400`. |
|
||||
| `no_cache` / `reset_cache` | boolean | `false` | Title cache controls. |
|
||||
|
||||
**Validation.** Invalid values return `400 INVALID_PARAMETERS`. `vcodec`, `acodec` and `sub_format` take the same spellings as the CLI options of the same name (`unshackle dl --help` prints each list). The API and the CLI read one list, so they cannot disagree. `range` must be one of SDR/HDR10/HDR10P/DV/HLG/HYBRID, and `HDR10+` is also valid. The bitrate, download worker, and download counts must be positive integers. You may set at most one of the `*_only` flags. You cannot combine `no_subs` with `subs_only`, or `no_audio` with `audio_only`. The language fields (`lang`, `v_lang`, `a_lang`, `s_lang`, `forced_s_lang` and the `require_*` fields) must be arrays of strings: a single string such as `"en"` returns `400`. The server checks every other field with the type of its `dl` option. A `null` or an empty string gets the default from the table.
|
||||
**Validation.** Invalid values return `400 INVALID_PARAMETERS`. `vcodec`, `acodec` and `sub_format` take the same spellings as the CLI options of the same name (`unshackle dl --help` prints each list). The API and the CLI read one list, so they cannot disagree. `range` must be one of SDR/HDR10/HDR10P/VIVID/DV/HLG/HYBRID, and `HDR10+` and `HDR Vivid` are also valid. The bitrate, download worker, and download counts must be positive integers. You may set at most one of the `*_only` flags. You cannot combine `no_subs` with `subs_only`, or `no_audio` with `audio_only`. The language fields (`lang`, `v_lang`, `a_lang`, `s_lang`, `forced_s_lang` and the `require_*` fields) must be arrays of strings: a single string such as `"en"` returns `400`. The server checks every other field with the type of its `dl` option. A `null` or an empty string gets the default from the table.
|
||||
|
||||
**Service options.** A service's own CLI options travel in the same body. Send them in a nested
|
||||
`service_params` object, or as flat top-level keys. When the body has `service_params`, the server
|
||||
|
||||
@@ -236,7 +236,7 @@ The request body accepts the full download parameter set. A few of the most usef
|
||||
| `quality` | `[]` | Resolution(s), e.g. `[1080]`; empty means best available |
|
||||
| `vcodec` | `[]` | Video codec, e.g. `"H265"` |
|
||||
| `acodec` | `[]` | Audio codec, e.g. `"EC3"` |
|
||||
| `range` | `["SDR"]` | Dynamic range: `SDR`, `HDR10`, `HDR10+`, `DV`, `HLG`, `HYBRID` |
|
||||
| `range` | `["SDR"]` | Dynamic range: `SDR`, `HDR10`, `HDR10+`, `VIVID`, `DV`, `HLG`, `HYBRID` |
|
||||
| `wanted` | `[]` | Episodes to fetch, e.g. `["S01E01"]`, or music tracks, e.g. `["1-5"]`. Empty means all |
|
||||
| `lang` | `["orig"]` | Video/audio language(s) |
|
||||
| `s_lang` | `["all"]` | Subtitle language(s) |
|
||||
|
||||
@@ -277,8 +277,8 @@ do not give a `name`, unshackle derives a human-readable name from the language.
|
||||
closed-caption descriptors. The relevant enums:
|
||||
|
||||
- **`Video.Codec`**: `AVC` (H.264), `HEVC` (H.265), `VC1`, `VP8`, `VP9`, `AV1`, `LCEVC`.
|
||||
- **`Video.Range`**: `SDR`, `HLG`, `HDR10`, `HDR10P` (HDR10+), `DV` (Dolby Vision), and
|
||||
`HYBRID`. `HYBRID` is special: it tells the pipeline to select both an HDR10(+) base and
|
||||
- **`Video.Range`**: `SDR`, `HLG`, `HDR10`, `HDR10P` (HDR10+), `VIVID` (HDR Vivid), `DV` (Dolby
|
||||
Vision), and `HYBRID`. `HYBRID` is special: it tells the pipeline to select both an HDR10(+) base and
|
||||
a DV track and merge the DV RPU onto the HDR10 track with `dovi_tool` (see
|
||||
[Dolby Vision hybrid](#dolby-vision-and-hybrid)).
|
||||
|
||||
@@ -290,7 +290,7 @@ closed-caption descriptors. The relevant enums:
|
||||
|
||||
`Audio` adds `bitrate`, `channels`, `joc` (Joint Object Coding, the Atmos indicator),
|
||||
and `descriptive`. `Audio.Codec` covers `AAC`, `AC3` (`DD`), `EC3` (`DD+`), `AC4`,
|
||||
`OPUS`, `OGG` (Vorbis), `DTS`, `DTSX`, `DTSHD`, `MPEGH`, `ALAC`, `FLAC`, and `MP3`. The `atmos` property is `True` when
|
||||
`OPUS`, `OGG` (Vorbis), `DTS`, `DTSX`, `DTSHD`, `MPEGH`, `AV3A` (Audio Vivid), `ALAC`, `FLAC`, and `MP3`. The `atmos` property is `True` when
|
||||
`joc` is set (or when `extra` flags it).
|
||||
|
||||
### `Subtitle`
|
||||
|
||||
@@ -79,7 +79,7 @@ unshackle dl [OPTIONS] SERVICE [SERVICE ARGS...]
|
||||
| `-ab`, `--abitrate` | highest | Exact audio bitrate in kbps. |
|
||||
| `-vb-range`, `--vbitrate-range` | - | Video bitrate range in kbps, e.g. `6000-7000`; picks highest within. Mutually exclusive with `--vbitrate`. |
|
||||
| `-ab-range`, `--abitrate-range` | - | Audio bitrate range in kbps, e.g. `128-256`. Mutually exclusive with `--abitrate`. |
|
||||
| `-r`, `--range` | `SDR` | Colour range(s): `SDR`, `HLG`, `HDR10`, `HDR10P` (HDR10+), `DV`, `HYBRID`. |
|
||||
| `-r`, `--range` | `SDR` | Colour range(s): `SDR`, `HLG`, `HDR10`, `HDR10P` (HDR10+), `VIVID` (HDR Vivid), `DV`, `HYBRID`. |
|
||||
| `--generate-dv` | off | With `-r HYBRID`, make Dolby Vision from the HDR10+ metadata or the decoded video when a title has no Dolby Vision track. With `-r HDR10` or `-r HDR10P`, make it for that base and ignore a Dolby Vision track. See [Dolby Vision without a DV track](downloading.md#dolby-vision-without-a-dv-track). |
|
||||
| `-c`, `--channels` | - | Audio channels; matches sub-layouts (5.1 ≈ 6.0). |
|
||||
| `-naa`, `--noatmos` | off | Exclude Dolby Atmos audio tracks. |
|
||||
|
||||
@@ -142,6 +142,7 @@ unshackle dl -v hevc,avc EXAMPLE 81234567
|
||||
| `HLG` | Hybrid Log-Gamma |
|
||||
| `HDR10` | HDR10 |
|
||||
| `HDR10P` | HDR10+ (`-r HDR10+` also works) |
|
||||
| `VIVID` | HDR Vivid (`-r "HDR Vivid"` also works) |
|
||||
| `DV` | Dolby Vision |
|
||||
| `HYBRID` | Fetch both an HDR10/HDR10+ base and a DV track, then merge them |
|
||||
|
||||
@@ -320,6 +321,7 @@ bitrate for the top renditions, overriding the manifest value:
|
||||
| `DTSX` | `DTS-X` | DTS:X Profile 2 (DTS-UHD) |
|
||||
| `DTSHD` | `DTS-HD` | DTS-HD High Resolution, Master Audio and Express |
|
||||
| `MPEGH` | `MPEG-H` | MPEG-H 3D Audio |
|
||||
| `AV3A` | `Audio Vivid` | Audio Vivid (AVS3-P3) |
|
||||
| `ALAC` | `ALAC` | Apple Lossless |
|
||||
| `FLAC` | `FLAC` | FLAC |
|
||||
| `MP3` | `MP3` | MPEG-1 Audio Layer III |
|
||||
|
||||
@@ -108,7 +108,7 @@ Every variable below is valid in both output and folder templates. unshackle tak
|
||||
| `quality` | Standard resolution with scan suffix | `1080p`, `2160p`, `576i` |
|
||||
| `resolution` | Standard resolution number only | `1080` |
|
||||
| `video` | Video codec | `H.264`, `H.265` |
|
||||
| `hdr` | Dynamic-range label | `HDR`, `HDR10P`, `DV`, `DV.HDR`, `DV.HDR10P`, `HLG` |
|
||||
| `hdr` | Dynamic-range label | `HDR`, `HDR10P`, `DV`, `DV.HDR`, `DV.HDR10P`, `HDRVivid`, `HLG` |
|
||||
| `hfr` | High frame rate marker (frame rate > 30) | `HFR` |
|
||||
| `3d` | Stereoscopic 3D marker (MV-HEVC video) | `3D` |
|
||||
| `edition` | Edition label from the track, if any | `Directors Cut` |
|
||||
|
||||
+1
-1
@@ -37,7 +37,7 @@ at a title and it will:
|
||||
| Media types | Movies, TV series/seasons/episodes, and music albums/tracks |
|
||||
| Manifests | DASH, HLS, and Smooth Streaming (ISM) |
|
||||
| DRM | Widevine and PlayReady, with ClearKey support |
|
||||
| Video | AVC, HEVC, VC-1, VP8, VP9, AV1; SDR, HLG, HDR10, HDR10+, Dolby Vision, and Dolby Vision **hybrid** merging |
|
||||
| Video | AVC, HEVC, VC-1, VP8, VP9, AV1; SDR, HLG, HDR10, HDR10+, HDR Vivid, Dolby Vision, and Dolby Vision **hybrid** merging |
|
||||
| Audio | AAC, AC-3, E-AC-3, AC-4, Opus, Vorbis, DTS, DTS-X, ALAC, FLAC, MP3; Dolby Atmos handling; per-channel-layout selection |
|
||||
| Subtitles | SRT, WebVTT, ASS/SSA, TTML and more, with optional SDH stripping and format conversion |
|
||||
| Key vaults | SQLite, MySQL, and HTTP/API vaults to store and share content keys |
|
||||
|
||||
@@ -38,7 +38,7 @@ Common config keys. This is a useful subset, and every `dl` flag works:
|
||||
| `quality` | list | `[]` (best) | Resolution(s), e.g. `[1080]`. |
|
||||
| `vcodec` | list | `[]` (any) | Video codec(s), e.g. `[H265]`. |
|
||||
| `acodec` | list | `[]` (any) | Audio codec(s), e.g. `[EC3]`. |
|
||||
| `range` | list | `["SDR"]` | Colour range(s): `SDR`, `HLG`, `HDR10`, `HDR10P`, `DV`, `HYBRID`. |
|
||||
| `range` | list | `["SDR"]` | Colour range(s): `SDR`, `HLG`, `HDR10`, `HDR10P`, `VIVID`, `DV`, `HYBRID`. |
|
||||
| `channels` | float | *(unset)* | Audio channels, e.g. `6` for 5.1. |
|
||||
| `sub_format` | str | *(unset)* | Convert subtitles to this format (`srt`, `vtt`, `original`, ...). |
|
||||
| `sub_font` | str | *(unset)* | Font for subtitles converted to ASS; implies `sub_format: ass`. Ignored when `sub_format` is another format. |
|
||||
@@ -433,6 +433,13 @@ probes the two highest and the two lowest bitrates first:
|
||||
| Some carry HDR10+ | The title has HDR10 and HDR10+ encodes side by side, so unshackle probes every track. |
|
||||
| None carry HDR10+ | Every track stays `HDR10`. |
|
||||
|
||||
The probe also finds HDR Vivid in HEVC HDR10 and HLG tracks. A track becomes `VIVID`, so
|
||||
`-r VIVID` selects it, when its first frame carries CUVA HDR Vivid metadata, or when its MPEG-TS
|
||||
program map table (PMT) has the `cuvv` registration. An HLG track never becomes `HDR10P`. An audio
|
||||
track with no codec gets its codec from the PMT of an MPEG-TS first segment: the `AVSA`
|
||||
registration or `stream_type` `0xD5` gives `Audio Vivid`. The probe has no HDR10+ result for an
|
||||
MPEG-TS track.
|
||||
|
||||
A probe that fails, or a fragment whose metadata is encrypted, keeps the label from the service.
|
||||
unshackle does not probe ISM tracks. Each probe makes one to three small requests: the 64 KiB
|
||||
fragment, plus the playlist of an HLS track or the header of a one-file track. The probes run in
|
||||
|
||||
@@ -144,8 +144,8 @@ Put the more specific rules first, because unshackle stops at the first match.
|
||||
an HDR10 base layer gives `DV.HDR`, and with an HDR10+ base layer `DV.HDR10P`. Plain `DV`
|
||||
does **not** match those. Name every variant you want, as the example above does. Note
|
||||
also that HDR10 gives `HDR`, not `HDR10`, and so does a PQ video track with no HDR10
|
||||
metadata. The full set of values is `HDR`, `HDR10P`, `DV`, `DV.HDR`, `DV.HDR10P`, and
|
||||
`HLG`, as listed in the
|
||||
metadata. The full set of values is `HDR`, `HDR10P`, `DV`, `DV.HDR`, `DV.HDR10P`,
|
||||
`HDRVivid`, and `HLG`, as listed in the
|
||||
[variable table](../../guide/output-and-naming.md#template-variables).
|
||||
|
||||
### Operators
|
||||
|
||||
@@ -12,7 +12,7 @@ def adaptation_set(scheme: str, value: str) -> ET.Element:
|
||||
@pytest.mark.parametrize(
|
||||
"scheme,value",
|
||||
[
|
||||
# "description" is the value the DASH role scheme defines. Stan writes it, and reading only
|
||||
# "description" is the value the DASH role scheme defines. A service writes it, and reading only
|
||||
# "descriptive" left every audio-description track unflagged.
|
||||
("urn:mpeg:dash:role:2011", "description"),
|
||||
("urn:mpeg:dash:role:2011", "descriptive"),
|
||||
|
||||
@@ -11,7 +11,7 @@ import pytest
|
||||
|
||||
from unshackle.commands.dl import dl
|
||||
from unshackle.core.config import config
|
||||
from unshackle.core.export_name import ExportNamer, export_name, move_export, season_episodes
|
||||
from unshackle.core.export_name import ExportNamer, export_name, move_export, quality_parts, season_episodes
|
||||
from unshackle.core.titles import Episode, Movie, Song
|
||||
from unshackle.core.tracks import Audio, Subtitle, Video
|
||||
|
||||
@@ -251,3 +251,7 @@ def test_summary_when_nothing_was_exported(tmp_path: Path, caplog: pytest.LogCap
|
||||
with caplog.at_level(logging.INFO, logger="download"):
|
||||
instance.log_export_summary()
|
||||
assert "Nothing was exported" in caplog.messages
|
||||
|
||||
|
||||
def test_a_range_value_with_a_space_has_none_in_the_name() -> None:
|
||||
assert "hdrvivid" in quality_parts([video(codec=Video.Codec.HEVC, range_=Video.Range.VIVID)])
|
||||
|
||||
@@ -5,11 +5,14 @@ from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.core.test_export_name import video
|
||||
from unshackle.core.config import Config, config
|
||||
from unshackle.core.import_service import video_range
|
||||
from unshackle.core.manifests.snapshot import match_track
|
||||
from unshackle.core.remote_service import RemoteService
|
||||
from unshackle.core.tracks import Video
|
||||
from unshackle.core.tracks import Audio, Video
|
||||
from unshackle.core.utils import hdr10plus
|
||||
from unshackle.core.utils.hdr10plus import HDR10PLUS_T35, fragment_has_hdr10plus
|
||||
from unshackle.core.utils.hdr10plus import HDR10PLUS_T35, TS_PACKET, fragment_has_hdr10plus
|
||||
|
||||
|
||||
def box(kind: bytes, body: bytes) -> bytes:
|
||||
@@ -76,8 +79,10 @@ def probe(monkeypatch: pytest.MonkeyPatch, videos: list[Any]) -> list[str]:
|
||||
return video.id[-1].encode()
|
||||
|
||||
monkeypatch.setattr(hdr10plus, "first_fragment", fake_fragment)
|
||||
monkeypatch.setattr(hdr10plus, "fragment_has_hdr10plus", lambda data: data == b"+")
|
||||
hdr10plus.probe_hdr10plus(videos, None, log=logging.getLogger("test"), workers=4) # type: ignore[arg-type]
|
||||
monkeypatch.setattr(
|
||||
hdr10plus, "fragment_has_hdr10plus", lambda data, t35=HDR10PLUS_T35: t35 == HDR10PLUS_T35 and data == b"+"
|
||||
)
|
||||
hdr10plus.probe_dynamic_hdr(videos, None, log=logging.getLogger("test"), workers=4) # type: ignore[arg-type]
|
||||
return probed
|
||||
|
||||
|
||||
@@ -132,7 +137,7 @@ def test_on_start_runs_only_when_there_is_a_track_to_probe(monkeypatch: pytest.M
|
||||
started: list[int] = []
|
||||
sdr = SimpleNamespace(id="s", range=Video.Range.SDR, codec=Video.Codec.AVC, bitrate=1, session=None)
|
||||
for videos in ([sdr], [hevc("v", 2160)]):
|
||||
hdr10plus.probe_hdr10plus(
|
||||
hdr10plus.probe_dynamic_hdr(
|
||||
videos, # type: ignore[arg-type]
|
||||
None, # type: ignore[arg-type]
|
||||
log=logging.getLogger("test"),
|
||||
@@ -165,14 +170,16 @@ def test_first_segment_of_an_hls_playlist_keeps_its_byte_offset() -> None:
|
||||
def probe_calls(monkeypatch: pytest.MonkeyPatch, service: Any, **kwargs: Any) -> list[int]:
|
||||
calls: list[int] = []
|
||||
monkeypatch.setattr(
|
||||
hdr10plus, "probe_hdr10plus", lambda videos, session, *, log, workers, on_start: calls.append(workers)
|
||||
hdr10plus, "probe_dynamic_hdr", lambda videos, session, *, log, workers, on_start: calls.append(workers)
|
||||
)
|
||||
hdr10plus.probed_tracks(service, None, **kwargs)
|
||||
return calls
|
||||
|
||||
|
||||
def test_probed_tracks_follows_the_config_switch_and_the_workers_count(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
service = SimpleNamespace(get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)]), session=None)
|
||||
service = SimpleNamespace(
|
||||
get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)], audio=[]), session=None
|
||||
)
|
||||
assert probe_calls(monkeypatch, service, workers=3) == [3]
|
||||
monkeypatch.setattr(config, "hdr10plus_probe", False)
|
||||
assert probe_calls(monkeypatch, service, workers=3) == []
|
||||
@@ -180,13 +187,15 @@ def test_probed_tracks_follows_the_config_switch_and_the_workers_count(monkeypat
|
||||
|
||||
def test_probed_tracks_leaves_a_remote_service_to_its_server(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
remote = RemoteService.__new__(RemoteService)
|
||||
remote.get_tracks = lambda title: SimpleNamespace(videos=[hevc("v", 2160)]) # type: ignore[method-assign]
|
||||
remote.get_tracks = lambda title: SimpleNamespace(videos=[hevc("v", 2160)], audio=[]) # type: ignore[method-assign]
|
||||
assert probe_calls(monkeypatch, remote, workers=3) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("workers", [-1, "8"])
|
||||
def test_a_bad_workers_count_is_an_error(monkeypatch: pytest.MonkeyPatch, workers: Any) -> None:
|
||||
service = SimpleNamespace(get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)]), session=None)
|
||||
service = SimpleNamespace(
|
||||
get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)], audio=[]), session=None
|
||||
)
|
||||
with pytest.raises(ValueError, match="workers"):
|
||||
probe_calls(monkeypatch, service, workers=workers)
|
||||
|
||||
@@ -212,7 +221,9 @@ def test_a_fragment_inside_the_first_window_needs_one_request(monkeypatch: pytes
|
||||
|
||||
|
||||
def test_a_quoted_dl_workers_value_works_like_the_cli(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
service = SimpleNamespace(get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)]), session=None)
|
||||
service = SimpleNamespace(
|
||||
get_tracks=lambda title: SimpleNamespace(videos=[hevc("v", 2160)], audio=[]), session=None
|
||||
)
|
||||
monkeypatch.setitem(config.dl, "workers", "8")
|
||||
assert probe_calls(monkeypatch, service) == [8]
|
||||
|
||||
@@ -221,3 +232,114 @@ def test_a_dv_track_with_an_hdr10_base_lists_as_dv_plus_hdr10() -> None:
|
||||
video = Video(url="https://x/v.mp4", language="en", codec=Video.Codec.HEVC, range_=Video.Range.DV, hdr10_base=True)
|
||||
assert "[H.265, DV+HDR10]" in str(video)
|
||||
assert Video.from_dict(video.to_dict()).hdr10_base
|
||||
|
||||
|
||||
def ts_packet(pid: int, section: bytes) -> bytes:
|
||||
body = b"\x00" + section
|
||||
return bytes([0x47, 0x40 | (pid >> 8), pid & 0xFF, 0x10]) + body + b"\xff" * (184 - len(body))
|
||||
|
||||
|
||||
def ts(*streams: tuple[int, bytes]) -> bytes:
|
||||
"""A PAT and a PMT that list each (stream type, registration) on its own PID."""
|
||||
pat = b"\x00\xb0\x0d\x00\x01\xc1\x00\x00\x00\x01\xf0\x00" + b"\x00" * 4
|
||||
entries = b""
|
||||
for index, (stream_type, registration) in enumerate(streams):
|
||||
descriptors = b"\x05\x04" + registration if registration else b""
|
||||
entries += bytes([stream_type, 0xE1, index, 0xF0, len(descriptors)]) + descriptors
|
||||
body = b"\x00\x01\xc1\x00\x00\xe1\x00\xf0\x00" + entries + b"\x00" * 4
|
||||
pmt = b"\x02" + bytes([0xB0 | (len(body) >> 8), len(body) & 0xFF]) + body
|
||||
return ts_packet(0, pat) + ts_packet(0x1000, pmt)
|
||||
|
||||
|
||||
def test_ts_streams_reads_the_pmt_stream_types_and_registrations() -> None:
|
||||
assert hdr10plus.ts_streams(ts((0x24, b"cuvv"), (0x0F, b""))) == [(0x24, b"cuvv"), (0x0F, b"")]
|
||||
assert hdr10plus.ts_streams(ts_packet(0, b"\x00" * 16)) is None
|
||||
assert hdr10plus.is_ts(ts((0xD5, b""))) and not hdr10plus.is_ts(fragment(nal(1, b"")))
|
||||
|
||||
|
||||
def serve_segments(monkeypatch: pytest.MonkeyPatch, segments: dict[str, bytes]) -> None:
|
||||
"""Serve each track's first segment through the real first_fragment, keyed by track id."""
|
||||
monkeypatch.setattr(hdr10plus, "first_segment", lambda track, session: (track.id, 0))
|
||||
monkeypatch.setattr(hdr10plus, "ranged_get", lambda url, session, byte_range: segments[url])
|
||||
|
||||
|
||||
def test_a_cuvv_registered_ts_track_becomes_vivid(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
serve_segments(
|
||||
monkeypatch,
|
||||
{"vivid": ts((0x24, b"cuvv"), (0x0F, b"")), "plain": ts((0x24, b"")), "dv": ts((0x24, b"cuvv"))},
|
||||
)
|
||||
videos = [hevc("vivid", 2160), hevc("plain", 1080), hevc("dv", 2160, range_=Video.Range.DV)]
|
||||
hdr10plus.probe_dynamic_hdr(videos, None, log=logging.getLogger("test"), workers=4) # type: ignore[arg-type]
|
||||
assert [v.range for v in videos] == [Video.Range.VIVID, Video.Range.HDR10, Video.Range.DV]
|
||||
assert not videos[2].hdr10plus
|
||||
|
||||
|
||||
def test_cuva_sei_makes_an_fmp4_hdr10_or_hlg_track_vivid(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
cuva = fragment(nal(32, b"vps"), sei(4, hdr10plus.CUVA_T35 + b"\x00" * 8), nal(1, b"\xff" * 64))
|
||||
plus = fragment(nal(32, b"vps"), sei(4, HDR10PLUS_T35 + b"\x00" * 8), nal(1, b"\xff" * 64))
|
||||
serve_segments(monkeypatch, {"pq": cuva, "hlg": cuva, "hlg+": plus})
|
||||
videos = [
|
||||
hevc(i, 2160 - n, range_=r)
|
||||
for n, (i, r) in enumerate([("pq", Video.Range.HDR10), ("hlg", Video.Range.HLG), ("hlg+", Video.Range.HLG)])
|
||||
]
|
||||
hdr10plus.probe_dynamic_hdr(videos, None, log=logging.getLogger("test"), workers=4) # type: ignore[arg-type]
|
||||
assert [v.range for v in videos] == [Video.Range.VIVID, Video.Range.VIVID, Video.Range.HLG]
|
||||
|
||||
|
||||
def test_ts_streams_skips_program_zero_and_survives_bad_lengths() -> None:
|
||||
pat = b"\x00\xb0\x11\x00\x01\xc1\x00\x00" + b"\x00\x00\xe0\x10" + b"\x00\x01\xf0\x00" + b"\x00" * 4
|
||||
data = ts_packet(0, pat) + ts((0x24, b"cuvv"))[TS_PACKET:]
|
||||
assert hdr10plus.ts_streams(data) == [(0x24, b"cuvv")]
|
||||
pmt = bytearray(ts((0x24, b"cuvv"))[TS_PACKET:])
|
||||
pmt[5 + 12 + 4] = 0xFF
|
||||
pmt[28:] = bytes(len(pmt) - 28)
|
||||
assert hdr10plus.ts_streams(ts((0x24, b""))[:TS_PACKET] + bytes(pmt)) == [(0x24, b"cuvv")]
|
||||
|
||||
|
||||
def test_audio_without_a_codec_takes_it_from_the_ts_stream_type(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
fetched: list[str] = []
|
||||
|
||||
def fake_fragment(track: Any, session: Any) -> bytes:
|
||||
fetched.append(track.id)
|
||||
return ts((0xD5, b""))
|
||||
|
||||
monkeypatch.setattr(hdr10plus, "first_fragment", fake_fragment)
|
||||
vivid = SimpleNamespace(id="a1", codec=None, session=None)
|
||||
named = SimpleNamespace(id="a2", codec=Audio.Codec.AAC, session=None)
|
||||
hdr10plus.probe_audio_codecs([vivid, named], None, log=logging.getLogger("test"), workers=4) # type: ignore[arg-type,list-item]
|
||||
assert (vivid.codec, named.codec, fetched) == (Audio.Codec.AV3A, Audio.Codec.AAC, ["a1"])
|
||||
assert "[Audio Vivid]" in str(Audio(url="https://x/a.m3u8", language="zh", codec=Audio.Codec.AV3A))
|
||||
|
||||
|
||||
def test_an_avsa_registration_names_audio_vivid(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(hdr10plus, "first_fragment", lambda track, session: ts((0x06, b"AVSA")))
|
||||
track = SimpleNamespace(id="a", codec=None, session=None)
|
||||
hdr10plus.probe_audio_codecs([track], None, log=logging.getLogger("test"), workers=1) # type: ignore[arg-type,list-item]
|
||||
assert track.codec == Audio.Codec.AV3A
|
||||
|
||||
|
||||
def test_vivid_imports_by_value_and_matches_its_unprobed_reparse() -> None:
|
||||
assert video_range("HDR Vivid") == video_range("vivid") == Video.Range.VIVID
|
||||
for local, remote in ((Video.Range.HDR10, Video.Range.VIVID), (Video.Range.HLG, Video.Range.VIVID)):
|
||||
assert match_track(video("r", range_=remote), [video("l", range_=local)]) is not None
|
||||
assert match_track(video("r", range_=Video.Range.VIVID), [video("l", range_=Video.Range.HDR10P)]) is None
|
||||
|
||||
|
||||
def test_audio_vivid_has_no_music_container(tmp_path: Any) -> None:
|
||||
path = tmp_path / "a.ts"
|
||||
path.write_bytes(b"\x47")
|
||||
track = Audio(url="https://x/a.m3u8", language="zh", codec=Audio.Codec.AV3A)
|
||||
track.path = path
|
||||
with pytest.raises(ValueError, match="Audio Vivid"):
|
||||
track.to_music_container()
|
||||
|
||||
|
||||
def test_probed_tracks_probes_codecless_audio_without_video(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
calls: list[int] = []
|
||||
monkeypatch.setattr(
|
||||
hdr10plus, "probe_audio_codecs", lambda audio, session, *, log, workers: calls.append(len(audio))
|
||||
)
|
||||
service = SimpleNamespace(get_tracks=lambda title: SimpleNamespace(videos=[], audio=[SimpleNamespace(codec=None)]))
|
||||
service.session = None
|
||||
hdr10plus.probed_tracks(service, None, workers=2) # type: ignore[arg-type]
|
||||
assert calls == [1]
|
||||
|
||||
@@ -215,7 +215,7 @@ def test_a_quiet_remote_login_shows_its_last_server_lines_as_a_notice(monkeypatc
|
||||
shown = []
|
||||
monkeypatch.setattr(remote_service, "show_notice", shown.append)
|
||||
monkeypatch.setattr(remote_service.time, "sleep", lambda s: None)
|
||||
logs = iter([["+ IP Region: CA"], ["+ Open URL: https://disneyplus.com/start\n+ Enter pair code: 9835-2503"], []])
|
||||
logs = iter([["+ IP Region: CA"], ["+ Open URL: https://example.test/start\n+ Enter pair code: 9835-2503"], []])
|
||||
statuses = iter(["authenticating", "authenticating", "authenticating", "authenticated"])
|
||||
|
||||
class Client:
|
||||
@@ -236,7 +236,7 @@ def test_a_quiet_remote_login_shows_its_last_server_lines_as_a_notice(monkeypatc
|
||||
|
||||
remote.poll_auth_completion(poll_interval=0)
|
||||
# silent while the remote was still logging; the pair code once it went quiet; cleared when done
|
||||
assert shown == ["+ IP Region: CA\n+ Open URL: https://disneyplus.com/start\n+ Enter pair code: 9835-2503", None]
|
||||
assert shown == ["+ IP Region: CA\n+ Open URL: https://example.test/start\n+ Enter pair code: 9835-2503", None]
|
||||
assert remote.auth_notice is None
|
||||
|
||||
|
||||
|
||||
@@ -103,6 +103,20 @@ def test_sort_videos_resolution_outranks_bitrate() -> None:
|
||||
assert [t.id for t in tracks.videos] == ["lo-bitrate-2160", "hi-bitrate-720"]
|
||||
|
||||
|
||||
def test_sort_videos_orders_crops_of_one_resolution_by_bitrate() -> None:
|
||||
tracks = Tracks()
|
||||
tracks.videos = [
|
||||
make_video("1610-lo", height=1610, bitrate=4_000_000, width=3840),
|
||||
make_video("1606-hi", height=1606, bitrate=15_000_000, width=3840),
|
||||
make_video("1610-hi", height=1610, bitrate=16_000_000, width=3840),
|
||||
make_video("804", height=804, bitrate=20_000_000, width=1920),
|
||||
]
|
||||
|
||||
tracks.sort_videos()
|
||||
|
||||
assert [t.id for t in tracks.videos] == ["1610-hi", "1606-hi", "1610-lo", "804"]
|
||||
|
||||
|
||||
def test_sort_videos_unknown_bitrate_still_wins_on_resolution() -> None:
|
||||
tracks = Tracks()
|
||||
tracks.videos = [
|
||||
|
||||
@@ -952,7 +952,7 @@ async def download(request: web.Request) -> web.Response:
|
||||
- type: array
|
||||
items:
|
||||
type: string
|
||||
description: Audio codec(s) to download (e.g., "AAC" or ["AAC", "EC3"]) - accepts AAC, AC3, DD, EC3, DD+, EAC3, DDP, AC4, AC-4, OPUS, OGG, VORB, VORBIS, DTS, DTSX, DTS-X, DTSHD, DTS-HD, MPEGH, MPEG-H, ALAC, FLAC, MP3 (default - [])
|
||||
description: Audio codec(s) to download (e.g., "AAC" or ["AAC", "EC3"]) - accepts AAC, AC3, DD, EC3, DD+, EAC3, DDP, AC4, AC-4, OPUS, OGG, VORB, VORBIS, DTS, DTSX, DTS-X, DTSHD, DTS-HD, MPEGH, MPEG-H, AV3A, Audio Vivid, ALAC, FLAC, MP3 (default - [])
|
||||
vbitrate:
|
||||
type: integer
|
||||
description: Video bitrate in kbps (default - None)
|
||||
@@ -964,7 +964,7 @@ async def download(request: web.Request) -> web.Response:
|
||||
items:
|
||||
type: string
|
||||
description: >-
|
||||
Video colour range (SDR, HDR10, HDR10+, HLG, DV, HYBRID) (default - ["SDR"]).
|
||||
Video colour range (SDR, HDR10, HDR10+, VIVID, HLG, DV, HYBRID) (default - ["SDR"]).
|
||||
An API key with a server_cdm_max_height for the service cannot set HYBRID.
|
||||
channels:
|
||||
type: number
|
||||
|
||||
@@ -30,6 +30,7 @@ DYNAMIC_RANGE_MAP = {
|
||||
"Dolby Vision": "DV",
|
||||
"HDR10 / HDR10+": "HDR10P",
|
||||
"HDR10 / HDR10": "HDR",
|
||||
"HDR Vivid": "HDRVivid",
|
||||
}
|
||||
AUDIO_CODEC_MAP = {
|
||||
"E-AC-3": "DDP",
|
||||
|
||||
@@ -144,7 +144,7 @@ def quality_parts(tracks: Iterable[AnyTrack]) -> list[str]:
|
||||
groups = [
|
||||
[f"{h}p" for h in heights],
|
||||
[c.value.lower().replace(".", "") for c in codecs],
|
||||
[r.value.lower() for r in ranges],
|
||||
[r.value.lower().replace(" ", "") for r in ranges],
|
||||
[d.name.lower() for d in manifests],
|
||||
list(dict.fromkeys(audio_token(a) for a in audio_sorted)),
|
||||
]
|
||||
|
||||
@@ -190,10 +190,7 @@ def video_range(value: Any) -> Optional[Video.Range]:
|
||||
name = str(value or "").upper()
|
||||
if name in Video.Range.__members__:
|
||||
return Video.Range[name]
|
||||
try:
|
||||
return Video.Range(name)
|
||||
except ValueError:
|
||||
return None
|
||||
return next((r for r in Video.Range if r.value.upper() == name), None)
|
||||
|
||||
|
||||
def int_or_none(value: Any) -> Optional[int]:
|
||||
|
||||
@@ -19,6 +19,7 @@ from unshackle.core.manifests.dash import DASH, MissingLanguageError
|
||||
from unshackle.core.manifests.ism import ISM
|
||||
from unshackle.core.tracks import Audio, Subtitle, Tracks, Video
|
||||
from unshackle.core.tracks.track import Track
|
||||
from unshackle.core.tracks.video import PROBED_RANGES
|
||||
|
||||
log = logging.getLogger("snapshot")
|
||||
|
||||
@@ -273,9 +274,9 @@ def same_bitrate(local: Optional[int], remote: Optional[int]) -> bool:
|
||||
def match_track(remote_track: Track, local_tracks: list) -> Optional[Track]:
|
||||
"""Find the locally-parsed track that matches a remote track by ID or by attributes.
|
||||
|
||||
HDR10 matches HDR10P, because the server's HDR10+ probe relabels a track that the local re-parse calls HDR10.
|
||||
A range matches the ones ``PROBED_RANGES`` lets the server's probe relabel it to, because the local
|
||||
re-parse does not probe.
|
||||
"""
|
||||
hdr10_ranges = {Video.Range.HDR10, Video.Range.HDR10P}
|
||||
remote_id = str(remote_track.id)
|
||||
for lt in local_tracks:
|
||||
if str(lt.id) == remote_id:
|
||||
@@ -292,7 +293,11 @@ def match_track(remote_track: Track, local_tracks: list) -> Optional[Track]:
|
||||
if (
|
||||
lt.width == remote_track.width
|
||||
and lt.height == remote_track.height
|
||||
and (lt.range == remote_track.range or {lt.range, remote_track.range} == hdr10_ranges)
|
||||
and (
|
||||
lt.range == remote_track.range
|
||||
or remote_track.range in PROBED_RANGES.get(lt.range, ())
|
||||
or lt.range in PROBED_RANGES.get(remote_track.range, ())
|
||||
)
|
||||
and same_bitrate(lt.bitrate, remote_track.bitrate)
|
||||
):
|
||||
return lt
|
||||
|
||||
@@ -149,7 +149,7 @@ class Title:
|
||||
if base_layer and base_layer != "DV":
|
||||
context["hdr"] += f".{base_layer}"
|
||||
elif (primary_video_track.hdr_format or "").startswith("HDR Vivid"):
|
||||
context["hdr"] = "HDR"
|
||||
context["hdr"] = DYNAMIC_RANGE_MAP["HDR Vivid"]
|
||||
else:
|
||||
context["hdr"] = DYNAMIC_RANGE_MAP.get(hdr_format, "")
|
||||
if context["hdr"] == "HDR" and any(v.range == Video.Range.HDR10P for v in self.tracks.videos[:1]):
|
||||
|
||||
@@ -57,6 +57,7 @@ class Audio(Track):
|
||||
DTSX = "DTS-X" # https://en.wikipedia.org/wiki/DTS,_Inc.#DTS:X
|
||||
DTSHD = "DTS-HD" # https://en.wikipedia.org/wiki/DTS-HD_Master_Audio
|
||||
MPEGH = "MPEG-H" # https://en.wikipedia.org/wiki/MPEG-H_3D_Audio
|
||||
AV3A = "Audio Vivid" # AVS3-P3 Audio Vivid
|
||||
ALAC = "ALAC" # https://en.wikipedia.org/wiki/Apple_Lossless_Audio_Codec
|
||||
FLAC = "FLAC" # https://en.wikipedia.org/wiki/FLAC
|
||||
MP3 = "MP3" # https://en.wikipedia.org/wiki/MP3
|
||||
@@ -87,6 +88,8 @@ class Audio(Track):
|
||||
return Audio.Codec.DTSHD
|
||||
if mime in MPEGH_SAMPLE_ENTRIES:
|
||||
return Audio.Codec.MPEGH
|
||||
if mime == "av3a":
|
||||
return Audio.Codec.AV3A
|
||||
if mime == "alac":
|
||||
return Audio.Codec.ALAC
|
||||
if mime == "flac":
|
||||
@@ -97,6 +100,15 @@ class Audio(Track):
|
||||
return Audio.Codec.MP3
|
||||
raise ValueError(f"The MIME '{mime}' is not a supported Audio Codec")
|
||||
|
||||
@staticmethod
|
||||
def from_ts(stream_type: int, registration: bytes = b"") -> Optional[Audio.Codec]:
|
||||
"""The codec an MPEG-TS PMT entry names by its registration descriptor or ``stream_type``, else None."""
|
||||
if registration == b"AVSA":
|
||||
return Audio.Codec.AV3A
|
||||
return {0x0F: Audio.Codec.AAC, 0x81: Audio.Codec.AC3, 0x87: Audio.Codec.EC3, 0xD5: Audio.Codec.AV3A}.get(
|
||||
stream_type
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def from_ffprobe(codec_name: str, profile: str = "") -> Optional[Audio.Codec]:
|
||||
"""Map an FFprobe ``codec_name`` to a codec, or None for one core does not name."""
|
||||
@@ -332,10 +344,12 @@ class Audio(Track):
|
||||
"""Remux the track into the container of the codec FFprobe reads from the file, cut to the sample window.
|
||||
|
||||
The declared codec is used only when FFprobe cannot read the file. Every failure raises and keeps the
|
||||
downloaded file.
|
||||
downloaded file. Audio Vivid raises: FFprobe reads it as MP3, and no music container carries it.
|
||||
"""
|
||||
if not self.path or not self.path.exists():
|
||||
raise ValueError("Cannot remux a Track that has not been downloaded.")
|
||||
if self.codec == Audio.Codec.AV3A:
|
||||
raise ValueError(f"No music container carries Audio Vivid track {self.id}")
|
||||
|
||||
stream = self.probe_stream()
|
||||
probed = Audio.Codec.from_ffprobe(stream.get("codec_name", ""), stream.get("profile", ""))
|
||||
|
||||
@@ -342,9 +342,6 @@ class Hybrid:
|
||||
},
|
||||
)
|
||||
self.log.info("✓ Injection Completed")
|
||||
if self.source == ("itunes" or "appletvplus"):
|
||||
Path.unlink(config.directories.temp / "hdr10.mkv")
|
||||
Path.unlink(config.directories.temp / "dv.mkv")
|
||||
|
||||
def extract_stream(self, save_path: Path, type_: str) -> None:
|
||||
output = Path(config.directories.temp / f"{type_}.hevc")
|
||||
|
||||
@@ -354,13 +354,22 @@ class Tracks:
|
||||
def sort_videos(
|
||||
self, by_language: Optional[Sequence[Union[str, Language]]] = None, exact_match: bool = False
|
||||
) -> None:
|
||||
"""Sort video tracks by resolution, 2D before 3D, then bitrate, and optionally language.
|
||||
"""Sort video tracks by standard resolution, 2D before 3D, then bitrate, and optionally language.
|
||||
|
||||
The standard resolution puts a 3840x1606 and a 3840x1610 crop in one 2160p group, ordered by bitrate.
|
||||
A 3D track carries two views, so its higher bitrate would otherwise rank it over the 2D one.
|
||||
"""
|
||||
if not self.videos:
|
||||
return
|
||||
self.videos.sort(key=lambda x: (x.height or 0, not x.stereo, float(x.bitrate or 0.0)), reverse=True)
|
||||
self.videos.sort(
|
||||
key=lambda x: (
|
||||
standard_resolution(x.width, x.height) or 0,
|
||||
not x.stereo,
|
||||
float(x.bitrate or 0.0),
|
||||
x.height or 0,
|
||||
),
|
||||
reverse=True,
|
||||
)
|
||||
for language in reversed(by_language or []):
|
||||
if str(language) in ("all", "best"):
|
||||
language = self._original_language(self.videos, exact_match)
|
||||
|
||||
@@ -91,6 +91,7 @@ class Video(Track):
|
||||
HLG = "HLG" # https://en.wikipedia.org/wiki/Hybrid_log%E2%80%93gamma
|
||||
HDR10 = "HDR10" # https://en.wikipedia.org/wiki/HDR10
|
||||
HDR10P = "HDR10+" # https://en.wikipedia.org/wiki/HDR10%2B
|
||||
VIVID = "HDR Vivid" # CUVA HDR Vivid (T/UWA 005), dynamic metadata over a PQ or HLG base
|
||||
DV = "DV" # https://en.wikipedia.org/wiki/Dolby_Vision
|
||||
HYBRID = "HYBRID" # Selects both HDR10 and DV tracks for hybrid processing with DoviTool
|
||||
|
||||
@@ -857,6 +858,10 @@ class Video(Track):
|
||||
return True
|
||||
|
||||
|
||||
PROBED_RANGES = {
|
||||
Video.Range.HDR10: (Video.Range.HDR10P, Video.Range.VIVID),
|
||||
Video.Range.HLG: (Video.Range.VIVID,),
|
||||
}
|
||||
HDR_TRANSFER = {
|
||||
Video.Range.HDR10: "smpte2084",
|
||||
Video.Range.HDR10P: "smpte2084",
|
||||
|
||||
@@ -67,7 +67,7 @@ def get_local_storage_data(profile_path: Path, hosts: List[str], tmp_dir_path: P
|
||||
cursor = conn.cursor()
|
||||
|
||||
for host in hosts:
|
||||
# Firefox stores LocalStorage as reversed host (e.g., moc.viki.:https:443)
|
||||
# Firefox stores LocalStorage as reversed host (e.g., moc.example.:https:443)
|
||||
# Use strict prefix matching to prevent over-broad harvesting
|
||||
reversed_host = host[::-1]
|
||||
query = "SELECT key, value FROM webappsstore2 WHERE originKey LIKE ?"
|
||||
|
||||
@@ -3,7 +3,7 @@ from __future__ import annotations
|
||||
import logging
|
||||
import struct
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from typing import TYPE_CHECKING, Callable, Optional, Union
|
||||
from typing import TYPE_CHECKING, Callable, Optional, TypeVar, Union
|
||||
from urllib.parse import urljoin
|
||||
|
||||
from requests import Session
|
||||
@@ -15,10 +15,76 @@ from unshackle.core.utils.bitrate import PROBE_TIMEOUT, ranged_get
|
||||
if TYPE_CHECKING:
|
||||
from unshackle.core.service import Service
|
||||
from unshackle.core.titles import Title_T
|
||||
from unshackle.core.tracks import Tracks, Video
|
||||
from unshackle.core.tracks import Audio, Track, Tracks, Video
|
||||
|
||||
PROBE_BYTES = 64 * 1024
|
||||
HDR10PLUS_T35 = b"\xb5\x00\x3c\x00\x01\x04"
|
||||
CUVA_T35 = b"\x26\x00\x04\x00\x05"
|
||||
TS_PACKET = 188
|
||||
T = TypeVar("T")
|
||||
R = TypeVar("R")
|
||||
|
||||
|
||||
def map_probes(probe: Callable[[T], R], items: list[T], workers: int) -> list[R]:
|
||||
"""``probe`` over ``items`` in order, with no more than ``workers`` in flight."""
|
||||
if not items:
|
||||
return []
|
||||
with ThreadPoolExecutor(max_workers=min(workers, len(items))) as executor:
|
||||
return list(executor.map(probe, items))
|
||||
|
||||
|
||||
def is_ts(data: bytes) -> bool:
|
||||
"""Whether the bytes start as MPEG-TS packets."""
|
||||
return data[:1] == b"\x47" and (len(data) <= TS_PACKET or data[TS_PACKET : TS_PACKET + 1] == b"\x47")
|
||||
|
||||
|
||||
def ts_streams(data: bytes) -> Optional[list[tuple[int, bytes]]]:
|
||||
"""The ``stream_type`` and registration format identifier of each PMT entry, or None without a PMT.
|
||||
|
||||
PAT program 0 names the network information table, so it is skipped. Lengths that run past the packet
|
||||
are cut to it, so bad bytes give fewer entries, never an exception.
|
||||
"""
|
||||
# reads only a PAT and PMT that fit in one packet, as packagers write them; reassemble
|
||||
# sections across packets if a source with a longer PMT appears
|
||||
|
||||
def pid(b: bytes, i: int) -> int:
|
||||
return ((b[i] & 0x1F) << 8) | b[i + 1]
|
||||
|
||||
def length(b: bytes, i: int) -> int:
|
||||
return ((b[i] & 0x0F) << 8) | b[i + 1]
|
||||
|
||||
def section(packet: bytes, table_id: int) -> bytes:
|
||||
"""The packet's ``table_id`` section up to its CRC, or b"" when the packet holds no such section."""
|
||||
adaptation = (packet[3] >> 4) & 3
|
||||
payload = packet[5 + packet[4] if adaptation & 2 else 4 :] if adaptation & 1 else b""
|
||||
body = payload[1 + payload[0] :] if payload else b""
|
||||
body = body[: max(0, 3 + length(body, 1) - 4)] if len(body) >= 3 else b""
|
||||
return body if len(body) >= 12 and body[0] == table_id else b""
|
||||
|
||||
pmt_pid = None
|
||||
for pos in range(0, len(data) - TS_PACKET + 1, TS_PACKET):
|
||||
packet = data[pos : pos + TS_PACKET]
|
||||
if packet[0] != 0x47 or not packet[1] & 0x40:
|
||||
continue
|
||||
if pid(packet, 1) == 0 and pmt_pid is None:
|
||||
pat = section(packet, 0x00)
|
||||
pmt_pid = next((pid(pat, k + 2) for k in range(8, len(pat) - 3, 4) if pat[k] or pat[k + 1]), None)
|
||||
elif pid(packet, 1) == pmt_pid:
|
||||
pmt = section(packet, 0x02)
|
||||
if not pmt:
|
||||
return None
|
||||
i, streams = 12 + length(pmt, 10), []
|
||||
while i + 5 <= len(pmt):
|
||||
info_end = min(i + 5 + length(pmt, i + 3), len(pmt))
|
||||
registration, j = b"", i + 5
|
||||
while j + 2 <= info_end:
|
||||
if pmt[j] == 0x05 and j + 6 <= info_end:
|
||||
registration = pmt[j + 2 : j + 6]
|
||||
j += 2 + pmt[j + 1]
|
||||
streams.append((pmt[i], registration))
|
||||
i = info_end
|
||||
return streams
|
||||
return None
|
||||
|
||||
|
||||
def sei_number(rbsp: bytes, i: int) -> tuple[Optional[int], int]:
|
||||
@@ -32,8 +98,8 @@ def sei_number(rbsp: bytes, i: int) -> tuple[Optional[int], int]:
|
||||
return value + rbsp[i], i + 1
|
||||
|
||||
|
||||
def sei_has_hdr10plus(nal: bytes) -> bool:
|
||||
"""Whether an HEVC SEI NAL holds an ST 2094-40 message."""
|
||||
def sei_has_t35(nal: bytes, t35: bytes = HDR10PLUS_T35) -> bool:
|
||||
"""Whether an HEVC SEI NAL holds an ST 2094-40 message, or the T.35 message that ``t35`` starts."""
|
||||
rbsp = remove_emulation_prevention(nal[2:])
|
||||
i = 0
|
||||
while i < len(rbsp) and rbsp[i] != 0x80:
|
||||
@@ -41,15 +107,17 @@ def sei_has_hdr10plus(nal: bytes) -> bool:
|
||||
payload_size, i = sei_number(rbsp, i)
|
||||
if payload_type is None or payload_size is None:
|
||||
return False
|
||||
if payload_type == 4 and rbsp[i : i + len(HDR10PLUS_T35)] == HDR10PLUS_T35:
|
||||
if payload_type == 4 and rbsp[i : i + len(t35)] == t35:
|
||||
return True
|
||||
i += payload_size
|
||||
return False
|
||||
|
||||
|
||||
def fragment_has_hdr10plus(data: bytes) -> Optional[bool]:
|
||||
def fragment_has_hdr10plus(data: bytes, t35: bytes = HDR10PLUS_T35) -> Optional[bool]:
|
||||
"""Whether the first sample of an fMP4 fragment carries HDR10+ SEI, or None when the bytes cannot tell.
|
||||
|
||||
``t35`` picks another T.35 message, such as ``CUVA_T35`` for HDR Vivid.
|
||||
|
||||
The walk ends at the first VCL NAL, because a prefix SEI comes before it. CENC and cbcs keep
|
||||
non-VCL NALs and every NAL length in the clear, so this needs no content key. A sample that is
|
||||
encrypted whole can read as False as well as None; both leave the label alone.
|
||||
@@ -69,13 +137,13 @@ def fragment_has_hdr10plus(data: bytes) -> Optional[bool]:
|
||||
if nal_type == 39:
|
||||
if pos + 4 + size > len(data):
|
||||
return None
|
||||
if sei_has_hdr10plus(data[pos + 4 : pos + 4 + size]):
|
||||
if sei_has_t35(data[pos + 4 : pos + 4 + size], t35):
|
||||
return True
|
||||
pos += 4 + size
|
||||
return None
|
||||
|
||||
|
||||
def first_segment(track: "Video", session: Union[Session, RnetSession]) -> Optional[tuple[str, int]]:
|
||||
def first_segment(track: "Track", session: Union[Session, RnetSession]) -> Optional[tuple[str, int]]:
|
||||
"""The URL and start offset of the track's first media segment, looked up without a full parse."""
|
||||
from unshackle.core.manifests import DASH
|
||||
from unshackle.core.tracks.track import Track
|
||||
@@ -115,8 +183,8 @@ def first_segment(track: "Video", session: Union[Session, RnetSession]) -> Optio
|
||||
return None
|
||||
|
||||
|
||||
def first_fragment(track: "Video", session: Union[Session, RnetSession]) -> Optional[bytes]:
|
||||
"""The first PROBE_BYTES of the track's first media fragment."""
|
||||
def first_fragment(track: "Track", session: Union[Session, RnetSession]) -> Optional[bytes]:
|
||||
"""The first PROBE_BYTES of the track's first media fragment, or of its first segment when that is MPEG-TS."""
|
||||
segment = first_segment(track, session)
|
||||
if not segment:
|
||||
return None
|
||||
@@ -127,6 +195,8 @@ def first_fragment(track: "Video", session: Union[Session, RnetSession]) -> Opti
|
||||
data = ranged_get(url, session, f"bytes={start}-{start + PROBE_BYTES - 1}")
|
||||
if not data:
|
||||
return None
|
||||
if is_ts(data):
|
||||
return data
|
||||
skip = 0
|
||||
for kind, _, _, end in iter_boxes(data, 0, len(data)):
|
||||
if kind in (b"styp", b"moof", b"mdat"):
|
||||
@@ -138,7 +208,7 @@ def first_fragment(track: "Video", session: Union[Session, RnetSession]) -> Opti
|
||||
return None
|
||||
|
||||
|
||||
def probe_hdr10plus(
|
||||
def probe_dynamic_hdr(
|
||||
videos: list["Video"],
|
||||
session: Union[Session, RnetSession],
|
||||
*,
|
||||
@@ -146,16 +216,19 @@ def probe_hdr10plus(
|
||||
workers: int,
|
||||
on_start: Optional[Callable[[], None]] = None,
|
||||
) -> None:
|
||||
"""Relabel HEVC HDR10 tracks whose first sample carries HDR10+ SEI as HDR10P, and flag such DV tracks.
|
||||
"""Relabel HEVC tracks by the dynamic HDR metadata of their first sample, and flag DV tracks with HDR10+.
|
||||
|
||||
A DV track keeps its DV range and gets ``hdr10plus`` set, so selection and hybrid pairing are unchanged.
|
||||
HDR10 and DV are separate ladders. A ladder with one track per resolution is sampled at both ends
|
||||
first. Two tracks at one resolution can be an HDR10 and an HDR10+ encode side by side, so such a
|
||||
ladder is probed whole. The rules are in the ``hdr10plus_probe`` section of
|
||||
docs/reference/configuration/download.md. A failed probe never changes a label. ``workers`` caps the
|
||||
probes in flight, and ``on_start`` runs once before the first.
|
||||
HDR10+ SEI makes an HDR10 track HDR10P. CUVA SEI, or the ``cuvv`` registration of an MPEG-TS stream,
|
||||
makes an HDR10 or HLG track VIVID; ``PROBED_RANGES`` holds which label each range takes. A DV track
|
||||
keeps its DV range and gets ``hdr10plus`` set, so selection and hybrid pairing are unchanged. HDR10,
|
||||
HLG and DV are separate ladders. A ladder with one track per resolution is sampled at both ends first.
|
||||
Two tracks at one resolution can be an HDR10 and an HDR10+ encode side by side, so such a ladder is
|
||||
probed whole. The rules are in the ``hdr10plus_probe`` section of docs/reference/configuration/download.md.
|
||||
A failed probe never changes a label. ``workers`` caps the probes in flight, and ``on_start`` runs once
|
||||
before the first.
|
||||
"""
|
||||
from unshackle.core.tracks import Video
|
||||
from unshackle.core.tracks.video import PROBED_RANGES
|
||||
|
||||
ladders = [
|
||||
sorted(
|
||||
@@ -163,26 +236,32 @@ def probe_hdr10plus(
|
||||
key=lambda v: v.bitrate or 0,
|
||||
reverse=True,
|
||||
)
|
||||
for range_ in (Video.Range.HDR10, Video.Range.DV)
|
||||
for range_ in (*PROBED_RANGES, Video.Range.DV)
|
||||
]
|
||||
if any(ladders) and on_start:
|
||||
on_start()
|
||||
|
||||
def probe(video: "Video") -> Optional[bool]:
|
||||
def probe(video: "Video") -> Union["Video.Range", bool, None]:
|
||||
"""The range the first segment proves (HDR10P or VIVID), False for plain HDR10, None when it cannot tell."""
|
||||
found: Union[Video.Range, bool, None] = None
|
||||
try:
|
||||
data = first_fragment(video, video.session or session)
|
||||
found = fragment_has_hdr10plus(data) if data else None
|
||||
if data and is_ts(data):
|
||||
if any(registration == b"cuvv" for _, registration in ts_streams(data) or []):
|
||||
found = Video.Range.VIVID
|
||||
elif data and fragment_has_hdr10plus(data, CUVA_T35):
|
||||
found = Video.Range.VIVID
|
||||
elif data:
|
||||
hdr10plus = fragment_has_hdr10plus(data)
|
||||
found = Video.Range.HDR10P if hdr10plus else hdr10plus
|
||||
except Exception as e:
|
||||
log.debug(f"{video.id}: HDR10+ probe failed ({e})")
|
||||
log.debug(f"{video.id}: dynamic HDR probe failed ({e})")
|
||||
return None
|
||||
log.debug(f"{video.id}: HDR10+ probe {found}")
|
||||
log.debug(f"{video.id}: dynamic HDR probe {found}")
|
||||
return found
|
||||
|
||||
def probe_all(tracks: list["Video"]) -> dict[int, Optional[bool]]:
|
||||
if not tracks:
|
||||
return {}
|
||||
with ThreadPoolExecutor(max_workers=min(workers, len(tracks))) as executor:
|
||||
return dict(zip(map(id, tracks), executor.map(probe, tracks)))
|
||||
def probe_all(tracks: list["Video"]) -> dict[int, Union["Video.Range", bool, None]]:
|
||||
return dict(zip(map(id, tracks), map_probes(probe, tracks, workers)))
|
||||
|
||||
plans: list[tuple[list["Video"], list["Video"], list["Video"]]] = []
|
||||
for candidates in ladders:
|
||||
@@ -193,23 +272,39 @@ def probe_hdr10plus(
|
||||
|
||||
# one round for the samples of every ladder, then one for the rest of each mixed ladder
|
||||
found = probe_all([v for _, samples, _ in plans for v in samples])
|
||||
labels = [{found[id(s)] for s in samples} for _, samples, _ in plans]
|
||||
found.update(
|
||||
probe_all(
|
||||
[
|
||||
v
|
||||
for _, samples, rest in plans
|
||||
if any(found[id(s)] for s in samples) and not all(found[id(s)] for s in samples)
|
||||
for v in rest
|
||||
]
|
||||
)
|
||||
probe_all([v for (_, _, rest), seen in zip(plans, labels) if any(seen) and len(seen) > 1 for v in rest])
|
||||
)
|
||||
for candidates, samples, _ in plans:
|
||||
hits = candidates if all(found[id(s)] for s in samples) else [v for v in candidates if found.get(id(v))]
|
||||
for video in hits:
|
||||
for (candidates, _, _), seen in zip(plans, labels):
|
||||
whole = next(iter(seen)) if len(seen) == 1 else None
|
||||
for video in candidates:
|
||||
label = found.get(id(video), whole)
|
||||
if video.range == Video.Range.DV:
|
||||
video.hdr10plus = True
|
||||
else:
|
||||
video.range = Video.Range.HDR10P
|
||||
video.hdr10plus = video.hdr10plus or label == Video.Range.HDR10P
|
||||
elif isinstance(label, Video.Range) and label in PROBED_RANGES.get(video.range, ()):
|
||||
video.range = label
|
||||
|
||||
|
||||
def probe_audio_codecs(
|
||||
audio: list["Audio"], session: Union[Session, RnetSession], *, log: logging.Logger, workers: int
|
||||
) -> None:
|
||||
"""Name the codec of each audio track that has none from the PMT of its MPEG-TS first segment."""
|
||||
from unshackle.core.tracks import Audio
|
||||
|
||||
def probe(track: "Audio") -> None:
|
||||
try:
|
||||
data = first_fragment(track, track.session or session)
|
||||
streams = (ts_streams(data) if data and is_ts(data) else None) or []
|
||||
except Exception as e:
|
||||
log.debug(f"{track.id}: audio codec probe failed ({e})")
|
||||
return
|
||||
codec = next(filter(None, (Audio.Codec.from_ts(*stream) for stream in streams)), None)
|
||||
log.debug(f"{track.id}: audio codec probe {codec}")
|
||||
if codec:
|
||||
track.codec = codec
|
||||
|
||||
map_probes(probe, [a for a in audio if not a.codec], workers)
|
||||
|
||||
|
||||
def probed_tracks(
|
||||
@@ -219,26 +314,23 @@ def probed_tracks(
|
||||
workers: Optional[int] = None,
|
||||
on_probe: Optional[Callable[[], None]] = None,
|
||||
) -> "Tracks":
|
||||
"""The title's tracks from ``service.get_tracks``, with HDR10+ found in tracks labelled HDR10.
|
||||
"""The title's tracks from ``service.get_tracks``, with HDR10+ and HDR Vivid found in HDR10 and HLG tracks.
|
||||
|
||||
``workers`` defaults to ``dl.workers`` from the config. A remote service's tracks come back from a
|
||||
server that already probed them.
|
||||
Audio with no codec gets one from the MPEG-TS ``stream_type``. ``workers`` defaults to ``dl.workers`` from
|
||||
the config. A remote service's tracks come back from a server that already probed them.
|
||||
"""
|
||||
from unshackle.core.config import config
|
||||
from unshackle.core.downloaders import default_max_workers
|
||||
from unshackle.core.remote_service import RemoteService
|
||||
|
||||
tracks = service.get_tracks(title)
|
||||
if config.hdr10plus_probe and tracks.videos and not isinstance(service, RemoteService):
|
||||
codecless_audio = any(not a.codec for a in tracks.audio)
|
||||
if config.hdr10plus_probe and (tracks.videos or codecless_audio) and not isinstance(service, RemoteService):
|
||||
# int() as click does for --workers, so a quoted dl.workers works on the REST path too
|
||||
workers = workers or int(config.dl.get("workers") or 0) or default_max_workers()
|
||||
if not isinstance(workers, int) or isinstance(workers, bool) or workers < 1:
|
||||
raise ValueError(f"workers must be a whole number of 1 or more, not {workers!r}")
|
||||
probe_hdr10plus(
|
||||
tracks.videos,
|
||||
service.session,
|
||||
log=logging.getLogger("hdr10plus"),
|
||||
workers=workers,
|
||||
on_start=on_probe,
|
||||
)
|
||||
log = logging.getLogger("hdr10plus")
|
||||
probe_dynamic_hdr(tracks.videos, service.session, log=log, workers=workers, on_start=on_probe)
|
||||
probe_audio_codecs(tracks.audio, service.session, log=log, workers=workers)
|
||||
return tracks
|
||||
|
||||
Reference in New Issue
Block a user