diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index d3dba5cd..09473416 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -97,9 +97,15 @@ unshackle dl [OPTIONS] SERVICE [SERVICE ARGS...] | `--require-subs` | - | Required subtitle langs; keeps **all** subs only if these exist. **Cannot combine with `--s-lang`.** | | `-fs`, `--forced-subs` | off | Include forced subtitle tracks. | | `-fsl`, `--forced-s-lang` | none | Language(s) wanted for forced subtitles; implies `-fs`. | -| `--exact-lang` | off | Exact matching only: `-l es-419` matches `es-419`, not `es-ES`. | +| `--exact-lang` | off | Exact matching only: `-l es-419` matches `es-419`, not `es-ES`. Applies to selection and to sort order. | | `--sub-format` | - | Output subtitle format (`SRT`/`srt`, `VTT`/`webvtt`, `ASS`/`ssa`, `TTML`, `SMI`, ...), or `original` to keep the source format. | +!!! tip "These flags select languages, they do not order them" + Naming languages with `-l` or `-sl` removes the others. To keep every language but put + your preferred ones first, use `audio.language_priority` and + [`subtitle.language_priority`](../reference/configuration.md#subtitle) in your + configuration file. + The special language tokens `orig`, `all`, and `best` are honoured everywhere a language is expected. ### Title & episode selection diff --git a/docs/guide/downloading.md b/docs/guide/downloading.md index 8959e208..6686a9d4 100644 --- a/docs/guide/downloading.md +++ b/docs/guide/downloading.md @@ -254,6 +254,26 @@ it is used. You can override each stream type independently: unshackle dl -al en EXAMPLE 81234567 ``` +### Sort order + +`-l` and `-sl` **select** languages: naming some removes the rest. To keep every language +but decide which ones come first, set a priority list in your configuration file: + +```yaml title="unshackle.yaml" +audio: + language_priority: [orig, en] +subtitle: + group_by: language + language_priority: [en, es, fr] +``` + +Subtitles then start with English, Spanish, and French. The languages you leave out follow +alphabetically, after the title's original language. Nothing is removed. Audio works the +same way, but the languages you leave out keep their bitrate and codec order instead of an +alphabetical one. + +See [`subtitle.language_priority`](../reference/configuration.md#subtitle) for the details. + ### Exact vs fuzzy matching By default language matching is fuzzy: `-l en` also accepts `en-US`, `en-GB`, etc. (up diff --git a/docs/guide/subtitles.md b/docs/guide/subtitles.md index d272445a..62d43835 100644 --- a/docs/guide/subtitles.md +++ b/docs/guide/subtitles.md @@ -376,6 +376,8 @@ subtitle: | `convert_before_strip` | `true` | `true` / `false` | Convert a subtitle to SRT before stripping SDH (for non-SubtitleEdit engines). | | `output_mode` | `mux` | `mux`, `sidecar`, `both` | Whether subtitles are embedded, written as sidecars, or both. | | `sidecar_format` | `srt` | a format name (e.g. `srt`, `vtt`) or `original` | Format used for sidecar files. | +| `group_by` | `type` | `type`, `language` | Whether all tracks of one type stay together, or each language stays next to its own variants. | +| `language_priority` | *(unset)* | a list of languages | Languages to sort to the top. The rest keep the default order. Nothing is removed. | !!! tip "SubtitleEdit is optional but recommended" Several high-fidelity paths (the best-quality conversions and the SubtitleEdit SDH diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 4ededb9c..cab4e4ff 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -378,6 +378,7 @@ written as sidecar files. See [Subtitles](../guide/subtitles.md) for the full gu | `sidecar_format` | str | `"srt"` | Format for sidecar files when `output_mode: sidecar`. | | `type_priority` | list | *(unset)* | Ordered ranking of subtitle types (`forced`, `normal`, `sdh`; CC counts as SDH). Types you leave out fall to the end. | | `group_by` | str | `"type"` | Which key groups the subtitle tracks. `type` puts all forced tracks together, then all normal, then all SDH. `language` keeps each language next to its own variants. | +| `language_priority` | list | *(unset)* | Languages to sort to the top, in the order you give. The rest keep the default order: original language first, then alphabetical. Nothing is removed. | ```yaml subtitle: @@ -405,6 +406,33 @@ subtitle: `type_priority` still sets the order of the variants, so with `group_by: language` it controls the order inside each language. +!!! note "`language_priority` sorts languages, it does not select them" + By default the title's original language comes first and every other language follows + alphabetically. For an English title that gives English, Arabic, Bulgarian, and the + rest. Give `language_priority` the languages you want first: + + ```yaml + subtitle: + group_by: language + language_priority: [en, es, fr] + ``` + + | | Result for an English title | + |---|---| + | Default | English, Arabic, Bulgarian, Spanish, French, Japanese | + | With the list above | English, Spanish, French, Arabic, Bulgarian, Japanese | + + This is only an order. Every other language is still downloaded, which is what makes + it different from `-sl`: `-sl en,es,fr` **removes** all other subtitles. + + Use `orig` in the list for the title's original language. A base tag also covers its + regional forms, so `en` sorts `en-US` and `en-GB`, but `zh` does **not** cover + `zh-Hant`, which you must name. The sort ignores entries that match no track, and it + ignores entries that are not valid language tags. + + `--exact-lang` applies here too: with it, `en` sorts only `en`, and `en-US` falls to + the alphabetical tail with the rest. Name each tag you want when you use that flag. + !!! note "`type_priority` also picks the default subtitle" At mux time the first track in the default subtitle language gets the `default` flag. The built-in order is `forced`, `normal`, `sdh`, which means a forced track becomes @@ -421,13 +449,17 @@ subtitle: - **Type:** `dict`  ยท  **Default:** `{}` Audio handling options. The most notable sub-key is **`codec_priority`**, an ordered list of -audio codecs used to break ties in track selection. +audio codecs used to break ties in track selection. **`language_priority`** is an ordered list +of languages that sort to the top of the audio tracks. ```yaml audio: codec_priority: - EAC3 - AC3 + language_priority: + - orig + - en ``` !!! note "`codec_priority` is a soft priority: nothing is dropped" diff --git a/tests/lang_selection/test_matching_languages.py b/tests/lang_selection/test_matching_languages.py new file mode 100644 index 00000000..52a6eb19 --- /dev/null +++ b/tests/lang_selection/test_matching_languages.py @@ -0,0 +1,33 @@ +from __future__ import annotations + +from typing import Sequence + +import pytest + +from unshackle.core.utilities import matching_languages + +POOL = ["ar", "en", "en-GB", "en-US", "ja"] + + +@pytest.mark.parametrize( + ("language", "available", "exact", "expected"), + [ + ("en", POOL, False, {"en", "en-GB", "en-US"}), + # CLDR rates en/en-US at distance 0, so exact mode leans on the RFC 4647 string preference + ("en", POOL, True, {"en"}), + ("en-GB", POOL, True, {"en-GB"}), + # no string-equal tag, so exact mode falls back to the distance match + ("zh", ["ar", "cmn", "ja"], True, {"cmn"}), + ("en", ["ar", "ja"], False, set()), + ("en", [], False, set()), + ("en", ["ar", None, "en"], True, {"en"}), + # unparseable tags (config typos) must match nothing, never raise + ("engrish", POOL, False, set()), + ("engrish", POOL, True, set()), + ("", POOL, False, set()), + ("", POOL, True, set()), + ("f", POOL, False, set()), + ], +) +def test_matching_languages(language: str, available: Sequence[str], exact: bool, expected: set[str]) -> None: + assert matching_languages(language, available, exact) == expected diff --git a/tests/lang_selection/test_resolve_sort_langs.py b/tests/lang_selection/test_resolve_sort_langs.py new file mode 100644 index 00000000..cb396650 --- /dev/null +++ b/tests/lang_selection/test_resolve_sort_langs.py @@ -0,0 +1,24 @@ +from __future__ import annotations + +from typing import Optional, Sequence + +import pytest +from langcodes import Language + +from unshackle.core.utilities import resolve_sort_langs + + +@pytest.mark.parametrize( + ("tokens", "original", "expected"), + [ + (["orig", "en", "es"], "ja", ["ja", "en", "es"]), + (["orig", "en", "es"], None, ["en", "es"]), + (["orig"], Language.get("pt-BR"), ["pt-BR"]), + (["en", "orig"], "en", ["en"]), + (["en", "es", "en", "fr"], None, ["en", "es", "fr"]), + (["all", "best", "en"], "ja", ["all", "best", "en"]), + ([], "ja", []), + ], +) +def test_resolve_sort_langs(tokens: Sequence[str], original: Optional[str], expected: list[str]) -> None: + assert resolve_sort_langs(tokens, original) == expected diff --git a/tests/tracks/test_subtitle_sort.py b/tests/tracks/test_subtitle_sort.py index d114eae0..c583b494 100644 --- a/tests/tracks/test_subtitle_sort.py +++ b/tests/tracks/test_subtitle_sort.py @@ -3,8 +3,8 @@ import pytest from unshackle.core.tracks import Subtitle, Tracks -def sub(lang: str, forced: bool = False, sdh: bool = False, cc: bool = False) -> Subtitle: - return Subtitle( +def sub(lang: str, forced: bool = False, sdh: bool = False, cc: bool = False, orig: bool = False) -> Subtitle: + track = Subtitle( url=f"https://example.test/{lang}-{forced}-{sdh}-{cc}", language=lang, codec=Subtitle.Codec.SubRip, @@ -12,6 +12,8 @@ def sub(lang: str, forced: bool = False, sdh: bool = False, cc: bool = False) -> sdh=sdh, cc=cc, ) + track.is_original_lang = orig + return track def shape(tracks: Tracks) -> list[tuple[str, str]]: @@ -135,3 +137,82 @@ def test_language_grouping_keeps_regional_variants_adjacent_through_sections() - ("fi", "normal"), ("fi", "sdh"), ] + + +def langs(tracks: Tracks) -> list[str]: + return [str(s.language) for s in tracks.subtitles] + + +@pytest.fixture +def many() -> Tracks: + t = Tracks() + t.add([sub(lang, orig=lang == "ja") for lang in ("ar", "bg", "en", "es", "fr", "ja")]) + return t + + +def test_priority_orders_listed_then_rest_alphabetically(many: Tracks) -> None: + many.sort_subtitles(by_language=["en", "es", "fr"], group_by="language") + assert langs(many) == ["en", "es", "fr", "ar", "bg", "ja"] + + +def test_priority_outranks_the_original_language(many: Tracks) -> None: + # "all" resolves to the original language (ja), which must stay below the named ones + many.sort_subtitles(by_language=["en", "es", "all"], group_by="language") + assert langs(many) == ["en", "es", "ja", "ar", "bg", "fr"] + + +def test_priority_matches_regional_variants() -> None: + t = Tracks() + t.add([sub("ar"), sub("en-US"), sub("es-419"), sub("ja")]) + t.sort_subtitles(by_language=["en", "es"], group_by="language") + assert langs(t) == ["en-US", "es-419", "ar", "ja"] + + +def test_priority_overlapping_entries_are_harmless() -> None: + t = Tracks() + t.add([sub("ar"), sub("en-GB"), sub("en-US"), sub("es"), sub("ja")]) + t.sort_subtitles(by_language=["en", "en-US", "es"], group_by="language") + assert langs(t) == ["en-GB", "en-US", "es", "ar", "ja"] + + +def test_no_priority_leaves_order_unchanged(many: Tracks) -> None: + many.sort_subtitles(by_language=[], group_by="language") + assert langs(many) == ["ar", "bg", "en", "es", "fr", "ja"] + + +def test_all_alone_still_floats_the_original_language(many: Tracks) -> None: + many.sort_subtitles(by_language=["all"], group_by="language") + assert langs(many) == ["ja", "ar", "bg", "en", "es", "fr"] + + +@pytest.fixture +def regionals() -> Tracks: + t = Tracks() + t.add([sub("ar"), sub("en"), sub("en-GB"), sub("en-US"), sub("ja")]) + return t + + +def test_fuzzy_priority_floats_regional_variants(regionals: Tracks) -> None: + regionals.sort_subtitles(by_language=["en"], group_by="language") + assert langs(regionals) == ["en", "en-GB", "en-US", "ar", "ja"] + + +def test_exact_priority_floats_only_the_named_tag(regionals: Tracks) -> None: + # CLDR rates en/en-US at distance 0, so exact mode needs the RFC 4647 string preference + # to hold en-US back; without it en-US floats and en-GB does not. + regionals.sort_subtitles(by_language=["en"], group_by="language", exact_match=True) + assert langs(regionals) == ["en", "ar", "en-GB", "en-US", "ja"] + + +def test_exact_priority_still_falls_back_when_no_tag_is_string_equal() -> None: + t = Tracks() + t.add([sub("ar"), sub("cmn"), sub("ja")]) + t.sort_subtitles(by_language=["zh"], group_by="language", exact_match=True) + assert langs(t) == ["cmn", "ar", "ja"] + + +@pytest.mark.parametrize("exact", [False, True]) +def test_unparseable_priority_entry_is_ignored(many: Tracks, exact: bool) -> None: + # a config typo in language_priority must not crash the sort + many.sort_subtitles(by_language=["en", "engrish", "f", ""], group_by="language", exact_match=exact) + assert langs(many) == ["en", "ar", "bg", "es", "fr", "ja"] diff --git a/unshackle/commands/dl.py b/unshackle/commands/dl.py index b4b31193..56b1331b 100644 --- a/unshackle/commands/dl.py +++ b/unshackle/commands/dl.py @@ -83,6 +83,7 @@ from unshackle.core.utilities import ( is_exact_match, keep_forced_subtitle, log_event, + resolve_sort_langs, suggest_font_packages, time_elapsed_since, ) @@ -1859,8 +1860,12 @@ class dl: song.tracks.add(service.get_tracks(song), warn_only=True) song.tracks.chapters = service.get_chapters(song) song.tracks.sort_audio( - by_language=a_lang or lang, + by_language=resolve_sort_langs( + [*(config.audio.get("language_priority") or []), *(a_lang or lang)], + song.language, + ), codec_priority=config.audio.get("codec_priority"), + exact_match=exact_lang, ) select_music_audio(song) if not song.tracks.audio: @@ -2389,39 +2394,23 @@ class dl: ) with console.status("Sorting tracks by language and bitrate...", spinner="dots"): - video_sort_lang = v_lang or lang - processed_video_sort_lang = [] - for language in video_sort_lang: - if language == "orig": - if title.language: - orig_lang = str(title.language) - if orig_lang not in processed_video_sort_lang: - processed_video_sort_lang.append(orig_lang) - else: - if language not in processed_video_sort_lang: - processed_video_sort_lang.append(language) + audio_priority = config.audio.get("language_priority") or [] + subtitle_priority = config.subtitle.get("language_priority") or [] - audio_sort_lang = a_lang or lang - processed_audio_sort_lang = [] - for language in audio_sort_lang: - if language == "orig": - if title.language: - orig_lang = str(title.language) - if orig_lang not in processed_audio_sort_lang: - processed_audio_sort_lang.append(orig_lang) - else: - if language not in processed_audio_sort_lang: - processed_audio_sort_lang.append(language) - - title.tracks.sort_videos(by_language=processed_video_sort_lang) + title.tracks.sort_videos( + by_language=resolve_sort_langs(v_lang or lang, title.language), + exact_match=exact_lang, + ) title.tracks.sort_audio( - by_language=processed_audio_sort_lang, + by_language=resolve_sort_langs([*audio_priority, *(a_lang or lang)], title.language), codec_priority=config.audio.get("codec_priority"), + exact_match=exact_lang, ) title.tracks.sort_subtitles( - by_language=s_lang, + by_language=resolve_sort_langs([*subtitle_priority, *s_lang], title.language), type_priority=config.subtitle.get("type_priority"), group_by=config.subtitle.get("group_by"), + exact_match=exact_lang, ) if list_: diff --git a/unshackle/core/tracks/tracks.py b/unshackle/core/tracks/tracks.py index 02bb6d9d..268fe3e3 100644 --- a/unshackle/core/tracks/tracks.py +++ b/unshackle/core/tracks/tracks.py @@ -23,7 +23,7 @@ from unshackle.core.tracks.chapters import Chapter, Chapters from unshackle.core.tracks.subtitle import Subtitle from unshackle.core.tracks.track import Track from unshackle.core.tracks.video import Video -from unshackle.core.utilities import is_close_match, log_event, sanitize_filename +from unshackle.core.utilities import is_close_match, log_event, matching_languages, sanitize_filename from unshackle.core.utils.collections import as_list, flatten @@ -243,7 +243,9 @@ class Tracks: if duplicates: log.debug(f" - Found and skipped {duplicates} duplicate tracks...") - def sort_videos(self, by_language: Optional[Sequence[Union[str, Language]]] = None) -> None: + def sort_videos( + self, by_language: Optional[Sequence[Union[str, Language]]] = None, exact_match: bool = False + ) -> None: """Sort video tracks by resolution then bitrate, and optionally language.""" if not self.videos: return @@ -255,13 +257,15 @@ class Tracks: language = next((x.language for x in self.videos if x.is_original_lang), "") if not language: continue + wanted = matching_languages(language, [x.language for x in self.videos], exact_match) self.videos.sort(key=lambda x: str(x.language)) - self.videos.sort(key=lambda x: not is_close_match(language, [x.language])) + self.videos.sort(key=lambda x: str(x.language) not in wanted) def sort_audio( self, by_language: Optional[Sequence[Union[str, Language]]] = None, codec_priority: Optional[Sequence[str]] = None, + exact_match: bool = False, ) -> None: """Sort audio tracks by bitrate, codec priority, Atmos, descriptive, and optionally language.""" if not self.audio: @@ -283,13 +287,15 @@ class Tracks: language = next((x.language for x in self.audio if x.is_original_lang), "") if not language: continue - self.audio.sort(key=lambda x: not is_close_match(language, [x.language])) + wanted = matching_languages(language, [x.language for x in self.audio], exact_match) + self.audio.sort(key=lambda x: str(x.language) not in wanted) def sort_subtitles( self, by_language: Optional[Sequence[Union[str, Language]]] = None, type_priority: Optional[Sequence[str]] = None, group_by: Optional[str] = None, + exact_match: bool = False, ) -> None: """ Sort subtitle tracks by various track attributes to a common P2P standard. @@ -313,6 +319,9 @@ class Tracks: then every normal, then every SDH, each block ascending by language. "language" groups by language instead, so Finnish sits next to Finnish SDH, with the Type Order applied inside each language. + + exact_match makes a by_language entry sort only its own tag. By default "en" also + sorts "en-US" and "en-GB". """ if not self.subtitles: return @@ -346,7 +355,8 @@ class Tracks: language = next((x.language for x in self.subtitles if x.is_original_lang), "") if not language: continue - self.subtitles.sort(key=lambda x: is_close_match(language, [x.language]), reverse=True) + wanted = matching_languages(language, [x.language for x in self.subtitles], exact_match) + self.subtitles.sort(key=lambda x: str(x.language) in wanted, reverse=True) def select_video(self, x: Callable[[Video], bool]) -> None: self.videos = list(filter(x, self.videos)) diff --git a/unshackle/core/utilities.py b/unshackle/core/utilities.py index 3b008ce3..90c5e500 100644 --- a/unshackle/core/utilities.py +++ b/unshackle/core/utilities.py @@ -25,6 +25,7 @@ import pycountry from construct import ValidationError from fontTools import ttLib from langcodes import Language, closest_match +from langcodes.tag_parser import LanguageTagError from pymp4.parser import Box from unidecode import unidecode @@ -195,6 +196,49 @@ def as_requested(tokens: Sequence[str], orig_token: Optional[str]) -> str: return ", ".join("orig" if tok == orig_token else tok for tok in tokens) +def matching_languages( + language: Union[str, Language], + available: Sequence[Union[str, Language, None]], + exact: bool = False, +) -> set[str]: + """ + The languages out of `available` that `language` asks for, by the rules Tracks.by_language uses. + + In exact mode CLDR tag distance is not sufficient on its own: it rates a base language and + its "paradigm" regional variant as the same language (distance 0 for en/en-US, pt/pt-BR), so + it cannot separate en-US from en. Follow RFC 4647 Lookup: prefer the string-equal tag, and + fall back to the distance match only when no such tag is present (e.g. zh matches cmn). + """ + tags = {str(x) for x in available if x} + try: + want = Language.get(str(language)).to_tag().casefold() + except LanguageTagError: + # e.g. a config typo in language_priority; an unparseable tag matches nothing + return set() + if not exact: + return {tag for tag in tags if is_close_match(language, [tag])} + string_equal = {tag for tag in tags if Language.get(tag).to_tag().casefold() == want} + return string_equal or {tag for tag in tags if is_exact_match(language, [tag])} + + +def resolve_sort_langs(tokens: Sequence[str], original: Optional[Union[str, Language]] = None) -> list[str]: + """ + Prepare language tokens for a Tracks.sort_* by_language argument. + + "orig" becomes the title language, or is dropped when the title has none. "all" and + "best" pass through, because the sort methods resolve them against the tracks. The + result keeps the first position of each language and drops later repeats. + """ + resolved: list[str] = [] + for token in tokens: + language = str(original) if token == "orig" else str(token) + if token == "orig" and not original: + continue + if language not in resolved: + resolved.append(language) + return resolved + + def get_boxes(data: bytes, box_type: bytes, as_bytes: bool = False) -> Box: # type: ignore """ Scan a byte array for a wanted MP4/ISOBMFF box, then parse and yield each find.