fix(dl): read the vaults first under --all-drm

A DRM system sends a challenge only for a KID with no content key. --cdm-only keeps one challenge per system for each set of KIDs.

docs/dev/rest-api/endpoints.md

@@ -444,7 +444,7 @@ Make a download job. It requires `service` and `title_id`. Every other field is
 | `slow` | boolean/string | `null` | Randomized delay between downloads. |
 | `split_audio` | boolean | `null` | Separate files per audio codec. |
 | `skip_dl` | boolean | `false` | Only fetch keys, do not download. |
-| `all_drm` | boolean | `false` | License each track with both Widevine and PlayReady. Needs a CDM for each DRM system. The server refuses it with `cdm_only: false` and with `cdm`, which selects one device. |
+| `all_drm` | boolean | `false` | License each track with both Widevine and PlayReady. Needs a CDM for each DRM system. Key vaults are read first; `cdm_only: true` sends a challenge for each system. The server refuses it with `cdm`, which selects one device. |
 | `export` | boolean | `false` | Export manifest/keys/subs to JSON. |
 | `cdm_only` | boolean | `null` | Force CDM-only (`true`) or vault-only (`false`) key retrieval. |
 | `proxy` / `no_proxy` / `no_proxy_download` / `proxy_download` | string / bool / bool / string | `null` / `false` / `false` / `null` | Proxy controls. `proxy` and `proxy_download` must be full proxy URIs unless the API key has `server_proxy`. A Control D resolver, `controld://<resolver>@dns.controld.com`, counts as a full proxy URI: the server runs a forwarder for it. |

docs/guide/cli-reference.md

@@ -194,7 +194,7 @@ order the service used, then renumbers the episodes into the order you asked for
 | `--cdm-only` / `--vaults-only` | Use only the CDM, or only Key Vaults, for key acquisition. |
 | `--cdm <name>` | Use the named CDM device from the `cdm` config mapping for this run, ignoring the service/default mapping. Over `--remote`, that device licenses the remote session, not the server CDM. |
 | `--skip-dl` | Skip the download but still retrieve keys. |
-| `--all-drm` | License each track with both Widevine and PlayReady. Needs a CDM for each DRM system. Not for use with `--vaults-only` or `--cdm`. |
+| `--all-drm` | License each track with both Widevine and PlayReady. Needs a CDM for each DRM system. Key Vaults are read first; `--cdm-only` sends a challenge for each system. Not for use with `--cdm`. |
 | `--export` | Export track info and keys to a `mediaexport` JSON file in the exports directory. |
 
 ### Network & proxy

docs/guide/downloading.md

@@ -796,8 +796,9 @@ older unshackle `version: 2` files and unidl's own export files.
 unshackle dl --skip-dl --export EXAMPLE 81234567
 ```
 
-Add `--all-drm` to send a challenge for each DRM system the title offers. The export then
+Add `--all-drm` to get the content keys of each DRM system the title offers. The export then
-holds the DRM init data of both.
+holds the DRM init data of both. A Key Vault that has a content key stops the challenge for
+its KID; add `--cdm-only` to send a challenge for each DRM system.
 
 ```shell title="License with Widevine and PlayReady, no download"
 unshackle dl --skip-dl --all-drm --export EXAMPLE 81234567

docs/guide/drm-and-cdm.md

@@ -484,10 +484,10 @@ automatically.
 
 ### License both systems with `--all-drm`
 
-By default a track licenses with one DRM system. `--all-drm` sends a challenge for
+By default a track licenses with one DRM system. `--all-drm` gets the content keys for
 Widevine and for PlayReady on each track that offers both. Use it with `--skip-dl` and
 `--export` to keep the content keys and the DRM init data of both systems. The two
-systems can return different sets of KIDs, and unshackle keeps the content keys of both.
+systems can list different sets of KIDs, and unshackle keeps the content keys of both.
 
 ```shell
 unshackle dl --skip-dl --all-drm --export EXAMPLE 81234567
@@ -495,8 +495,12 @@ unshackle dl --skip-dl --all-drm --export EXAMPLE 81234567
 
 - The `cdm` mapping must name a device for each system. A single device name, or `--cdm`,
   stops the run with an error that names the missing system.
-- Each DRM system sends one challenge for a set of KIDs in a run. A content key in a
+- A Key Vault is read first, as in a run without the flag. A DRM system sends a challenge
-  Key Vault does not stop the challenge, so `--vaults-only` is refused.
+  only for a KID that no Key Vault and no earlier track of the run gave a content key
+  for. When the first system returns all the KIDs of the second, the second sends no
+  challenge.
+- With `--cdm-only`, unshackle does not read the Key Vaults, and each DRM system sends one
+  challenge for a set of KIDs in a run.
 - A service with a Widevine licence function only gets the PlayReady challenge through that
   function. If its licence server refuses the challenge, that DRM system does not license.
 - If one DRM system does not license, the title fails. With `--best-available` unshackle
@@ -504,8 +508,9 @@ unshackle dl --skip-dl --all-drm --export EXAMPLE 81234567
   unshackle does not send the challenge again for the same KIDs in that run.
 - A track gets its DRM systems from the manifest. For a DASH track or a direct URL,
   unshackle also reads the init segment for a system that the manifest does not carry.
-- The service gets a challenge from each DRM system for each set of KIDs. A licence server
+- With `--cdm-only`, the service gets a challenge from each DRM system for each set of
-  that limits the request rate can answer `429`, and unshackle then tries again.
+  KIDs. A licence server that limits the request rate can answer `429`, and unshackle then
+  tries again.
 
 !!! note "Where `--all-drm` licenses one system only"
     - A service that gets a different manifest for each DRM system. unshackle shows a

tests/core/test_all_drm.py

@@ -80,12 +80,12 @@ def cmd() -> dl:
     return cmd
 
 
-def test_each_drm_system_licenses_although_a_content_key_is_held(cmd: dl) -> None:
+def test_cdm_only_licenses_each_drm_system_although_a_content_key_is_held(cmd: dl) -> None:
     wv, pr = widevine(), playready()
     wv.content_keys[KID] = VAULT_KEY
     prepare = Prepare()
 
-    cmd.licence_all_drm(prepare, SimpleNamespace(drm=[wv, pr]), False, wv, track_kid=KID)
+    cmd.licence_all_drm(prepare, SimpleNamespace(drm=[wv, pr]), False, wv, track_kid=KID, force=True)
 
     assert prepare.forced() == ["Widevine", "PlayReady"]
     assert all(held == {} for _, _, held in prepare.calls)
@@ -96,7 +96,7 @@ def test_a_second_track_with_the_same_kids_sends_no_request(cmd: dl) -> None:
     prepare = Prepare(fail=PlayReady)
     for _ in range(2):
         wv, pr = widevine(), playready()
-        cmd.licence_all_drm(prepare, SimpleNamespace(drm=[wv, pr]), True, wv, track_kid=KID)
+        cmd.licence_all_drm(prepare, SimpleNamespace(drm=[wv, pr]), True, wv, track_kid=KID, force=True)
 
     assert prepare.forced() == ["Widevine", "PlayReady"]
     assert len(prepare.calls) == 4
@@ -107,7 +107,7 @@ def test_a_drm_system_that_fails_stops_the_title(cmd: dl) -> None:
     wv.content_keys[KID] = VAULT_KEY
 
     with pytest.raises(ValueError, match="licence refused"):
-        cmd.licence_all_drm(Prepare(fail=Widevine), SimpleNamespace(drm=[wv, pr]), False, wv, track_kid=KID)
+        cmd.licence_all_drm(Prepare(fail=Widevine), SimpleNamespace(drm=[wv, pr]), False, wv, track_kid=KID, force=True)
 
     assert wv.content_keys == {KID: VAULT_KEY}
 
@@ -167,7 +167,7 @@ def test_a_playready_track_with_two_widevine_objects_licenses(cmd: dl) -> None:
 
     cmd.licence_all_drm(prepare, SimpleNamespace(drm=[pr, widevine(), widevine()]), False, pr, track_kid=KID)
 
-    assert prepare.forced() == ["PlayReady", "Widevine"]
+    assert [name for name, _, _ in prepare.calls] == ["PlayReady", "Widevine"]
 
 
 class RealPrepare:
@@ -206,7 +206,7 @@ class RealPrepare:
         drm.get_content_keys = get_content_keys
         return drm
 
-    def run(self, wv: Any, pr: Any) -> None:
+    def run(self, wv: Any, pr: Any, force: bool = False) -> None:
         table = Table.grid()
         table.add_row("")
         track = SimpleNamespace(drm=[wv, pr], id="track", height=None)
@@ -220,21 +220,21 @@ class RealPrepare:
             cdm_only=False,
             vaults_only=False,
         )
-        self.cmd.licence_all_drm(prepare, track, False, wv, track_kid=KID)
+        self.cmd.licence_all_drm(prepare, track, False, wv, track_kid=KID, force=force)
 
 
-def test_prepare_drm_licenses_each_drm_system_and_does_not_read_the_vault() -> None:
+def test_cdm_only_licenses_each_drm_system_and_does_not_read_the_vault() -> None:
     real = RealPrepare(vault={KID: VAULT_KEY})
     wv, pr = real.drm(widevine(), {KID: KEY}), real.drm(playready(), {KID: KEY})
 
-    real.run(wv, pr)
+    real.run(wv, pr, force=True)
 
     assert real.requests == ["Widevine", "PlayReady"]
     assert real.vault_queries == []
     assert wv.content_keys == {KID: KEY} and pr.content_keys == {KID: KEY}
     assert is_widevine_cdm(real.cmd.cdm)
 
-    real.run(real.drm(widevine(), {KID: KEY}), real.drm(playready(), {KID: KEY}))
+    real.run(real.drm(widevine(), {KID: KEY}), real.drm(playready(), {KID: KEY}), force=True)
 
     assert real.requests == ["Widevine", "PlayReady"]
 
@@ -249,10 +249,28 @@ def test_the_other_drm_system_can_return_a_different_set_of_kids() -> None:
     assert wv.content_keys == {KID: KEY, OTHER_KID: OTHER_KEY}
 
 
-def test_the_api_refuses_all_drm_with_vaults_only_or_one_device() -> None:
+def test_the_api_refuses_all_drm_with_one_device() -> None:
-    assert "cdm_only" in (validate_download_parameters({"all_drm": True, "cdm_only": False}) or "")
     assert "cdm" in (validate_download_parameters({"all_drm": True, "cdm": "device"}) or "")
     assert validate_download_parameters({"all_drm": True, "cdm_only": True}) is None
+    assert validate_download_parameters({"all_drm": True, "cdm_only": False}) is None
+
+
+def test_a_content_key_in_a_vault_stops_the_challenge_of_each_drm_system() -> None:
+    real = RealPrepare(vault={KID: VAULT_KEY})
+    wv, pr = real.drm(widevine(), {KID: KEY}), real.drm(playready(), {KID: KEY})
+
+    real.run(wv, pr)
+
+    assert real.requests == []
+    assert wv.content_keys == {KID: VAULT_KEY} and pr.content_keys == {KID: VAULT_KEY}
+
+
+def test_the_other_drm_system_licenses_only_for_a_kid_with_no_content_key() -> None:
+    real = RealPrepare()
+
+    real.run(real.drm(widevine(), {KID: KEY}), real.drm(playready(), {KID: KEY}))
+
+    assert real.requests == ["Widevine"]
 
 
 def test_the_init_segment_adds_the_drm_system_the_track_does_not_have(monkeypatch: pytest.MonkeyPatch) -> None:

unshackle/commands/dl.py

@@ -1717,8 +1717,6 @@ class dl:
             vaults_only = None
         else:
             vaults_only = not cdm_only
-        if all_drm and vaults_only:
-            raise click.ClickException("--all-drm sends licence requests, so it cannot be used with --vaults-only.")
 
         config.decryption = resolve_decryption(
             config.decryption_map, config.decryption, service.__class__.__name__.upper()
@@ -2898,7 +2896,7 @@ class dl:
                     service_session=service.session,
                 )
                 if title_all_drm:
-                    return partial(self.licence_all_drm, prepare, track, best_available)
+                    return partial(self.licence_all_drm, prepare, track, best_available, force=bool(cdm_only))
                 return prepare
 
             server_cdm_type = None
@@ -4441,7 +4439,10 @@ class dl:
         return cdm
 
     def licence_one_drm(self, prepare: Callable, drm: DRM_T, track_kid: Optional[UUID]) -> None:
-        """Send one challenge for the KIDs of this DRM, the first time a run sees them for its DRM system.
+        """Send one challenge for the KIDs of this DRM, also when the DRM or a vault has the content keys.
+
+        This is the ``--cdm-only`` path. The challenge goes out the first time a run sees the KIDs
+        for the DRM system.
 
         prepare_drm asks the CDM only for a KID with no content key, so the content keys the DRM
         holds are removed for the call and put back after it.
@@ -4469,12 +4470,14 @@ class dl:
         tolerate: bool,
         drm: DRM_T,
         track_kid: Optional[UUID] = None,
+        force: bool = False,
     ) -> None:
         """License the track with each DRM system it offers, not only the one the downloader selected.
 
-        Each DRM system sends one challenge for a set of KIDs in a run, also when a vault
+        A DRM system sends a challenge only for a KID that no vault and no earlier track gave a
-        has the content keys. A DRM system that fails stops the title. With ``tolerate`` it is a
+        content key for. With ``force``, each DRM system sends one challenge for a set of KIDs in
-        warning, if the other DRM system gave the content key of the track.
+        a run. A DRM system that fails stops the title. With ``tolerate`` it is a warning, if the
+        other DRM system gave the content key of the track.
         """
         if type(drm) not in (Widevine, PlayReady):
             prepare(drm, track_kid=track_kid)
@@ -4499,7 +4502,10 @@ class dl:
                 for target in targets:
                     kid = track_kid if target is drm or track_kid in target.kids else None
                     try:
-                        self.licence_one_drm(prepare, target, kid)
+                        if force:
+                            self.licence_one_drm(prepare, target, kid)
+                        else:
+                            prepare(target, track_kid=kid)
                     except Exception as e:
                         if not tolerate:
                             raise

unshackle/core/api/handlers.py

@@ -1835,8 +1835,6 @@ def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]:
             return "output_dir must be a path under the server's downloads directory."
         data["output_dir"] = str(target)
 
-    if data.get("all_drm") and data.get("cdm_only") is False:
-        return "all_drm sends licence requests, so it cannot be used with cdm_only set to false."
     if data.get("all_drm") and data.get("cdm"):
         return "all_drm needs a CDM for each DRM system, so it cannot be used with cdm, which selects one device."
 

unshackle/core/api/routes.py

@@ -949,7 +949,7 @@ async def download(request: web.Request) -> web.Response:
                 description: Skip downloading, only retrieve decryption keys (default - false)
               all_drm:
                 type: boolean
-                description: License each track with both Widevine and PlayReady. Needs a CDM for each DRM system and cannot be used with `cdm` or with `cdm_only` set to false (default - false)
+                description: License each track with both Widevine and PlayReady. A DRM system sends a challenge only for a KID that no vault has, unless `cdm_only` is true. Needs a CDM for each DRM system and cannot be used with `cdm` (default - false)
               export:
                 type: boolean
                 description: Export manifest, DRM init data, keys, and track info to a JSON file in the exports directory (default - false)