unshackle/docs/guide/vaults.md

# Key Vaults

A **key vault** is a store of DRM **content keys** that unshackle remembers between downloads. Once unshackle has recovered a title's keys, it saves them to your configured vaults. The next time you download something that reuses those same keys, unshackle finds them in a vault instead of performing another license exchange. The download starts faster and puts less load on the service.

Vaults are entirely optional, but if you download regularly they are one of the biggest quality-of-life improvements you can configure.

!!! tip "Vaults save license calls even on the very first download"
    The benefit is not limited to repeat downloads. Within a single run, a service often issues the *same* Key ID and content key for both the video and audio tracks. The same pair frequently covers several resolutions or bitrates of the same title. Once unshackle has recovered a key it writes it to your vaults immediately. The moment the next track presents a KID it has already seen, the key comes from the vault instead of another license exchange. A single fetched key can therefore satisfy many tracks in one run.

## What a vault stores

A vault stores every content key as a **KID → KEY** pair, grouped by **service**:

- **KID** (Key ID): a 16-byte identifier for a specific encrypted track, stored as 32 lowercase hex characters with no dashes.
- **KEY**: the 16-byte content key that decrypts that track, also 32 hex characters.
- **Service**: the canonical service tag (for example `EXAMPLE`, not `ExampleService`) that namespaces the key. Each service gets its own logical table.

!!! note "Why keys are matched by KID"
    Vaults match keys by KID, never by PSSH/CENC header. A KID does not change unless the underlying video file itself changes, whereas the PSSH box can vary between requests for the very same content. That makes the KID a far more reliable match key.

!!! info "Null keys are ignored everywhere"
    A content key made entirely of zeroes (`00000000000000000000000000000000`) is treated as "no key". Vault lookups skip it, and every backend raises an error if you try to add one. This prevents a placeholder key from masking a real one.

## How lookups and caching work

When unshackle needs a content key during a download it asks your vaults **in the order they appear in your config**. It returns the first non-null key it finds and stops searching. If a vault raises a permission error or is unreachable, unshackle skips it and continues the search with the next one.

When unshackle recovers new keys from a license, it pushes them to **all** configured vaults, so every vault gradually converges on the same set of keys. Two rules apply to pushing:

- unshackle skips keys that already exist in a vault (it never overwrites or deletes existing data).
- A vault marked `no_push: true` receives lookups, but unshackle never writes to it. This is useful for read-only or shared upstream vaults you do not own.

unshackle pushes keys from every source, not only a local CDM. A run against a remote server that licenses with its own CDM (`--remote`) writes the keys the server returns, and `unshackle import` writes the keys it reads from an export file.

### Wrong keys from a poisoned vault

A public vault can return the right KID with a wrong content key. The decrypters accept a wrong content key and write noise. unshackle therefore trusts a content key from a vault only after FFmpeg decodes the decrypted track with strict error detection. For a fragmented MP4, unshackle reads from the encrypted file which fragments each KID encrypts, and FFmpeg decodes up to three windows of each KID: from its first, middle and last fragment on through the next fragments of that KID, for up to 4 seconds. A window never runs into a fragment of another KID or a clear fragment. A clear lead or a lead under another KID therefore cannot hide a wrong content key, and only the KID that fails is flagged. When every KID in the map has a content key, the map is complete, and a KID that is not in it does not encrypt this file. This file does not judge that KID; the file of its own track judges it. A file with no encrypted sample entry and no encrypted fragment, for example a clear audio track, judges no KID, so it neither flags nor copies a content key. A KID that is not in an incomplete map, for example one with an all-zero KID, and every KID of any other file, gets one check of the whole file. FFmpeg decodes the first seconds and the last seconds. A failure there flags each of those KIDs. When a mapped KID fails, the KIDs that are not in the map get no verdict from that pass; the retry with the replaced key judges them. Content keys from a licence skip this check. A file this FFmpeg build cannot judge counts as a pass, never as a wrong content key: a codec it has no decoder for, or a header it cannot read. A pass is not proof. A wrong content key for HEVC video can decode without an error for some seconds, most often at a low resolution, and that content key passes the check. Without FFmpeg there is no check at all.

When the decode fails, unshackle:

1. Flags the KID:KEY in the `bad_keys` table of every local SQLite vault, with the service and the name of the vault it came from. The pair also goes out of the service table, except in a vault marked `no_push`, where only the flag is written. A flagged pair cannot go back into a local vault from another vault, a later run, or `kv copy`. A licence that returns the pair clears the flag and stores it. An `API` vault that served the pair gets a report only when a different content key for that KID is accepted. When the CDM returns the same content key, that vault gets no report. `HTTP` and `MySQL` vaults have no report route. A pair from a local vault goes to no remote vault, even when a remote vault holds the same pair.
2. Restores the ciphertext from a hard link it made before decryption, then asks for the content key again. Every vault lookup skips a flagged pair, so the request reaches the remaining vaults and then the CDM. A remote answer that arrived after the winning one is reused instead of queried again.
3. Removes the flag when the CDM returns the same content key, because then the decode check was wrong, not the content key. A content key from another vault is checked in its turn, and flagged as well if it also fails.

The download stops with an error instead of writing noise when no ciphertext is left to retry with, or when every vault and the CDM are used up. Without a local SQLite vault nothing remembers a flag, so the same vault serves the same pair again and the download stops there.

Per-segment decryption gives up the retry. With `decrypt_segments` on, and always with MonaLisa, the plaintext replaces each segment as it arrives, so no ciphertext is left. A failed decode then flags the pair and stops the download. The next run skips the flagged pair.

unshackle copies a content key from one vault to the others only after this check passes, so a content key that fails the check does not spread. Without FFmpeg, a content key from a vault is not copied at all. To see or clear the flags, open the SQLite file and read or delete rows in `bad_keys`.

!!! tip "A good two-vault setup"
    A common arrangement is a fast local **SQLite** vault plus a shared remote vault (**MySQL**, **HTTP**, or **API**). The local vault answers instantly and works offline; the remote vault lets you share keys across machines or with a group.

## Configuring vaults

Declare vaults under the `key_vaults` list in your unshackle config file. Each entry is a mapping with two required fields, `type` (which backend to use) and `name` (a label you choose), plus whatever settings that backend needs.

```yaml title="unshackle.yaml"
key_vaults:
  - type: SQLite
    name: local
    path: ~/.local/share/unshackle/vaults/local.db
  - type: MySQL
    name: shared
    host: vault.example.com
    database: unshackle
    username: reader
    password: "s3cret"
```

The `type` value is matched to a vault backend by name. The four backends that ship with unshackle are `SQLite`, `MySQL`, `HTTP`, and `API`, described below.

!!! warning "SQLite vaults are loaded critically"
    During a download, a **SQLite** vault that fails to load will **abort the run**. The assumption is that a local database you configured should always be available. `MySQL`, `HTTP`, and `API` vaults are loaded leniently: if one fails to load it is logged and skipped, and the download continues with the remaining vaults.

### `vault_timeout`

Network-backed vaults (`HTTP` and `API`) accept an optional per-vault `timeout` in seconds. If you omit it, unshackle injects the global default:

```yaml title="unshackle.yaml"
vault_timeout: 10.0   # default, in seconds
```

A `timeout` set inline on an individual vault always wins over this global value. The `SQLite` and `MySQL` backends do not take a `timeout` field. `MySQL` uses `vault_timeout` as its `connect_timeout` instead, and falls back to the pymysql default of 10 seconds when `vault_timeout` is `0`. To use a different connect timeout, set `connect_timeout` on the vault.

unshackle pushes content keys to all vaults at the same time, so one slow or unreachable vault costs only its own timeout and does not delay the others.

---

## SQLite backend

A locally-accessed SQLite database file. This is the simplest and fastest option and needs no server.

```yaml title="unshackle.yaml"
key_vaults:
  - type: SQLite
    name: local
    path: ~/.local/share/unshackle/vaults/local.db
```

| Field | Required | Description |
|-------|----------|-------------|
| `type` | yes | Must be `SQLite`. |
| `name` | yes | A label for this vault. |
| `path` | yes | Path to the database file. `~` is expanded. The file (and its tables) are created automatically on first use. |
| `no_push` | no | If `true`, keys are read but never written. Defaults to `false`. |

Each service gets its own table (named after the service tag), created on demand with `kid` and `key_` columns. Connections use WAL journaling with a 30-second busy timeout, so multiple threads can safely use the same database file.

One more table, `bad_keys`, is created on demand and holds every KID:KEY that failed the decode check described in [Wrong keys from a poisoned vault](#wrong-keys-from-a-poisoned-vault), with the service, the vault it came from, and the time it was flagged. It is not a service table, and every command that walks the service tables skips it.

!!! warning "Moving the .db file silently forks your vault"
    Because the file is created on demand, if you relocate an existing `.db` without updating `path` to match, unshackle will create a fresh empty database at the old location. That leaves you with two divergent vaults and no error to warn you. Always update the config path whenever you move the file.

!!! tip "Keep a local SQLite vault even when you share a MySQL one"
    The SQLite vault is best thought of as an offline-only backup should anything ever happen to a shared `MySQL` vault. Each member of a group should keep their own local SQLite vault alongside the shared database, so no one loses their keys if the shared vault goes down or away.

---

## MySQL backend

A remotely-accessed MySQL/MariaDB database, ideal for sharing keys across several machines. It connects via `pymysql`. Prefer MariaDB over MySQL where you have the choice.

!!! warning "The vault connects directly to the database server"
    A `MySQL` vault (like `SQLite`) speaks the database wire protocol directly to the host or IP you configure. It cannot sit behind a PHP API or any other application layer. A common real-world snag: many hosting providers firewall their MySQL server to their own internal network. A database on shared hosting is therefore frequently unreachable from the outside world even when your credentials are perfectly correct. If connections time out, confirm the server accepts remote connections from your address before suspecting the config.

```yaml title="unshackle.yaml"
key_vaults:
  - type: MySQL
    name: shared
    host: vault.example.com
    database: unshackle
    username: unshackle
    password: "your-password"
    port: 3306
```

| Field | Required | Description |
|-------|----------|-------------|
| `type` | yes | Must be `MySQL`. |
| `name` | yes | A label for this vault. |
| `host` | yes | Database server hostname or IP. |
| `database` | yes | Database (schema) name. |
| `username` | yes | Login user. |
| `no_push` | no | If `true`, keys are read but never written. |

Any additional fields you provide (such as `password`, `port`, or `ssl`) are passed straight through to `pymysql.connect`, so you can use the full range of its connection options.

!!! warning "Account security is on you, not on unshackle"
    unshackle enforces none of this. It is operational policy for a database that may be shared with a group. Never connect with the `root` account, not even for yourself. Give every account a password, and never let two users share a username/password. Restrict each account's access to only the `unshackle` database, and grant normal users only `SELECT` and `INSERT`. Reserve the `CREATE` grant for a single trusted user who bootstraps the service tables (see [`kv prepare`](#kv-prepare-pre-create-service-tables)); everyone else can insert new keys but should not be creating tables.

!!! note "Grants determine what the vault can do"
    On connection the backend reads its own `SHOW GRANTS`. It requires at least `SELECT` to load at all (a missing `SELECT` grant makes the vault fail to load). Writing keys additionally needs `INSERT`, and creating a new service table needs `CREATE`. A read-only account still works fine as a lookup source. It cannot push new keys. Because `MySQL` is loaded leniently during downloads, a permission problem will not abort the run.

---

## HTTP backend

A flexible HTTP vault that speaks one of three wire protocols, selected with `api_mode`. Use this to integrate with a community or self-hosted vault service.

| `api_mode` | Transport | Authentication | Notes |
|------------|-----------|----------------|-------|
| `query` (default) | `GET` with query parameters | `username` + `password` | Full read and write. |
| `json` | `POST` with a JSON payload | `password` used as a token | Read and write per content key; cannot bulk-enumerate keys. |
| `decrypt_labs` | `POST` in DecryptLabs format | `api_key` sent as a header | **Read-only**. `no_push` is forced on. |

=== "query mode"

    ```yaml title="unshackle.yaml"
    key_vaults:
      - type: HTTP
        name: community
        host: https://vault.example.com/api
        api_mode: query
        username: myuser
        password: "your-password"
    ```

    Query mode requires `username`. Both the username and password are sent as query parameters on each request.

=== "json mode"

    ```yaml title="unshackle.yaml"
    key_vaults:
      - type: HTTP
        name: jsonvault
        host: https://vault.example.com/rpc
        api_mode: json
        password: "your-api-token"
    ```

    Requests are `POST`ed as `{"method": ..., "params": ..., "token": ...}` using methods like `GetKey`, `InsertKey`, and `GetServices`. Bulk writes (`kv copy`, `kv add`) first try `InsertKeys` with `params: {"keys": {kid: key, ...}, "service": ..., "title": ...}` (up to 500 content keys per call) and expect `{"inserted": <count>}` back. A server that answers HTTP 404 or `Unknown JSON-RPC method` to `InsertKeys` gets one `InsertKey` call per content key for the rest of the run instead. This mode cannot list all keys in bulk, so a `kv copy` from a json vault relies on the remote server's own duplicate handling.

=== "decrypt_labs mode"

    ```yaml title="unshackle.yaml"
    key_vaults:
      - type: HTTP
        name: decrypt_labs
        host: https://api.decryptlabs.example/
        api_mode: decrypt_labs
        api_key: "your-decrypt-labs-key"
    ```

    A read-only lookup source. The API key is sent in a `decrypt-labs-api-key` header and the vault refuses all writes.

| Field | Required | Description |
|-------|----------|-------------|
| `type` | yes | Must be `HTTP`. |
| `name` | yes | A label for this vault. |
| `host` | yes | Base URL of the vault API. |
| `api_mode` | no | `query` (default), `json`, or `decrypt_labs`. |
| `username` | for `query` | Required in query mode; ignored otherwise. |
| `password` | one of these | Password (query mode) or token (json mode). |
| `api_key` | one of these | Alternative to `password`; used for `decrypt_labs`. If you give both, `api_key` wins. |
| `no_push` | no | Read-only when `true` (always `true` for `decrypt_labs`). |
| `timeout` | no | Request timeout in seconds; defaults to `vault_timeout`. |

You must supply either `password` or `api_key`, or the vault will fail to load.

---

## API backend

A simple RESTful HTTP vault with a fixed, well-defined protocol. Authentication is a bearer token.

```yaml title="unshackle.yaml"
key_vaults:
  - type: API
    name: myapi
    uri: https://vault.example.com/api
    token: "your-bearer-token"
    headers:
      CF-Access-Client-Id: "CLIENT_ID_HERE"
      CF-Access-Client-Secret: "CLIENT_SECRET_HERE"
```

| Field | Required | Description |
|-------|----------|-------------|
| `type` | yes | Must be `API`. |
| `name` | yes | A label for this vault. |
| `uri` | yes | Base URI of the vault API (a trailing slash is trimmed). |
| `token` | yes | Bearer token, sent as `Authorization: Bearer <token>`. |
| `no_push` | no | If `true`, keys are read but never written. |
| `timeout` | no | Request timeout in seconds; defaults to `vault_timeout`. |
| `headers` | no | Extra HTTP `headers` sent with every request, for example the Cloudflare Access service-token headers `CF-Access-Client-Id` and `CF-Access-Client-Secret`. |

The backend talks to endpoints under `uri` (for example `GET {uri}/{service}/{kid}` to look up a single key). It interprets a numeric `code` field in each JSON response to detect errors such as an invalid token, rate limiting, or an invalid service tag.

!!! warning "This is unshackle's own protocol, not a generic vault client"
    The `API` backend expects unshackle's specific HTTP request and response shape. An API or HTTP key-vault endpoint built for a different project or service will generally *not* be compatible. You cannot point this at an arbitrary third-party vault API and expect it to work. To bridge another service, put an adapter in front of it that speaks this format, or use the `HTTP` backend if one of its wire protocols matches. When pushing many keys at once it batches requests and automatically shrinks the batch size if the server rejects an over-large request.

!!! tip "DecryptLabs shortcut"
    An `API` vault whose `name` contains `decrypt_labs` will have its `token` filled in automatically from a global `decrypt_labs_api_key` in your config if you leave `token` unset. This lets you keep the API key in one place.

---

## Managing vaults with `unshackle kv`

The `kv` command group manages the contents of your configured vaults. Every subcommand refers to vaults **by the `name` you gave them** in `key_vaults`; an unknown name is an error.

```console
$ unshackle kv --help
```

### `kv add`: import keys from a file

Add `KID:KEY` pairs to one or more vaults for a given service. The file must contain one pair per line as `HEX:HEX` (32 hex characters each); lines that do not match are ignored.

```console
$ unshackle kv add keys.txt EXAMPLE local shared
```

```text title="keys.txt"
9a04f07998404286ab92e65be0885f95:b1c2d3e4f5a6978012345678deadbeef
0e7cc1a4e2f9411db0f24a3fbb8f8b21:00112233445566778899aabbccddeeff
```

The service argument (`EXAMPLE` above) is normalized to its canonical tag automatically. Keys that already exist in a vault are reported as skipped.

### `kv search`: find a content key by KID

Search your vaults for a KID and print any matching content key.

```console
$ unshackle kv search 9a04f07998404286ab92e65be0885f95
```

The KID must be 32 hex characters (dashes are stripped for you). By default the search scans every configured vault and every service table; narrow it with options:

| Option | Description |
|--------|-------------|
| `-s`, `--service` | Limit the search to a single service tag. |
| `-v`, `--vault` | Limit the search to one configured vault by name. |

!!! note
    Remote vaults that cannot enumerate their service tables (for example an `HTTP` vault in `json` or `decrypt_labs` mode) can only be searched when you also pass `--service`. Without it, they are skipped.

### `kv copy`: copy keys between vaults

Copy every content key from one or more source vaults into a single destination vault. `kv copy` never alters existing rows; it adds only genuinely new KIDs and skips null keys.

```console
$ unshackle kv copy shared local
```

The first argument is the destination; the rest are sources. You can supply several sources at once:

```console
$ unshackle kv copy local shared community
```

| Option | Description |
|--------|-------------|
| `-s`, `--service` | Only copy the given service. |
| `-l`, `--local-only` | Only copy services you have installed locally, skipping tables for unknown services. |

`--service` and `--local-only` are mutually exclusive.

To import an SQLite vault file that is not in the config, give its path as the source:

```console
$ unshackle kv copy local ~/Downloads/key_store.db -s EXAMPLE
```

`kv copy` only reads from a source, so the file stays unchanged. `kv sync` also accepts a path, but it writes content keys into that file.

With `--service`, `kv copy` reads the table named by the service's vault tag and the table named by its plain tag. It writes both into the vault-tag table, which is the table a download reads. Without `--service`, each table keeps its name.

A pair the destination vault has flagged in `bad_keys` is left behind, so a copy does not undo a flag. `bad_keys` is never treated as a service, so its rows are never copied into a service table.

### `kv sync`: mirror vaults both ways

Make each of two or more vaults hold every content key the others have. This is effectively a two-way `copy` between each pair, so afterwards all listed vaults hold the same set of keys.

```console
$ unshackle kv sync local shared
```

It accepts the same `--service` and `--local-only` options as `copy`, and requires at least two vaults. It runs `copy` between each pair, so it skips flagged pairs and the `bad_keys` table in the same way.

### `kv prepare`: pre-create service tables

Create the per-service tables on table-based vaults (`SQLite`, `MySQL`) ahead of time, for every service you have installed. Vaults that do not use tables (the network-backed `HTTP` and `API` backends) are skipped.

```console
$ unshackle kv prepare shared
```

This is mainly useful with a `MySQL` vault where you want the tables created once by an account that has the `CREATE` grant.

---

## See also

- [Downloading](downloading.md): how keys are recovered and used during a download.