---
title: Upgrades
---

An upgrade replaces the image (and the document extractor's, which holds no state) and
applies the migrations the new build carries. No other state moves: everything durable
is in ClickHouse (and in the document bucket, for originals written with
`LOOM_DOCUMENT_OBJECT_STORE=s3`), and the code mirrors are mounted unchanged. This page
is the operator side. How a client sees unknown and removed fields is in the [HTTP API
reference](/docs/es/reference/http-api#unknown-and-removed-fields). Which releases get fixes, and which
SDK works with which server: [supported versions](/docs/es/operations/supported-versions).

## How migrations run

The journal, numbering, runner directives and the rules for writing a migration are in
[migrations](/docs/es/development/migrations). What an operator needs:

- **At boot.** The server applies pending files before it reports ready or starts
  workers (`LOOM_BOOT_MIGRATE=true`, the default). Every process, in any container, takes
  the `KeeperMap` lock in `loom._migration_lock`, so each file applies once. A server
  without a Keeper is refused.
- **Provider rebuilds.** A boot with files pending first enrolls as a provider
  participant, then checks under the provider configuration lock (one Keeper key, held for
  that check only) that no provider rebuild is unfinished, and refuses to migrate if one
  is: `Finish the provider rebuild before applying schema migrations`. The files then
  apply under the migration lock alone. A reconfiguration started meanwhile waits until
  the booting process acknowledges it, so a rebuild never runs beside a migration.
- **A container killed while it migrates or reconfigures** blocks the others only for a
  while. The migration lock is taken over once its holder stops renewing it (`ttl_s`).
  The provider lock (TTL 60 s, renewed every 20 s) is taken over once it expired and its
  holder's participant row expired too: no heartbeat for 60 s (one every 10 s) and no
  query tagged with its id in `system.processes` on any node. Containers booting together
  wait up to 120 s for the provider lock instead of failing. A lock whose holder has
  no participant row is freed with `loom provider-lock clear --holder <holder>` once
  that process is confirmed stopped; the command refuses while the holder renews,
  heartbeats or has a query running.
- **Mutations.** A file is recorded only once its mutations are done. A failed mutation,
  or one running past `LOOM_MIGRATE_MUTATION_WAIT_S` (default 600), fails the file and
  stops the boot with the file named in the log.
- **Key changes.** A migration that changes a sort or partition key rebuilds an empty
  table itself and fails the boot on one with rows. Run `scripts/ops/table_rebuild.py`
  for that table before the deploy; the strict dry run catches it.
- **Fresh install.** A fresh install and an upgrade are one path: the server replays
  whatever the ledger lacks. On an empty database that is the whole journal and takes a
  few minutes.
- **Stranded ledger.** A database whose ledger stopped partway through the files a
  baseline replaces is refused (`stranded`): migrate it with a build from before that
  baseline first.
- **Renumbered files.** A ledger row whose file the repository no longer has is
  `ledger-only`, and `--strict` exits 1 on it, except when its recorded SHA-256 is a
  current file's: that is the same file under a new number. The runner reports it as
  renamed, records the new name without running it again if it is pending, and
  `--strict` passes. A file edited after it ran under its old name has another SHA-256,
  so it stays ledger-only: its new name runs as a pending file, and the old row is
  removed by hand ([troubleshooting](/docs/es/operations/troubleshooting#migrations-pending-or-checksum-drift)).
  `/v1/health` lists the old row under `schema.ledger_only` until then.

The CLI runs the same runner by hand:

```bash
loom migrate --status              # applied and pending; changes nothing
loom migrate --dry-run --strict    # pending files; exit 1 on ledger-only rows (not renames) or checksum drift
loom migrate                       # apply pending migrations
loom migrate --target N            # apply through file number N only
```

`make migrate`, `make migrate-status` and `make migrate-dry` run the same module from a
checkout.

## Memory writer fence cutover

For the ClickHouse authority, moving from a build without memory insertion fences to
one that takes them requires a write pause. Schema compatibility during a rolling
upgrade does not make older writers participate in those fences
([storage authority](/docs/es/architecture/storage-authority)).

1. Pause memory write ingress, scheduled jobs and maintenance commands. Inventory
   every process that can insert memory versions: serving replicas, their background
   workers, standalone workers, CLI jobs and scripts. Prevent older images or jobs
   from restarting. Keep replacement writers stopped while earlier sends settle.
2. Stop those older writers gracefully, allowing active requests and background work
   to drain, and confirm that every older process has exited. Removing a replica from
   the load balancer alone does not stop its background workers. Record when the last
   older writer stops and the carried query-id prefixes of the retired processes.
3. From that last stop, wait at least `15 + W + 5` seconds: the guarded insert's transit
   allowance, its largest server execution bound `W`, and the settlement margin.
   Use the largest effective `max_execution_time` among retired writers, including
   per-statement overrides; the write class uses `LOOM_CH_TIMEOUT_WRITE_S`, or 60
   seconds when that setting is zero. These bounds are defined by
   `server/core/rmt_carry.py` (`send_bound_s`) and `server/core/ch.py`
   (`write_limit_s`). A successful client response or an early empty process list
   does not replace this wait: a timed-out send may still start within its allowance.
4. After the bound, inspect `system.processes` on every ClickHouse replica for the
   retired processes' `loom-carry/<process-id>/` query-id prefixes. None may remain.
   If a replica cannot be checked, a writer's execution bound is unknown, or a writer
   bypasses guarded inserts, keep writes paused until its queued and running work is
   accounted for; elapsed time alone does not establish settlement for that writer.
5. Deploy only fence-aware serving and background writers, check their image versions
   and readiness as in the upgrade procedure, and then resume jobs and write ingress.
   The source-head guarantee starts only after all earlier sends have settled and
   every active writer takes the fences. Configure optional
   `LOOM_CH_WRITE_FENCE_CONCURRENCY` and `LOOM_CH_RESTORE_BATCH_ROWS` only on that fleet.
   A rollback to writers without fences loses this guarantee; remove settings those
   builds do not accept before starting them.

## Upgrade procedure

1. Read the release's [GitHub Release](https://github.com/ClickHouse/loom/releases) notes
   for operator steps (new settings, backfills a deployment must run) and the
   client-visible wire changes.
2. From the new build's checkout, against the production ClickHouse, run
   `loom migrate --dry-run --strict`. It lists what the boot will apply and stops on
   ledger problems ([troubleshooting](/docs/es/operations/troubleshooting#migrations-pending-or-checksum-drift)).
   If the release declares a table rebuild, run `scripts/ops/table_rebuild.py` for that
   table now. A deployment of the Postgres authority moving to a build that keeps
   knowledge document heads in Postgres unsets `LOOM_RETENTION_DOCUMENTS_DAYS` now, by
   hand: the newer build refuses to boot with it set, and under that authority no
   retention TTL reaches `loom.documents`, and a namespace's `retention_days` does not
   reach `loom.chunks` either
   ([rolling upgrades](/docs/es/architecture/rolling-upgrades), rule 38). A
   [cell](/docs/es/architecture/cells) moving to a build that marks its sessions and
   carries the org triggers (rule 47) first points its `LOOM_PG_DSN` straight at
   Postgres, past any pooler ([Postgres](/docs/es/operations/postgres)), and a later cell prepared and
   then backed out by unsetting `LOOM_CELL` is cleared first
   ([troubleshooting](/docs/es/operations/troubleshooting#a-process-with-loom_cell-unset-is-gated)):
   the newer build gates its processes on its state. On a later cell already prepared,
   the newer build's first migrating boot creates the org triggers, so an older pod
   still running there has every insert of a tenant org's row refused (an unmapped
   error to its callers) until it is replaced: roll the cell out in one step.
3. Complete the write pause and settlement steps of the
   [memory writer fence cutover](#memory-writer-fence-cutover) before starting a
   fence-aware build over writers that do not take those fences.
   Deploy the new image. The first container to boot applies the migrations; the others
   wait for it and then find nothing pending.
4. Wait for `/v1/health` to report `status: ok` and an empty `schema.pending` before
   routing traffic to it. The [zero-downtime pair](/docs/es/operations/deployment#zero-downtime-pair)
   does steps 2 to 4 for you, except unsetting the setting step 2 names: `redeploy.sh`
   runs the strict dry run in its preflight and swaps only after the new colour is
   ready.
5. Run the release's backfills, if any ([maintenance](/docs/es/operations/maintenance#backfills-and-repairs)).
6. Update client plugins when the release mentions a plugin change: hooks run on the
   client machine and change only through the client's plugin manager
   (`claude plugin update loom@loom` for Claude Code), never through a server deploy.
7. Once the last old replica has stopped, re-verify the erasures completed during the
   rollout: `loom erasure-re-verify --completed-since <when step 3 began> --resume`. An
   old replica's saga does not know the tables the new build added
   ([rolling upgrades](/docs/es/architecture/rolling-upgrades), rule 14). The command
   deletes only orphan row links: rows of `entity_triple_rows` and `entity_edge_rows`
   in each erasure's scope whose memory has no live row and no row written since the
   erasure's census. Without `--resume` it counts them and exits 1 if any are left.
   Data written after the erasure stays.
8. Turn on a feature the previous build cannot honour only once every replica runs
   the new build: `memory_extract_agent_learnings`, since the previous build does not
   erase `loom.learnings_supporters`
   ([rolling upgrades](/docs/es/architecture/rolling-upgrades), rule 23);
   `LOOM_DOCUMENT_OBJECT_STORE=s3`, since the previous build reads a bucket row's empty
   `data` and answers its download `410 BLOB_GONE` (rule 9); and
   `LOOM_PG_PAYLOAD_CLEAR_ENABLED`, no sooner than one clearing horizon after the new
   build serves alone, since the previous build does not stamp a late delivery (rule 43).
9. A deployment with `LOOM_CODE_ASK_MEMORY_NAMESPACE` set, now or at any time before,
   that moves from a build whose cached `code.ask` answers do not name the memory they
   quoted (one before `086_code_answer_memory_scope.sql`) to one that does, a roll
   forward after a rollback included: once the newer build serves alone, with no
   provider rebuild running, clear the answer cache with
   `TRUNCATE TABLE <LOOM_CH_DATABASE>.code_answers` (`loom.code_answers` by default). The
   old build's answers and tombstones name no memory, and an erasure or redact it ran
   during the rollout did not know the table
   ([rolling upgrades](/docs/es/architecture/rolling-upgrades), rule 36), so no saga can find
   what they hold. A rebuild publishes its copy of the table over it, so one running
   across the truncate brings the rows back. A truncate drops the parts at once, on every
   replica of a replicated table, with no mutation to wait for, and the loom user holds
   the grant. Each cleared question costs one synthesis on its next ask.
10. Moving to a build whose user erasure reaches what a consolidated summary of the
    user's memories fed, a roll forward after a rollback included: once the newer build
    serves alone, request again each user erasure without `purge` requested before then
    (the `user_id` rows of `loom.erasure_requests`, kept 90 days), including one the newer
    build finished: an older build's saga left the summary's history, chunks, vectors,
    graph rows and shared copies ([rolling upgrades](/docs/es/architecture/rolling-upgrades),
    rule 37).
    The new saga also erases what the user wrote since: request it for a user with no
    live memory row, or only where that is wanted. A user purge an older build completed
    left the same rows, and no later user erasure finds them: the user's keys are gone.
    One it ran past `shared.tombstone` also left the search rows of shared copies of the
    user's own memories, whichever build finished it. A shared copy an older build
    promoted with `agent="*"` names `*` for its source agent, and no erasure of the
    user it copied may find it. List them, and review each, with `memory.demote` for one
    that holds an erased user's text:

    ```sql
    SELECT org, namespace, memory_key FROM loom.shared FINAL
    WHERE tombstoned_at IS NULL AND arrayExists(x -> splitByChar('/', x)[3] = '*', promoted_from)
    ```

11. A deployment of the Postgres authority that moves to a build keeping knowledge
    document heads in Postgres (`002_knowledge_documents.sql`), a roll forward after a
    rollback included: once the newer build serves alone, run again every erasure that
    completed while a replica of an older build served since the newer build first
    served. An older build's erasure removes no head or delivery: an erased document's
    text stays in `loom.knowledge_heads` until the erasure runs again, and a delivery it
    left is published again by the delivery worker
    ([rolling upgrades](/docs/es/architecture/rolling-upgrades), rule 38). Do it before
    retrying a parked delivery.
12. A deployment of the Postgres authority that moves to a build keeping knowledge
    ingestion jobs (`005_knowledge_jobs.sql`) or the dataset catalog
    (`007_knowledge_datasets.sql`) in Postgres, a roll forward after a rollback
    included: once the newer build serves alone, run again every erasure that completed
    while a replica of an older build served since the newer build first served. An
    older build's erasure removes no job from `loom.knowledge_jobs` and no catalog row
    from `loom.knowledge_datasets`: the newer build's worker projects such a job back
    into `loom.knowledge_ingestions` and runs it, and the catalog rows stay until the
    erasure runs again ([rolling upgrades](/docs/es/architecture/rolling-upgrades), rules
    40 and 42). Step 11's runs cover the heads, the jobs and the catalog rows alike; one
    run per erasure is enough. Ask again, too, each `knowledge.delete` and
    `knowledge.bulk_delete` an older build answered in that window: it removed the
    sources' datasets from `loom.datasets` alone.
13. Moving to a build that keeps document sources on any namespace, a roll forward
    after a rollback included: once the newer build serves alone, retire the sources an
    older build's `namespace.delete`, `collection.delete` or namespace erasure left
    active ([rolling upgrades](/docs/es/architecture/rolling-upgrades), rule 44). The
    sources on a deleted namespace:

    ```sql
    SELECT s.org, s.namespace, s.name
    FROM (SELECT org, namespace, name FROM loom.knowledge_sources FINAL WHERE active) AS s
    INNER JOIN (SELECT org, namespace FROM loom.namespaces FINAL WHERE NOT active) AS n
        ON s.org = n.org AND s.namespace = n.namespace
    ```

    and the sources registered before a namespace erasure completed that it did not
    retire (one an older build ran, or one an older build added while a newer build's
    erasure ran):

    ```sql
    SELECT s.org, s.namespace, s.name, e.id
    FROM (
        SELECT org, namespace, name, active, created_at, updated_at
        FROM loom.knowledge_sources FINAL
    ) AS s
    INNER JOIN (
        SELECT id, org, completed_at,
               arrayJoin(if(scope_type = 'namespace', [scope_value],
                            if(JSONType(scope_value) = 'Array',
                               JSONExtract(scope_value, 'Array(String)'),
                               JSONExtract(scope_value, 'namespaces', 'Array(String)')))) AS namespace
        FROM loom.erasure_requests FINAL
        WHERE scope_type IN ('namespace', 'namespace_list') AND status = 'completed'
    ) AS e ON s.org = e.org AND s.namespace = e.namespace
    WHERE s.created_at <= e.completed_at AND (s.active OR s.updated_at > e.completed_at)
    ```

    For the first query, remove each listed source (`knowledge.source.remove`), or
    delete its namespace again with the newer build, which retires it. For the second,
    remove each listed source with `knowledge.source.remove` and
    `delete_documents=true`, which retires the documents it wrote back (it accepts a
    source already retired, by a later delete of its namespace for one), and delete with
    `memory.delete` the decision rows mined from them. A namespace erasure with the
    newer build does both, but it also erases everything written to the namespace since
    the first erasure; request it again only for a namespace nobody has written to
    since. A source registered after the erasure is not listed: the erasure never
    covered it. The second list is a worklist to act on once: a source removed as above
    stays listed, since its removal row is newer than the erasure.

    A namespace erasure an older build started and the newer build finished needs
    nothing here: its final census finds the documents and decisions its source wrote
    back and reopens their steps, so the saga completes only once they are gone.

14. Moving to a build whose namespace erasure reaches `loom.shared`, a roll forward after a
    rollback included: once the newer build serves alone, clear the promoted rows an older
    build's namespace erasure left ([rolling upgrades](/docs/es/architecture/rolling-upgrades),
    rule 45). The query below lists each namespace that a `namespace` or `namespace_list`
    erasure in the ledger (kept 90 days) reached and that holds a promoted row, with the
    time the last one without a purge (`erased_at`) and with one (`purged_at`) ran its
    census. The statements below take rows promoted before that second, so a row promoted
    after the census stays, and so may one promoted earlier in the census's own second,
    which `promoted_at`'s whole seconds cannot tell apart; the newer build's step already
    took such a row in an erasure it ran.
    Only a `namespace_list` purges, and an older build's `namespace` row may hold the flag
    it was asked for.

    ```sql
    SELECT org, namespace,
           maxIf(counted_at, NOT purged) AS erased_at,
           maxIf(counted_at, purged) AS purged_at
    FROM (
        -- The census's time: a row promoted after it was never in the erasure's reach.
        SELECT org, ifNull(verified_at, completed_at) AS counted_at,
               scope_type = 'namespace_list' AND purge AS purged,
               arrayJoin(if(scope_type = 'namespace', [scope_value],
                            if(JSONType(scope_value) = 'Array',
                               JSONExtract(scope_value, 'Array(String)'),
                               JSONExtract(scope_value, 'namespaces', 'Array(String)')))) AS namespace
        FROM loom.erasure_requests FINAL
        WHERE scope_type IN ('namespace', 'namespace_list') AND status = 'completed'
    )
    -- Only a namespace that holds a promoted row has anything to clear.
    WHERE (org, namespace) IN (SELECT org, namespace FROM loom.shared)
    GROUP BY org, namespace
    ```

    For each, run what the newer build's step runs, over every version promoted before the
    erasure's census, a version a later promotion of the same key hides from `FINAL`
    included. Where `purged_at` is set, delete them:

    ```sql
    ALTER TABLE loom.shared DELETE
    WHERE org = '<org>' AND namespace = '<namespace>' AND promoted_at < '<purged_at>'
    SETTINGS mutations_sync = 2
    ```

    and where `erased_at` is set, tombstone the live ones left:

    ```sql
    ALTER TABLE loom.shared UPDATE tombstoned_at = toDateTime('<the time you run it>')
    WHERE org = '<org>' AND namespace = '<namespace>' AND tombstoned_at IS NULL
      AND promoted_at < '<erased_at>'
    SETTINGS mutations_sync = 2
    ```

    The setting waits for each statement on every replica. The time is written as a
    literal, as the step binds it, because a replicated table refuses a mutation that calls
    a non-deterministic function such as `now()`. A namespace erasure requested
    before the ledger's 90 days is not listed, and the promoted rows it left cannot be told
    apart from a live namespace's; they stay as the older build left them.

15. Moving to a build where every CLI command declaring an org (a code-index command, an
    import) checks its deployment: under either authority its ClickHouse user needs
    `SELECT` on `loom.storage_authority`, and the command refuses against a deployment that
    records another authority. Under the Postgres authority it also needs `LOOM_PG_DSN`
    and reads the cell state (with `LOOM_CELL` unset, on a session of its own). With
    `LOOM_CELL` set, a command that writes for no org but changes the cell's stores
    (`loom settings set`, `loom secrets reseal`, `loom code compact` and the rest) or
    revokes access checks its deployment the same way and reads the cell's `first` row on
    a session of its own. Grant the Postgres role it connects as:
    - with `LOOM_CELL` unset, `USAGE` on schema `loom`, `SELECT` on `loom.cell_first`,
      `loom.tenancy_mode` and `loom.authority`, `SELECT (updated_at)` on `loom.cell_self`
      and `SELECT (since)` on `loom.tenancy` (the column grants keep the fingerprints and
      the orgs out of its reach), and a `CONNECTION LIMIT` at least `LOOM_PG_POOL_MAX_SIZE`
      times the number of such commands that run at once, since each opens its own pool for
      its writes (the state read retries a full limit briefly, then refuses);
    - with `LOOM_CELL` set, the same with `SELECT` on `loom.cell_self` and `loom.tenancy`
      in full, and, to record a new org while the first cell is `recording`, `INSERT` on
      `loom.tenancy` and `UPDATE (since)` on `loom.tenancy_mode`; revoke both with
      `LOOM_CELL_ROUTER_TOKEN` when the cell attaches. Its document bucket settings
      (`LOOM_DOCUMENT_S3_BUCKET`, `LOOM_DOCUMENT_S3_PREFIX`) equal the servers'.

    A write that a role holding only the column grants makes on a cell past `recording` is
    refused as a permission error (SQLSTATE 42501) rather than `LT001`.

16. Before a first cell's attach, every CLI and CI image runs a build with that state read,
    and every CLI that writes for an org has `LOOM_CELL` set. A CLI publishes no instance
    or cell heartbeat for attach to read, so attach cannot check either, and a CLI of an
    older build is not refused on an `enforced` cell
    ([cells](/docs/es/architecture/cells#failure-modes)).

17. Moving to a build whose first-cell processes read their cell's state from the router on
    a timer: every router replica (`loom router serve`) runs this build before any cell's
    processes do. Each process of a prepared first cell asks the router's state route from
    its first cell-state read until it reads `serving`, and a router of an earlier build
    reads the directory on every such call: about one read per server process per
    `min(LOOM_INSTANCE_HEARTBEAT_SECONDS, LOOM_TENANCY_MODE_CACHE_S)`, so 1,600 reads a
    second for a thousand pods of eight workers at the defaults; a router of this build
    answers them from one read per replica per `LOOM_ROUTER_DIRECTORY_CACHE_S`
    ([cells](/docs/es/architecture/cells#the-heartbeats-keys)).

18. Before a first cell's attach, once `LOOM_CELL` is set on every process and every CLI and
    CI configuration of the cell (step 16), rotate the ClickHouse credential they write
    with: a new user or password for the cell's servers and CLIs, the old one dropped. A
    CLI publishes no instance or cell heartbeat for attach to read, so attach cannot see a
    job still configured with `LOOM_CELL` unset; the rotation cuts off any the rollout
    missed before it can write an org's rows
    past attach's last scan. Name the new user with `--ch-allow-user`
    ([attaching the first cell](/docs/es/operations/cell-attach)).

While a migration applies, the old build keeps serving: every migration is additive.
The rules that let the two builds serve side by side are in
[rolling upgrades](/docs/es/architecture/rolling-upgrades).

## Rollback

Redeploy the previous image; with the pair, `./redeploy.sh --rollback`. Before rolling
back to a build without assistant-turn learnings, turn `memory_extract_agent_learnings`
off and delete the rows of `loom.learnings_supporters` and the `assistant_turn` rows of
`loom.learnings_queue`
([extraction](/docs/es/features/memory/extraction#rules-the-assistant-states-about-itself)).
The previous build serves a promoted bot rule to every agent in the namespace, not only
its bot. To prevent that, delete the documents tagged `source-kind:assistant_turn` with
`knowledge.delete` before rolling back; otherwise accept the wider serving.
A namespace erasure the previous build runs once namespace guides exist leaves that
namespace's guide; run the erasure again once the newer build serves
([rolling upgrades](/docs/es/architecture/rolling-upgrades), rule 24). A namespace erasure
the previous build runs leaves the namespace's promoted rows live; step 14 of the upgrade
procedure lists them once the newer build serves again (rule 45).
After a rollback to a build without bot token maximums, every bot's tokens last an hour,
and a bot that build saves keeps 3600 as its maximum; reapply a shorter or longer one
once the newer build serves
([rolling upgrades](/docs/es/architecture/rolling-upgrades), rule 26).
A user erasure the previous build runs leaves what a summary of the user's memories
fed; request again each one run without `purge` once the newer build serves, as in step
10; a purge it completed is not found again (rule 37).
Do not roll back past the build that adds the document bucket while any catalog row
names an object in it: an older build answers those downloads `410 BLOB_GONE`, its
erasure deletes the rows and cannot reach their objects, and its sweep never walks the
bucket ([where originals are stored](/docs/es/features/knowledge#where-originals-are-stored)).
After a rollback to a build before `086_code_answer_memory_scope.sql` with a memory
namespace bound to `code.ask`, clear the answer cache as in step 9 twice: once the older
build serves alone, and again once the newer build serves again. The migration is already
in the ledger, so the roll forward applies nothing that would prompt the second.
Turn `LOOM_PG_PAYLOAD_CLEAR_ENABLED` off before rolling back, and on again no sooner
than one clearing horizon after the newer build serves alone again; once a clearing pass
has run, run `loom reconcile` only with a build that carries
`010_revision_delivered_at.sql`, start the deployment on that build after it, and roll
back only once the newer build owes no claimable delivery for a cleared revision, which
this returns as 0 (rule 43):

```sql
SELECT count(*) FROM loom.memory_deliveries d
JOIN loom.memory_revisions r USING (org, namespace, seq)
WHERE r.row IS NULL AND NOT d.settled AND d.lease_until <> 'infinity';
```

Until it does, let the newer build deliver them. The previous build never claims a
parked delivery, so a parked one may stay. While the previous build serves, a retry
leaves out the cleared revisions
([Postgres operations](/docs/es/operations/postgres#what-the-boot-checks)), and no `loom reconcile` runs:
it would release reconcile's holds to the previous build.
Before rolling a prepared cell (one with a `loom.cell_first` row) back to a build
without the [cell](/docs/es/architecture/cells) org triggers, stop every pod of the newer
build, then run only the trigger drop of
[A process with LOOM_CELL unset is gated](/docs/es/operations/troubleshooting#a-process-with-loom_cell-unset-is-gated),
not its state transaction (the cell stays prepared), and run it again if a newer pod
boots with migrations on before the rollback ends: the older build's sessions send no
tenancy marker, so once the newer build has created the triggers and the `loom.tenancy`
guard, each of its writes for an org the cell holds no tenancy row for is refused (on a
later cell before attach, every tenant org), as is its tenancy row for each newly placed
org, and its other writes pay the triggers' state read.
There is no migration to undo: the previous build reads and writes the newer schema.
Its version writes to `loom.memory` and the other `ReplacingMergeTree` memory tables
keep the columns it does not know
([rolling upgrades](/docs/es/architecture/rolling-upgrades)).

There are no down-migrations. To return to an earlier schema, restore a backup taken
before the upgrade ([backup and recovery](/docs/es/operations/backup-and-recovery)).

### Downgrading an installer install

Each installer bundle names the ClickHouse release it was built with, so installing an
earlier Loom version also runs an earlier ClickHouse on a volume the newer one wrote,
which it may not read. To roll back Loom alone, keep the newer ClickHouse: before running
the earlier installer (it starts the stack at once), pin the image ClickHouse runs now
in the install directory's `.env`. This prints the line to put there, replacing any
earlier `LOOM_CLICKHOUSE_IMAGE` line:

```bash
dir=${LOOM_INSTALL_DIR:-$HOME/.loom}
echo "LOOM_CLICKHOUSE_IMAGE=$(docker inspect -f '{{.Config.Image}}' \
  "$(docker compose --project-directory "$dir" ps -q clickhouse)")"
```

Then run the earlier release's installer naming its version, as an older installer over a
newer install refuses otherwise:

```bash
curl -fsSL https://github.com/ClickHouse/loom/releases/download/vX.Y.Z/install.sh -o loom-install.sh && LOOM_VERSION=X.Y.Z sh loom-install.sh && rm loom-install.sh
```

Run it from your home directory, or another
[directory only you can write to](/docs/es/getting-started/quickstart#fast-path-released-image-no-clone).

The installer keeps `.env`, so the pin holds ClickHouse through later upgrades too:
delete the `LOOM_CLICKHOUSE_IMAGE` line before installing a newer Loom version, and do not
leave `LOOM_CLICKHOUSE_IMAGE` exported in the shell, since the installer writes a value it
is given back into `.env`. To go
back on both, restore a backup taken before the upgrade.

## What stays the same across an upgrade

| Across an upgrade                           | What holds                                                                                                          | Enforced by                               |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
| The previous build against the new schema   | keeps reading and writing every table it knew, so a rollback needs no schema step                                   | `tests/unit/test_migrations_additive.py`  |
| An applied migration file                   | never edited or deleted; a correction is a new file                                                                 | the ledger checksum; `--dry-run --strict` |
| A client sending a field the server dropped | served; the field is named in `ignored_args`                                                                        | `server/transport/arg_check.py`           |
| The previous and new build serving together | each reads what the other wrote, new columns included, except a feature switched on only after the rollout (step 8) | `make test-mixed-version`                 |

An upgrade can change or remove operations and fields. Check what a deployment serves
with the capability catalogue
([HTTP API](/docs/es/reference/http-api#discovering-what-a-deployment-serves)), and list what
changed between two builds with `scripts/codegen/wire_changelog.py <previous> <new>`
([code generation](/docs/es/development/code-generation#wire-changelog)).

## Related

- [Deployment](/docs/es/operations/deployment#zero-downtime-pair): the pair that makes an upgrade a swap
- [Migrations](/docs/es/development/migrations): the rules a new migration follows
- [Rolling upgrades](/docs/es/architecture/rolling-upgrades): the rules for builds N-1 and N
- [Troubleshooting](/docs/es/operations/troubleshooting#migrations-pending-or-checksum-drift)
