← All projects

Project field notes

Migrating reading progress to Kavita

Two Python scripts that import per-issue reading progress into Kavita's SQLite database — one from a ComicRack XML library, one from a Codex server — without touching any other table.

stableUpdated

tl;dr

https://github.com/DieselTech/Kavita-Database-Migration

Three scripts:

Coming fromScriptSource file
ComicRackComicRack_Migration.pyComicDb.xml
CodexCodex_Migration.pycodex.sqlite
KavitaKavita_Migrate_Progress.pykavita.db

ComicRack and Codex scripts can be ran with --dry-run --report report.txt to get an idea what it will do. If you like what you see, then remove --dry-run and let it rip. The Kavita to Kavita script just has --dry-run to test.

Why??

People have told me they wanted to move over to Kavita but didn’t want to lose their years if not decades of reading progress stored in ComicRack. Then the same thing came up for Codex, which is a much newer server but has the same “problem”.

ComicRack and Codex scripts write to exactly one Kavita table, AppUserProgresses, for exactly one named user. Nothing else in the database is touched. The Kavita to Kavita script was made to let you start with a fresh copy of your database without bringing forward years of heavy changes.

Supported reading servers

ComicRackCodex
Storageone XML file, ComicDb.xmlDjango SQLite database, codex.sqlite
Userssingle usermulti-user, plus anonymous sessions
File pathsnot usable for matchingfull path per comic, and Kavita has them too
Match strategyfuzzy: series name + run identity + issue numberexact: relative folder path, then issue number

That difference drives everything below. ComicRack forces a scoring contest since their “metadata” is weak. Codex migrations should be a lot better because it has proper metadata and even file paths we can look at.


Part 1 — ComicRack

ComicRack stores its library and reading state in a single XML file, ComicDb.xml.

The script reads ComicDb.xml, matches each book that has reading progress against a series and chapter in Kavita, and writes rows into Kavita’s AppUserProgresses table for one named user. Since ComicRack is a “single user” application, you just pick the Kavita user to land on.

The two data models

The relevant fields on each side:

ComicRack <Book>Kavita
Series — series name, no yearSeries.Name — name with a (year) or (volume) suffix
Volume — run ordinal (8), sometimes a yearVolume.Name — the run’s start year as a string
Year / Monththe issue’s cover dateSeries.NormalizedNamename + year, lowercased, alphanumeric
Number — issue number, may be non-numericChapter.MinNumber / MaxNumber (REAL), Chapter.Range (TEXT)
CurrentPage / LastPageReadAppUserProgresses.PagesRead
Opened — .NET timestamp with offsetCreated / LastModified (+ ...Utc), naive local time

Year means different things on each side

Kavita’s NormalizedName embeds the year the run started: Avengers (2023) normalizes to avengers2023. ComicRack’s Year is the cover date of the individual issue.

So Avengers #7, published in 2024 as part of the run that began in 2023, has Year=2024 in ComicRack and lives under avengers2023 in Kavita.

Hahah you thought because it is called Issue Number that they would be int or float

Marvel loves to screw with people - 1.MU, 16.HU, 50.LR, 78.BEY. Because of this, we need to treat “issue numbers” as strings. Kavita stores these in Chapter.Range as text and sets MinNumber to 0.0. Any matcher that casts the issue number to a float drops them, and any matcher that trusts MinNumber collides them with a genuine issue #0.

Matching

Name normalization

Series names are reduced to lowercase alphanumerics with the trailing discriminator split off, so Avengers (2023) yields base avengers and discriminator 2023. A discriminator ≥ 1900 is treated as a year; below that it is a run ordinal, which is how Kavita records series like Black Panther (7).

A second lookup key is generated with a leading article stripped, so ComicRack’s Punisher: War Zone reaches Kavita’s The Punisher: War Zone (1992).

Candidate gating

A base name like avengers resolves to a dozen Kavita runs. Every one of them is a candidate, and each candidate is checked for whether it actually contains the issue number before it is scored. A run that does not hold the issue cannot win on name similarity alone.

This gate is what stops the single worst failure mode. A Kavita library will often contain a stub series — a bare Black Panther with one chapter in it, alongside the real Black Panther (1998), (2005), (2016) and (2018). Name-only matching sends every Black Panther book to the stub. Coverage gating spreads them across the correct runs.

Scoring

Surviving candidates are scored on how well the run identity lines up:

EvidenceScore
ComicRack Volume is a 4-digit year equal to the run’s year120
ComicRack Volume is an ordinal equal to the run’s ordinal110
Issue cover year equals the run’s start year100
Run started at or before the issue’s cover year60 − (gap, capped at 50)
Run started after the issue’s cover yearnegative
Candidate has no discriminator at all (likely a stub)−20

The “started at or before” rule is the one that recovers the bulk of the library: it accepts that an issue dated 2025 belongs to the run that began in 2023, and prefers the most recent run that could plausibly contain it.

Issue lookup

Text numbers are tried first, against Chapter.Range and Chapter.Number, because Kavita stores them with MinNumber = 0.0 and a numeric-first lookup would mis-hit issue #0. Numeric lookup then tries an exact match on the parsed Range, and falls back to any chapter whose MinNumber ≤ n ≤ MaxNumber to catch collected or merged chapters.

Conflict arbitration

Matching is per book, so two different ComicRack runs can independently land on the same Kavita chapter. That is not hypothetical. In the test dataset:

  • ComicRack Punisher, volume 54, cover year 2009 → Punisher (2009) #1. Correct, score 100.
  • ComicRack The Punisher, volume 6, cover year 2011 → Punisher (2009) #1. Wrong.

Writes are therefore deferred. All books are matched, claims are grouped by chapter, and any chapter claimed by two different ComicRack runs is resolved before anything is written:

  • Same run claiming a chapter twice is a duplicate file. It merges — highest page count wins.
  • Different runs claiming one chapter: the higher-scoring claim is written, the other is dropped and reported. A tie between distinct runs is unresolvable evidence, so both are dropped.
  • Only the overlapping issues are dropped. The losing run keeps every issue it alone claims.

Skipping a book costs nothing. Writing progress to the wrong issue is silent bad data, and this is the only class of error the script cannot detect after the fact.

Usage

# Preview. Reads the database, writes nothing.
python ComicRack_Migration.py \
  --comicrack-xml ComicDb.xml \
  --kavita-db kavita.db \
  --username $Name \
  --dry-run

# Preview, saving the full list of what would be skipped.
python ComicRack_Migration.py --comicrack-xml ComicDb.xml --kavita-db kavita.db --username $Name --dry-run --report skipped.txt

# Apply.
python ComicRack_Migration.py --comicrack-xml ComicDb.xml --kavita-db kavita.db --username $Name

--username is the Kavita username, matched against AspNetUsers.UserName and case-sensitive. An unknown name aborts and lists the valid ones. --verbose prints a line per book; --report FILE writes the complete skip breakdown, grouped by reason, with every affected series.

Output

ComicRack -> Kavita  |  user: you  |  DRY RUN (no changes written)
  source: ComicDb.xml
  target: kavita.db

Read    3911 books with reading progress (4228 total in ComicRack)
Indexed 29103 series / 177718 chapters from Kavita

Would match 3578 of 3911 books (91.5%)
        3573 new, 1 would raise existing progress, 4 already at or beyond this point

Skipped 333 books
    120  series not in Kavita          35 series: PunisherMAX, Deadpool: Merc With a Mouth, ...
     98  ambiguous run, not guessed    11 series: The Punisher, Punisher: War Zone, Daredevil, ...
    101  no series/number in ComicRack
     14  issue not in Kavita           3 series: Fray, Blood Hunt: Red Band, ...

Run with --report FILE for the full list of skipped series.

DRY RUN - nothing was written. Re-run without --dry-run to apply.

Part 2 — Codex

Codex is a Django app, so its database is a plain SQLite file, codex.sqlite. Progress lives in codex_bookmark: one row per (user, comic) holding a 0-indexed page and a finished flag.

The important difference is that Codex stores the real file path of every comic in codex_comic.path, and Kavita stores the same thing in MangaFile.FilePath. If both servers point at the same library tree, the path is the match. There is no name normalization, no run scoring and no arbitration phase in this script, because there is nothing to guess at.

Lining up the two library roots

The two servers rarely mount the library at the same place. Codex might call it /comics, Kavita /data/media/comics. Both databases declare their own roots — codex_library.path and Kavita’s FolderPath.Path — so the script strips the declared root from each side and matches on what is left:

Codex   /comics/Marvel/Avengers (2023)/Avengers 007.cbz
Kavita  /data/media/comics/Marvel/Avengers (2023)/Avengers 007.cbz
            → both become  Marvel/Avengers (2023)

Kavita’s chapters are indexed by that relative directory. A chapter can own more than one file (a .cbr and a .cbz of the same issue), so the index is keyed on chapter and remembers the first file’s page count for tie-breaking.

Two escape hatches for when the trees don’t line up that cleanly:

  • Suffix fallback. If the relative directory doesn’t hit, the longest trailing fragment that identifies exactly one Kavita directory is used instead. A fragment matching two directories is treated as no match, not as a coin flip.
  • --path-map OLD=NEW. For libraries that were genuinely reorganised below the root, not just remounted. Repeatable, applied longest-prefix-first before root stripping.

Picking the issue inside the folder

Once the folder is anchored, the issue is resolved on Codex’s issue_number against Kavita’s parsed Chapter.MinNumber:

  1. Exactly one chapter carries that number → done.
  2. Several do → narrow by page count, because a duplicate scan of the same issue is the usual cause. If they all resolve to the same ChapterId it was never a real ambiguity, just two files. Genuinely different chapters with the same number and no page-count separation are skipped.
  3. No issue number at all, or Kavita parsed the run’s numbering differently → a unique page count inside that directory is still solid evidence. Anything less is skipped.

Translating a bookmark into PagesRead

Codex counts from page 0 and tracks a finished flag; Kavita counts pages completed. The mapping is not a straight copy:

  • Finished → written as fully read using Kavita’s page count, not Codex’s. The two servers may hold different scans of the same issue.
  • Unfinished → clamped to one page short of the end, so a partial read can never be mistaken for a completed one.
  • Chapter Kavita hasn’t analyzed yet (0 pages known) → Codex’s count is used. Any positive value already reads as finished, and the row becomes exactly right after the next library scan.
  • Opened but never read past page 1 → skipped entirely.

Which user?

Codex is multi-user and also keeps anonymous, session-scoped bookmarks. Guessing between two real users would silently import a stranger’s history, so:

  • One user with progress -> used automatically.
  • More than one and the script aborts and lists them with bookmark counts. Pass --codex-user.
  • Only anonymous session bookmarks = aborts, because there is nothing attributable to migrate.

Usage

# Preview. Reads both databases, writes nothing.
python Codex_Migration.py \
  --codex-db codex.sqlite \
  --kavita-db kavita.db \
  --username $Name \
  --dry-run --report skipped.txt

# Pick one of several Codex users.
python Codex_Migration.py --codex-db codex.sqlite --kavita-db kavita.db --username $Name --codex-user "[email protected]"

# The tree below the root differs too, not just the mount point.
python Codex_Migration.py --codex-db codex.sqlite --kavita-db kavita.db --username $Name --path-map "/comics/Marvel=/comics/Publishers/Marvel"

# Apply.
python Codex_Migration.py --codex-db codex.sqlite --kavita-db kavita.db --username $Name

Output

Codex -> Kavita  |  [email protected] -> you  |  DRY RUN (no changes written)
  source: codex.sqlite
  target: kavita.db

Read    1415 bookmarks for Codex user [email protected] (1415 total)
Indexed 178960 chapters across 18250 folders from Kavita

Would match 1139 of 1415 bookmarks (80.5%)
        992 new, 109 would raise existing progress, 38 already at or beyond this point

Note    508 matched comics have a different page count in Kavita than in Codex
        (different scans of the same issue; Kavita's count was used)

Note    2 matched comics have not been analyzed by Kavita yet (0 pages known)
        Codex's page count was used; run a Kavita library scan to correct them

Skipped 276 bookmarks
    208  not in this Kavita library        50 series: X-Force, X-Men, New Mutants, ...
     22  issue not in Kavita               12 series: Astonishing X-Men Infinity Comic, Avengers, ...
     46  no progress recorded in Codex

Run with --report FILE for the full list of skipped series.

DRY RUN - nothing was written. Re-run without --dry-run to apply.

Note the shape of the skips: 208 of 276 are series the target Kavita library simply does not hold. Because the path anchor is exact, a Codex skip is almost always a statement about the library, not about the matcher.


What both scripts share

Stop Kavita before running either one. SQLite tolerates concurrent readers, but Kavita caches progress in memory and can write it back over the imported rows.

Existing progress is never lowered. A row is updated only when the imported page count exceeds the stored one, which makes re-running a no-op.

Dry runs simulate their own pending writes. Duplicate source entries resolve against each other exactly as they would in a real run, so the preview summary and the applied summary are byte-identical. The preview is a prediction, not an approximation.

One user per invocation. Progress is per-user in Kavita. Run once per username.

AppUserProgresses is denormalized — it carries VolumeId, SeriesId and LibraryId alongside ChapterId. Both scripts derive those from the chapter’s own Series row in the same insert rather than carrying them through Python, so the columns cannot drift from the chapter they describe.

Timestamps come from the source: ComicRack’s Opened field, Codex’s created_at / updated_at. .NET writes seven fractional digits and a timezone offset; strptime accepts six and Kavita stores naive local time, so the offset is stripped and the fraction truncated. Books with no readable date fall back to the current time — visible in the data as a cluster of rows all stamped with the run time.

Known limitations

ComicRack

Cover year colliding with a neighbouring run’s start year. An issue whose own cover date happens to equal a different run’s debut year scores 100 against the wrong run. ComicRack Daredevil volume 7 holds the 2023-dated issues of the run that began in 2022; every one of them scores a perfect match against Daredevil (2023). Resolving this needs an order-preserving map from ComicRack volume ordinals to Kavita run years, which the source data cannot support. In practice these usually collide with the correct run’s own books and get caught by arbitration.

Legacy numbering. Marvel restarts issue numbering and periodically reverts to a cumulative count. Avengers #673 exists in ComicRack; the Kavita run that shipped it files it under its relaunch number. The coverage gate correctly refuses the run, then picks an older run that happens to contain a #673.

Codex

The libraries have to be the same tree. Path matching is the whole design. If Kavita’s library was rebuilt with a different folder layout — renamed series folders, a flattened publisher level — the suffix fallback will carry some of it and --path-map the rest, but a wholesale reorganisation is out of scope.

Page counts drift between scans. Where Kavita and Codex hold different rips of the same issue, a partial read lands on a proportionally different page. Finished reads are always exact; unfinished ones are approximate by however much the two files differ.

Anonymous bookmarks are not migrated. Codex records progress for logged-out sessions. There is no user to attribute them to, so they are skipped.