MARKDOWNER(1) User Commands MARKDOWNER(1)
NAME
markdowner - convert documents, mail, images, and audio to Markdown;
vault, search, ask
SYNOPSIS
markdowner [-v|-D] [-o FILE] [-L LANG] [--archive KIND] [--exclude
KIND] FILE
markdowner vault [-v|-D] [-j N] [-o DIR] [-L LANG] [--archive KIND]
[--exclude KIND] [-E] INDIR [INDIR...]
markdowner zettel [-v|-D] [--rebuild] VAULT [VAULT...]
markdowner logseq [-v|-D] VAULT [VAULT...]
markdowner index [-v|-D] [--no-rag] [--rebuild] [-j N] VAULT [VAULT...]
markdowner search [-v|-D] [OPTION]... VAULT [VAULT...] [QUERY...]
markdowner ask [-v|-D] [--dry-run] [--deep[=0|1|2|3|4|5]] [--no-deep]
[OPTION]... VAULT [VAULT...] [QUESTION...]
markdowner models [-v|-D] [status|update]
markdowner metrics [-v|-D] VAULT [VAULT...]
markdowner status [-v|-D] VAULT [VAULT...]
markdowner webui [-v|-D] [--no-browser] [-p PORT]
DESCRIPTION
markdowner is a single CLI (C + C++17) that converts office documents,
mail, calendars, contacts, chat exports, web/text files, images (OCR),
and audio (ASR) to Markdown for a Zettelkasten vault, FTS5 search, and
hybrid RAG ask (local BGE-M3 embeddings by default after
./tools/fetch-embed.sh; no silent hash fallback). Optional Mistral chat
via libcurl + nlohmann/json. Without MISTRAL_API_KEY, ask stays local
(retrieve-only). OCR and ASR are open-source and local (Tesseract,
Python sidecar, faster-whisper); no OCR cloud. A whole collection of
one material can skip per-page classification with --archive /
OCR_ARCHIVE (see ARCHIVE). Skip engines with --exclude / OCR_EXCLUDE
(see EXCLUDE). Converts emit quality frontmatter (quality: ok|weak,
lang, doc_kind, validation) and vault runs append _meta/metrics.jsonl.
Local plugins under plugins/*.json extend convert for extra extensions.
Graph visualization uses a local Logseq file graph (AGPL), prepared by
logseq (also run automatically after zettel). Only open-source li‐
braries are used. Each input project directory becomes its own vault
under ./vaults/ with a private sources/, notes/, and _meta/ tree (never
share one mega- sources/ across projects). Commands that take a vault
path accept one or more paths so a shell glob works (markdowner zettel
./vaults/*, markdowner search ./vaults/* foo). Directories that look
like a vault (sources/ or _meta/) are processed sequentially with a
header per vault; other glob entries (plain files, ., .., non-vault di‐
rectories) are skipped with a warning. Combined exit status is the
worst of 0, 1, or 2. For search and ask , those vault directories are
the vaults; leftover words that are not existing files or directories
are the query/question, run against each vault (no extra quoting needed
after ./vaults/*). vault accepts one or more source INDIR paths (one
vault per directory).
GLOBAL OPTIONS
-h, --help
Print help and exit.
-V, --version
Print version and exit.
-v, --verbose
Extra diagnostics on stderr (paths, OCR ensemble/HTR scores, in‐
dex/ask detail). Same as MARKDOWNER_LOG=verbose. Accepted glob‐
ally (before the subcommand) and on every subcommand. Level
ladder: INFO = progress (convert/ok/fail/done); VERBOSE = deci‐
sions/scores; DEBUG = skip size/mtime, text previews, sidecar
traces.
-D, --debug
Full diagnostics including OCR text previews; sets OCR_SIDE‐
CAR_DEBUG for Python traceback/detail. Same as MARK‐
DOWNER_LOG=debug. Always appends debug lines to a log file (see
MARKDOWNER_LOG_FILE). Vault progress events at INFO and OCR de‐
cisions at VERBOSE are included; unchanged skip lines are DEBUG
(so debug.log is not a skip dump). Accepted globally and on
every subcommand.
MARKDOWNER_LOG
quiet|info|verbose|debug (or 0|1|2|3). Default info keeps nor‐
mal progress lines.
MARKDOWNER_LOG_FILE
Append log lines to this path. With --debug / MARKDOWNER_LOG=de‐
bug a file is always used: this path if set; else
VAULT/_meta/debug.log for any vault-bound subcommand (vault,
zettel, logseq, index, search, ask, metrics, status); else
$XDG_STATE_HOME/markdowner/logs/markdowner-YYYYMMDD.log (fall‐
back ~/.local/state/markdowner/logs/). Set to 0 / off to dis‐
able file logging. Line format (UTC): YYYY-MM-DDThh:mm:ss.sss‐
ZLEVELmessage. Stderr prints markdowner: log file → when a file
is opened.
MARKDOWNER_LOG_DIR
Directory for auto daily logs when MARKDOWNER_LOG_FILE is unset
(default under XDG_STATE_HOME).
CONVERT MODE
Default mode: convert one FILE to Markdown.
-o, --output=PATH
Output path (or directory for multi-message formats such as
mbox/pst).
-L, --lang=LANG
OCR/ASR language for image and audio inputs (e.g. nld or
nld+eng). Default is nld when Dutch tessdata is installed;
overridable with OCR_LANG / ASR_LANG. For ASR, Tesseract-style
codes are mapped (nld→nl).
-A, --archive=KIND
Force OCR engines for a whole collection: auto (default: clas‐
sify each page), print / drukwerk (RapidOCR/Tesseract, no HTR),
newspaper / krant (same, dense print), handwriting / handschrift
(VLM: Qwen2.5-VL, HTR fallback), historic / arletta (AR‐
letta/kraken only), modern / trocr (TrOCR only), mixed (print
then always HTR), form / formulier (printed form + handwritten
fields). Same as OCR_ARCHIVE. See ARCHIVE for the full table
of kinds and which engines they force.
-X, --exclude=KIND[,KIND...]
Skip engines. Same kinds as --archive (comma-separated). Exam‐
ple: --exclude historic skips ARletta/kraken so only TrOCR re‐
mains for handwriting. See EXCLUDE.
-v, --verbose / -D, --debug
Same as GLOBAL OPTIONS; debug writes a UTC log file (daily under
XDG state unless MARKDOWNER_LOG_FILE is set).
Supported extensions: .docx .doc .odt .pdf .xlsx .ods .pptx .odp .epub,
.eml .mbox .msg .pst .ost .ics .vcf .vcard, .html .htm .md .markdown
.txt .text .csv .tsv .rtf .json, .tif .tiff .jpg .jpeg .png, .wav .mp3
.m4a .aac .ogg .flac .webm .wma .opus.
Office Open XML / ODF / EPUB / HTML / RTF prefer pandoc (subprocess)
when installed; otherwise ZIP+XML or lightweight extractors (with a
warning). If pandoc is installed but fails, the builtin parser is re‐
fused unless MARKDOWNER_LEGACY=1 . Disable pandoc with MARKDOWNER_PAN‐
DOC=0. Spreadsheets and presentations prefer LibreOffice ( soffice →
HTML → pandoc, not the builtin HTML parser) when installed. The ZIP ex‐
tractors cap at 500 rows / 50 columns / 200 slides and refuse truncated
output unless MARKDOWNER_LEGACY=1. Disable LibreOffice with MARK‐
DOWNER_SOFFICE=0. Legacy .doc uses antiword / catdoc / LibreOffice.
PDF uses pdftotext (Poppler); if the result is sparse and pdftoppm or
pdftocairo is on PATH, pages are rendered and OCR'd (Paddle sidecar
when available, else Tesseract). Audio files are transcribed with a
local faster-whisper sidecar (requires ffmpeg and tools/fetch-whis‐
per.sh). See AUDIO (ASR) below. Chat: WhatsApp-style .txt and JSON
messages arrays (also .json).
Optional convert plugins: plugins/*.json with ext, cmd, and type (over‐
ride directory with MARKDOWNER_PLUGINS). See plugins/example.json.
QUALITY
Markdowner does not silently degrade Office ingest or RAG embeddings.
Office (pandoc)
docx / odt / epub / html / rtf use pandoc when it is on PATH.
If pandoc is present but conversion fails, the ZIP/XML builtin
parser is not used unless MARKDOWNER_LEGACY=1. If pandoc is
missing, the builtin parser runs with a warning.
Sheets / slides (LibreOffice + pandoc)
xlsx / ods / pptx / odp use soffice / libreoffice to HTML, then
pandoc to Markdown. Builtin html_to_md is only a warned fallback
when pandoc is missing or fails on that HTML. If soffice is
present but conversion fails, ZIP extractors are refused unless
MARKDOWNER_LEGACY=1.
Truncation
ZIP sheet/slide extractors stop at 500 rows / 50 columns / 200
slides and refuse to write a truncated document unless MARK‐
DOWNER_LEGACY=1.
RAG embeddings
ASK_RAG_EMBED=auto (the default) requires a fetched local model
(BAAI/bge-m3 via ./tools/fetch-embed.sh). Without that model,
index/ask fail (exit 2); they never silently switch to hash‐
ing-trick vectors. Skip embeddings with index --no-rag or MARK‐
DOWNER_RAG_INDEX=0. ASK_RAG_EMBED=hash is debug-only (weak re‐
call).
See ENVIRONMENT for MARKDOWNER_PANDOC, MARKDOWNER_SOFFICE, MARK‐
DOWNER_LEGACY, and ASK_RAG_EMBED.
AUDIO (ASR)
markdowner transcribes audio to Markdown via a local faster-whisper
sidecar (offline after model download). Supported extensions: .wav .mp3
.m4a .aac .ogg .flac .webm .wma .opus.
Setup (once):
• install ffmpeg on PATH
• pip install -r tools/requirements-asr.txt into third_party/ocr-venv
• ./tools/fetch-whisper.sh (default model large-v3 → third_party/mod‐
els/whisper-large-v3/)
Usage:
markdowner -L eng -o interview.md interview.mp3
ASR_TIMESTAMPS=1 markdowner -o ts.md clip.wav
markdowner vault ./recordings
-L / OCR_LANG / ASR_LANG select the language (nld maps to Whisper nl;
use auto to detect). Set ASR_TIMESTAMPS=1 for lines prefixed with
[mm:ss]. Output frontmatter uses type: asr, asr_engine, and optional
asr_confidence.
Related environment variables: ASR_MODEL, ASR_MODEL_DIR, ASR_SIDE‐
CAR_SERVE, ASR_SIDECAR_TIMEOUT, MARKDOWNER_ASR_SIDECAR (see ENVIRON‐
MENT).
CALENDAR (ICS)
markdowner parses iCalendar (RFC 5545) files into one Markdown note per
calendar file. YAML frontmatter uses type: ics plus events, optional
calname/prodid/method, and words. Each VEVENT (and VTODO) becomes a
heading with start/end, location, categories, recurrence, organizer,
attendees, and description. Folded lines and text escapes (\n, \,) are
handled; timezone definition blocks are skipped.
ARCHIVE
--archive / -A / OCR_ARCHIVE declares the material of a whole collec‐
tion so Markdowner can skip per-page classification and force the
matching engines from RapidOCR, Tesseract, Qwen2.5-VL, ARletta/kraken,
and TrOCR. The web UI Archief dropdown is the same flag (stored as
OCR_ARCHIVE outside the engine-slider profile). The slider still con‐
trols how hard OCR runs; the archive flag wins for print vs handwriting
routing.
Canonical values (Dutch and library aliases in parentheses). Unknown
--archive is a usage error. Unknown OCR_ARCHIVE falls back to auto with
a warning. Non-auto kinds are written to YAML frontmatter as
ocr_archive.
auto (any, unknown, dynamic, default) Classify each page (the de‐
fault). Print OCR first; handwriting or print-garbage uses a lo‐
cal VLM (Qwen2.5-VL) then TrOCR/ARletta as fallback. Does not
rewrite OCR_HTR / OCR_SIDECAR_PIPELINE from the slider.
print (drukwerk, typed, typeset, printed) RapidOCR/Tesseract only;
OCR_HTR=0. Use for typed reports and books.
newspaper
(krant, kranten, news, newsprint) Same as print; treat the col‐
lection as dense newsprint (no HTR).
handwriting
(handschrift, htr, handwritten, cursive) Force the handwriting
pipeline (OCR_SIDECAR_PIPELINE=vlm unless OCR_VLM=0): local
Qwen2.5-VL, then TrOCR/ARletta if the VLM is off or empty.
historic
(historisch, arletta, kraken, archival) HTR with ARletta/kraken
only (early-modern / 19–20c Dutch).
modern (nieuw, trocr, notebook, contemporary) HTR with TrOCR only (mod‐
ern notebooks and forms of contemporary cursive).
mixed (gemengd, mix, both) Print OCR, then VLM (and HTR fallback) on
handwriting or print-garbage.
form (formulier, forms) Printed form plus handwritten fields (print
then VLM/HTR).
Examples: markdowner --archive handwriting -o letter.md scan.jpg
markdowner vault --archive print ./scans-newspapers
OCR_ARCHIVE=historic markdowner vault ./letters
EXCLUDE
--exclude / -X / OCR_EXCLUDE turns engines off using the same kinds as
--archive (comma, plus, or space separated). Archive says what the ma‐
terial is; exclude says which of those engines not to run. Applied af‐
ter archive. The web UI checkboxes write OCR_EXCLUDE outside the en‐
gine-slider profile.
historic
(historisch, arletta, kraken) Skip ARletta/kraken. Handwriting
still uses VLM + TrOCR.
modern (nieuw, trocr) Skip TrOCR. Handwriting still uses VLM + ARletta.
handwriting
(handschrift, htr) Skip VLM and HTR (print-only).
print / newspaper (drukwerk, krant) Skip RapidOCR/Tesseract (VLM/HTR-
only).
mixed / form Do not force HTR after print for that archive kind (dy‐
namic skip remains).
Unknown --exclude tokens are a usage error. A combination that leaves
no engine (--archive historic --exclude historic) is a usage error. Un‐
known OCR_EXCLUDE is ignored with a warning. Non-empty lists are writ‐
ten to YAML as ocr_exclude.
Examples: markdowner --archive handwriting --exclude historic -o let‐
ter.md scan.jpg
OCR_EXCLUDE=arletta markdowner vault ./notebooks
OCR
Image conversion uses a dynamic quality ensemble by default (OCR_EN‐
GINE=auto). Per page it may run in-process Tesseract and/or the Python
sidecar (tools/ocr_sidecar), then keep the better text by text-quality
score (HTR is never chosen solely because the engine name is HTR/AR‐
letta). The sidecar classifies each page as print, handwriting, or
mixed from ink/line regularity and print-OCR quality without path or
folder rules. For a whole vault of one material, set --archive /
OCR_ARCHIVE (see ARCHIVE) instead of guessing per page. Handwriting
pages use ARletta / kraken HTR when models are present
(./tools/fetch-arletta.sh); mixed pages may merge a RapidOCR letterhead
with an HTR body. When both print OCR and HTR run, the higher quality
score wins; a strong print score keeps print text even if the page was
tentatively classified as handwriting. Set OCR_QUALITY=fast to skip
the Tesseract+sidecar ensemble (single print engine; HTR fallback may
still run).
Overrides (optional): OCR_SIDECAR_PIPELINE=htr forces handwriting;
rapidocr / paddleocr / structurev3 force those print paths. OCR_HTR=0
disables HTR; default is always (run HTR when print OCR did not already
solve the page — handwriting, mixed, weak print). OCR_HTR=auto uses
the same dynamic skip. English TrOCR remains a fallback
(./tools/fetch-trocr.sh) when ARletta/kraken is unavailable or weak.
YAML frontmatter includes type: ocr, ocr_engine (tesseract / rapidocr /
arletta:super / trocr-handwritten / paddle-…), optional ocr_confidence,
lang, lang_detected, doc_kind, ocr_archive, quality, validation, plus
ingest metrics image_width, image_height, dpi_est, skew_deg, bright‐
ness, contrast.
Environment (common): OCR_ENGINE (auto|paddle|tesseract), OCR_ARCHIVE
(auto|print|newspaper|handwriting |historic|modern|mixed|form), OCR_EX‐
CLUDE (comma-separated kinds to skip; see EXCLUDE), OCR_QUALITY (fast
skips ensemble), MARKDOWNER_OCR_SIDECAR, OCR_SIDECAR_TIMEOUT (default
1800 when HTR is enabled; override up to 7200), OCR_SIDECAR_SERVE,
OCR_SIDECAR_PIPELINE (auto|rapidocr|htr|arletta|paddleocr|structurev3),
OCR_HTR (auto|always|0), OCR_HTR_ENGINE (auto|kraken|arletta|trocr),
OCR_HTR_ARLETTA_DIR, OCR_HTR_MODEL, OCR_HTR_LINE_ENSEMBLE, OCR_PRE‐
PROCESS (handwriting optional gentler cursive preprocess), OCR_BRIGHT‐
NESS_AUTO (0 disables auto brightness TRC on dark/bright scans),
OCR_CRITICAL (1 uses a longer sidecar timeout, 3600 s, for hard pages),
OCR_LANG, OCR_PSM (Tesseract page segmentation 0–13; default AUTO),
OCR_BINARIZE, OCR_MIN_LONG_EDGE (minimum long-edge pixels after up‐
scale; default 2400), OCR_MAX_LONG_EDGE (clamp; default 4500),
OCR_PDF_DPI (default 350), OCR_PDF_JOBS (parallel page OCR; default 2),
OCR_SIDECAR_INIT_TIMEOUT (StructureV3 isolate timeout; default 120).
Long OCR runs print progress on stderr (ocr: sidecar still running…
every 10 s; sidecar backend lines). Vault -j > 1 sets
OMP_NUM_THREADS=1 when unset to avoid OpenMP oversubscription. Scanned
PDFs with little extractable text are rendered via pdftoppm / pdfto‐
cairo when available, then OCR'd (no hard page cap). See
third_party/README.md for sidecar install (Python 3.10–3.12) and HTR
model fetch.
Traineddata is resolved from TESSDATA_PREFIX (the tessdata directory),
a build-time default under third_party/prefix, or common system paths
(/usr/share/tesseract-ocr/5/tessdata, …). Install language packs such
as tesseract-ocr-eng and tesseract-ocr-nld when using the system
Tesseract. Optional tesseract-ocr-osd enables orientation detection.
QUALITY AND VALIDATION
Every convert writes YAML quality fields when possible: quality:
ok|weak, lang / lang_detected (NL/DE/EN/FR frequency guess), doc_kind
(e.g. letter, newspaper, form, handwritten), and validation (remark
list). Empty bodies fail; weak text is marked ok-weak in vault progress
and still written. Optional local hunspell adds spell remarks; if
missing, remarks include spell: unavailable. Vault appends one JSON
line per file to _meta/metrics.jsonl (timings and quality). Summarize
with markdowner metrics VAULT.
MODELS SUBCOMMAND
markdowner models [-v|-D] status reports on-disk OSS models (ARletta,
TrOCR, Whisper, tessdata, BGE-M3). markdowner models [-v|-D] update
runs the local fetch scripts (tools/fetch-arletta.sh, fetch-trocr.sh,
fetch-whisper.sh, fetch-embed.sh); network is used only for downloads.
-v, --verbose / -D, --debug
Same global log levels.
METRICS SUBCOMMAND
markdowner metrics [-v|-D] VAULT [VAULT...] prints counts from
_meta/metrics.jsonl (lines / ok / weak / fail).
-v, --verbose / -D, --debug
Same global log levels; debug log under VAULT/_meta/debug.log.
STATUS SUBCOMMAND
markdowner status [-v|-D] VAULT [VAULT...] prints a health summary per
vault: resolved path, source counts, manifest ok/weak/fail/skip,
zettel/logseq, FTS doc count, RAG (chunks.jsonl vs embeddings.f32 row
counts, including mismatch/corrupt), last run.log vault/index lines,
and inconsistencies (missing source files, empty RAG dir, unreadable
SQLite). Multiple paths are accepted so a shell glob works: markdowner
status ./vaults/*. Non-vault entries (plain files, ., ..) are skipped
with a warning. Soft warnings (no RAG yet) exit 0; exit 1 if no path
was a valid vault; exit 2 if any vault has hard corruption (RAG mis‐
match, unreadable FTS). Alias: vault-status.
-v, --verbose / -D, --debug
Same global log levels; debug log under VAULT/_meta/debug.log.
WEBUI SUBCOMMAND
markdowner webui starts the local Web UI (tools/webui/server.py) on
127.0.0.1:8877 (next free port in +20 if busy) and opens that URL in
the default browser. Alias: markdowner ui. Stop with Ctrl+C.
The server is stdlib Python (no extra pip packages). It must be able to
find tools/webui/server.py (run from the source tree, or set MARK‐
DOWNER_ROOT / MARKDOWNER_WEBUI). Jobs spawned by the UI use this mark‐
downer binary when MARKDOWNER_BIN is unset (set automatically from
/proc/self/exe).
--no-browser, --no-open
Start the server without opening a browser (headless / SSH).
-p, --port=PORT
Preferred bind port (same as MARKDOWNER_WEBUI_PORT).
-v, --verbose / -D, --debug
Same global log levels.
VAULT SUBCOMMAND
markdowner vault [-v|-D] INDIR [INDIR...] walks each INDIR recur‐
sively, converts every supported document, mail, or image file, and
writes a Zettelkasten vault (full conversions under sources/; run
zettel afterwards for atomic notes). Source files are never modified.
Re-runs skip unchanged files (size+mtime, then SHA-256 in the mani‐
fest). Conversion uses a worker pool. Progress is printed on stderr as
vault: [n/N] convert|ok|skip|fail|ok-weak with per-file timing and to‐
tal elapsed time. Also appends _meta/metrics.jsonl. Multiple source
directories (shell glob markdowner vault ./bron/*) build one vault per
INDIR. Non-directories are skipped. Combined exit is the worst fail‐
ure. With a Paddle OCR sidecar, prefer -j 2 to 4 (heavy models);
higher -j is fine with OCR_ENGINE=tesseract.
-v, --verbose / -D, --debug
Same global log levels; debug and run logs under VAULT/_meta/de‐
bug.log and VAULT/_meta/run.log.
-o, --output=DIR
Vault directory. Default: ./vaults/basename(INDIR) (trailing
slashes on INDIR are ignored when deriving the name, so ./fix‐
tures/testdata/ becomes vaults/testdata, not vaults/vault). If
DIR is the vaults container (final path component vaults, or a
directory that already holds sibling vaults), Markdowner uses
DIR/basename(INDIR) instead, so each project keeps its own
sources/, notes/, and _meta/. Do not use -o ./vaults expecting
a single shared sources/ tree.
-E, --each
Create one vault per immediate subdirectory of INDIR (under
./vaults or under -o).
-j, --jobs=N
Number of parallel workers. Default and maximum: 90% of online
CPUs (at least 1). A higher -j is reduced with a warning. With
the OCR sidecar the pool may be capped lower still (typically
2–4).
-L, --lang=LANG
OCR language for images found during ingest.
-A, --archive=KIND
Same as convert --archive : force print vs handwriting engines
for every image in this vault (OCR_ARCHIVE; see ARCHIVE).
-X, --exclude=KIND[,KIND...]
Same as convert --exclude (see EXCLUDE).
ZETTEL SUBCOMMAND
markdowner zettel [-v|-D] [--rebuild] VAULT [VAULT...] reads
VAULT/sources, splits documents on Markdown headings and horizontal
rules (---), writes Folgezettel-linked notes under notes/, updates in‐
dex.md (keyword register), writes _meta/isolation.md, and prepares a
local Logseq file graph (see LOGSEQ). Multiple vaults (shell glob
markdowner zettel ./vaults/*) run sequentially; non-vaults are skipped.
Combined exit is the worst failure.
Default is incremental : unchanged sources (SHA-256, recorded in
_meta/zettel.jsonl) are skipped and their existing notes are reused.
Sources that gained or lost sections overwrite notes in place and drop
leftover numbered files. Notes whose source file disappeared are re‐
moved. The keyword register and isolation report are always rewritten
from the current notes. On a first run after upgrading from 1.x,
sources whose notes/*/0001.md is at least as new as the source are
treated as unchanged (bootstrap).
--rebuild
Delete existing notes/ , index.md , _meta/isolation.md , and
_meta/zettel.jsonl then rebuild from sources/. Use this after a
botched incremental run, or when you want a guaranteed clean
tree. A full rebuild can take a long time on large vaults.
-v, --verbose / -D, --debug
Same global log levels; debug log under VAULT/_meta/debug.log.
LOGSEQ SUBCOMMAND
markdowner logseq [-v|-D] VAULT [VAULT...] turns VAULT into a Logseq
file graph for offline open-source graph view: creates pages/, jour‐
nals/, assets/, and logseq/config.edn (sync/journals off; home page in‐
dex; sources/ and _meta/ hidden). Notes under notes/ are exposed as
page symlinks using Logseq namespace filenames (stem/0001 →
pages/stem___0001.md). Re-run after regenerating notes; safe and idem‐
potent. Install Logseq with tools/fetch-logseq.sh or your distro pack‐
age, then Add local graph on the vault folder. See LOGSEQ.md in the
vault.
-v, --verbose / -D, --debug
Same global log levels; debug log under VAULT/_meta/debug.log.
INDEX SUBCOMMAND
markdowner index [-v|-D] [--no-rag] [--rebuild] [-j N] VAULT [VAULT...]
builds or refreshes an on-disk FTS5 index at VAULT/_meta/inv/fts.db
and, unless skipped, a hybrid RAG store under VAULT/_meta/rag/ (chunks
+ embeddings used by ask). After upgrading Markdowner, run a full
reindex if the store format changed (older MDDC002 shard trees are un‐
used). Multiple vaults (shell glob markdowner index ./vaults/*) run
sequentially; non-vaults are skipped.
Default is incremental docs table and only inserts, updates, or deletes
changed rows. RAG reuses chunks and vectors for files whose SHA still
matches _meta/rag/file_cache.json (same embed model and chunk set‐
tings); only new or changed files are embedded. An unchanged vault
prints FTS5 up to date and ask_rag: up to date and exits quickly. If a
previous RAG write was interrupted (chunks.jsonl longer than embed‐
dings.f32), the next incremental index does not abort; it resumes a
matching embed checkpoint or rebuilds vectors without reusing the mis‐
matched store.
Search and ask prefer this index. After vault or zettel runs, invoke
index explicitly (ingest does not always fully reindex). Incremental
index after adding a few documents is the cheap path; --rebuild is the
expensive full rewrite.
Local embeddings default to BAAI/bge-m3 after ./tools/fetch-embed.sh
(sentence-transformers; CPU can take many hours on large vaults).
Without a fetched model, ASK_RAG_EMBED=auto fails (no silent hash). Use
./tools/fetch-embed.sh or index --no-rag (or explicit ASK_RAG_EM‐
BED=hash for debug). Local embeds checkpoint to embeddings.partial.f32
/ embed_progress.json so an interrupted run can resume (same new-chunk
fingerprint). On GPU the default encode batch soft-bumps to 64 (over‐
ride with ASK_RAG_EMBED_BATCH). Tune chunk size / batch / backend via
ASK_RAG_* (see ENVIRONMENT).
-j, --jobs=N
Parallel FTS workers (default and maximum: 90% of online CPUs).
Progress lines index: [n/N] docs are printed periodically on
stderr.
--no-rag
Build FTS only; skip _meta/rag/ embeddings. Same as MARK‐
DOWNER_RAG_INDEX=0.
--rebuild
Wipe the FTS database and _meta/rag/ store, then rebuild from
scratch. Skips the “up to date” short-circuit and the incremen‐
tal FTS/RAG path, and does not resume embed checkpoints. On
large vaults this can take a long time (especially BGE-M3 embed‐
dings).
-v, --verbose
Extra FTS/RAG progress; also raises MARKDOWNER_LOG to verbose
and passes -v to tools/ask_rag.
-D, --debug
Full diagnostics and VAULT/_meta/debug.log. Implies verbose for
the RAG sidecar.
SEARCH SUBCOMMAND
markdowner search [-v|-D] VAULT [VAULT...] [QUERY...] searches Mark‐
down under each vault. Multiple QUERY words are combined with AND at
document level; matching lines are printed as path:line: text.
After a shell glob (markdowner search ./vaults/* foo), directories that
look like a vault are vaults; other directories, existing files, ., and
.. are skipped with a warning; leftover words that are not existing
paths are the query. The same query runs on every vault; results are
labeled when more than one path was given. Combined exit is the worst
of 0, 1, or 2. Multiple vaults require a QUERY.
If _meta/inv/fts.db is present, the query uses FTS5 (no full corpus
scan). Otherwise it scans and prints a hint to run index. Case-sensi‐
tive search always uses the scan path.
If only VAULT is given on a TTY, an interactive search> prompt reads
queries until quit, EOF, or Ctrl+C.
-n, --max-count=N
Maximum number of hits to print (default: 30).
-s, --sources
Search only sources/.
-t, --notes
Search only notes/.
-c, --case-sensitive
Match case-sensitively (default is case-insensitive).
-v, --verbose / -D, --debug
Same global log levels; debug log under VAULT/_meta/debug.log.
ASK SUBCOMMAND
markdowner ask [-v|-D] VAULT [VAULT...] [QUESTION...] answers ques‐
tions over one or more vaults. Vault vs question uses the same split as
search (vault directories vs leftover words). The same question runs on
each vault (markdowner ask ./vaults/* Multiple vaults require a QUES‐
TION.
Preferred path: hybrid RAG via tools/ask_rag (BM25 over chunks + embed‐
dings, RRF fusion, then grounded chat). markdowner index builds both
the FTS5 index (_meta/inv/fts.db) and the RAG store (_meta/rag/) with
local OSS embeddings by default (one-time ./tools/fetch-embed.sh; no
API key). Skip embeddings with index --no-rag or MARKDOWNER_RAG_IN‐
DEX=0. Embed backends (ASK_RAG_EMBED): auto (local BGE-M3 if fetched,
else fail; no silent hash), local (BGE-M3 / sentence-transformers),
hash (debug only; fast deterministic vectors; weaker semantic recall),
mistral (cloud; needs MISTRAL_API_KEY). Large corpora: raise
ASK_RAG_CHUNK_CHARS (default 1800) to cut chunk count, raise
ASK_RAG_EMBED_BATCH (default 16 on CPU; soft 64 on GPU when unset), use
ASK_RAG_EMBED=hash for a quick index, or install GPU PyTorch
(./tools/ensure-torch-gpu.sh) so local encodes use CUDA/ROCm. Inter‐
rupted local embeds resume from checkpoint. You can still run
tools/ask_rag index VAULT
manually. When the RAG store exists, markdowner ask delegates to the
sidecar automatically. Without a RAG store, ask falls back to FTS hits
from fts.db (or errors if neither exists). Default is RAG/FTS only;
keep the older C++ lexical retrieve with MARKDOWNER_ASK=legacy. Over‐
ride the script path with MARKDOWNER_ASK_RAG.
Default ask is --deep=5 (investigative loop). Use --deep=0 or --no-deep
for one hybrid retrieve (BM25 pool + vector pool, RRF fusion, about 12
chunks) then one grounded chat — a nearest-neighbour sample, not a scan
of the vault. The whole sources/ and notes/ trees are already in the
store (recursive); deep does not “open extra subdirs”. It re-queries
the same store, then (from L2) partitions only along axes that this
archive actually has. A flat scan vault without dates skips the year
axis and still uses unread threads. L4 adds spelling variants for
zero-hit tokens, hop-2 names (new names searched alone), and adjacent
files in the same folder. L5 pairs those names with every seed term
and adds extra vector queries. Never a full census: gaps stay gaps.
Shop/order/newsletter chunks are downranked unless the question names
them.
Legacy lexical ask retrieves passages with the vault index/search and
hand-tuned keyword heuristics (frozen; do not extend synonym tables
there). When both sources/ and notes/ hit the same converted document,
legacy ask keeps the notes/ passage (dedupe) so RAG context is not dou‐
bled. Chat uses local Ollama when ASK_RAG_CHAT=ollama (or auto without
a Mistral key), else Mistral Chat Completions if MISTRAL_API_KEY is
set. Optional cloud embed model: MISTRAL_EMBED_MODEL (default mis‐
tral-embed). Without chat backend (or with --dry-run), ask stays re‐
trieve-only and prints chunks. Optional MISTRAL_MODEL / ASK_RAG_OL‐
LAMA_MODEL. With only VAULT on a TTY, an interactive ask> prompt reads
questions until quit, EOF, or Ctrl+C. A new markdowner ask process
starts with empty history (nothing is loaded from an earlier CLI ask or
from the WebUI). Prior questions and answers are kept in this session
only and sent with each call (follow-ups like “what did she say about
that?” work). Type clear to reset this session. Phrases search deeper,
more sources, dieper zoek, grondiger zoek, and meer bronnen raise the
deep level by one (max L5) and repeat the last content question.
Retrieval regression (example vault):
tools/ask_rag eval VAULT testdata/ask_rag_cases_mt_beraad.json
-v, --verbose
Raises log level and ask/RAG sidecar verbosity.
-D, --debug
Full diagnostics and VAULT/_meta/debug.log. Implies --verbose
for the ask sidecar.
--dry-run
Retrieve only (no chat API call).
--deep[=0|1|2|3|4|5]
Investigative retrieve (default 5; 0 turns it off). Not a bigger
top-k; not a vault census. Also --no-deep for level 0. The
store already contains every subdirectory; levels add rounds and
axes, not extra folders.
L1 --deep=1. Decompose the question (synonyms + names), ex‐
tra BM25 sweeps plus a few vector queries, dedupe
notes/sources/copies/page splits, then lexical follow-ups
on names/years found in the evidence.
L2 --deep=2 or --deep2. L1 plus partitions that exist in
this store: unread threads (high-scoring chunks whose
thread was not kept); path prefixes if the corpus is not
one blob; years if enough paths contain 19xx/20xx. A
vault without those signals skips those axes (no assumed
INBOX or year directories).
L3 --deep=3 or --deep3. L2 plus sibling chunks of chosen
threads (other pages of the same document) and a larger
packed context.
L4 --deep=4 or --deep4. L3 plus spelling variants for to‐
kens that scored 0 BM25 (e.g. teeken→teken), hop-2 names
searched alone (not only paired with the original ques‐
tion), and adjacent files in the same folder.
L5 --deep=5 or --deep5. Default. L4 plus pairing hop-2
names with every seed term, extra hop-2 vector queries,
and more thread/neighbor expansion.
Optional ASK_RAG_DEEP_LLM=1 adds chat synonyms (slower). Caps:
ASK_RAG_DEEP_ROUNDS, ASK_RAG_DEEP_FINAL, ASK_RAG_DEEP_MAX_CHARS
(see ENVIRONMENT). L2+ use the GPU embed sidecar for query vec‐
tors; L4/L5 add more. Do not run them beside a heavy index em‐
bed on a small iGPU.
-s, --sources / -t, --notes
Restrict to sources/ or notes/.
-n, --max-count=N
Legacy lexical hit count (default 40).
-C, --context=N
Legacy context lines (default 24).
--max-chars =N
Context budget (default 48000).
--retrieve =WORDS
Override retrieval keywords (legacy lexical only).
Vault snippets are sent to the Mistral API only when MISTRAL_API_KEY is
set; use cloud ask only on approved corpora. Load the key from ./.env
or ~/.config/markdowner/.env (see .env.example; chmod 600; never commit
.env). Already-exported environment variables always win; Markdowner
never prints secret values. Override the file with MARKDOWNER_DOTENV.
MAIL EXPORTS
Gmail Takeout
Export mail as .mbox then markdowner vault on the Takeout folder
(or convert the mbox directly).
Outlook
Save messages as .eml or .msg , or place a .pst in the ingest
tree. Encrypted PST/OST is skipped with an error; prefer
EML/MBOX when possible. PST extraction is best-effort for mes‐
sages that still carry MIME on disk.
Each message becomes one Markdown file with YAML frontmatter (from, to,
subject, date, message_id). Attachments are listed but not extracted
as binary files. Thread grouping is not implemented yet.
VAULT LAYOUT
One input directory becomes one vault under the vaults/ container
(never use the container itself as vault root):
vaults/
<project>/
home.md
index.md
sources/ full document / mail / OCR / ASR conversions
notes/ atomic zettels
pages/ Logseq page symlinks (→ notes/)
journals/ Logseq journals (disabled in config)
assets/ Logseq assets
logseq/ config.edn (local file graph)
LOGSEQ.md how to open the graph
_meta/
manifest.jsonl
metrics.jsonl per-file timings / quality (ok|weak|fail)
run.log vault run (UTC ISO lines)
debug.log --debug / MARKDOWNER_LOG=debug (when no LOG_FILE)
isolation.md
inv/ FTS5 index (fts.db)
rag/ hybrid RAG (chunks.jsonl, embeddings.f32, vectors.usearch, manifest.json)
Each vault has its own sources/. Sibling projects do not share a
sources tree.
ENVIRONMENT
MARKDOWNER_PANDOC
Set to 0 / off / legacy to skip Pandoc and use built-in Of‐
fice/HTML/RTF/EPUB extractors (with a warning). Default: use
pandoc on PATH when the format is supported (docx, odt, epub,
html, rtf). If pandoc is installed but conversion fails, Mark‐
downer refuses the builtin parser unless MARKDOWNER_LEGACY=1.
LibreOffice HTML for sheets/slides is also converted with pandoc
when available.
MARKDOWNER_SOFFICE
Set to 0 / off / legacy to skip LibreOffice for
xlsx/ods/pptx/odp and use ZIP+XML extractors (with a warning;
truncated output is still refused unless MARKDOWNER_LEGACY=1).
Default: use soffice / libreoffice on PATH when present (HTML →
pandoc). If soffice is installed but conversion fails, ZIP ex‐
tractors are refused unless MARKDOWNER_LEGACY=1.
MARKDOWNER_LEGACY
Set to 1 to allow builtin Office parsers after pandoc/soffice
failure, and to accept truncated ZIP sheet/slide extractors (500
rows / 50 columns / 200 slides). Default: refuse those fall‐
backs when the preferred tool was available, and refuse trun‐
cated ZIP ingest.
MARKDOWNER_WEBUI_HOST
Bind address for markdowner webui (default 127.0.0.1).
MARKDOWNER_WEBUI_PORT
Preferred port for markdowner webui (default 8877; the server
tries the next 19 ports if busy).
MARKDOWNER_WEBUI
Override path to tools/webui/server.py .
MARKDOWNER_BIN
Override path to the markdowner binary used by the Web UI for
jobs. Set automatically by markdowner webui when unset.
MISTRAL_API_KEY
API key for cloud ask chat (and optional ASK_RAG_EMBED=mistral).
Prefer storing it in ./.env or ~/.config/markdowner/.env (chmod
600; see .env.example) rather than exporting it in shell his‐
tory. If unset, ask uses local Ollama when available, else re‐
trieve-only (same as --dry-run without a chat backend).
MISTRAL_MODEL
Chat model name (default mistral-large-latest).
MISTRAL_EMBED_MODEL
Cloud embed model when ASK_RAG_EMBED=mistral (default mis‐
tral-embed).
MARKDOWNER_RAG_INDEX
Set to 0 to skip RAG embeddings during index (same as --no-rag).
MARKDOWNER_ASK
rag (default: hybrid RAG / FTS via tools/ask_rag; no C++ lexical
fallthrough), legacy (frozen C++ heuristics only), or auto (same
as rag).
MARKDOWNER_ASK_RAG
Override path to tools/ask_rag / ask_rag.py.
ASK_RAG_EMBED
auto (default: local BGE-M3 if fetched, else fail; no silent
hash), local, hash (debug, weak recall), mistral, or fake
(tests).
ASK_RAG_CHUNK_CHARS
Chunk size in characters when building _meta/rag/ with chonkie
RecursiveChunker (default 1800). Larger values → fewer chunks →
faster embeds on big vaults.
ASK_RAG_CHUNK_OVERLAP
Overlap passed to chonkie OverlapRefinery (default 200).
ASK_RAG_EMBED_BATCH
Texts per encode batch (default 16 on CPU). When unset on
CUDA/ROCm, soft-default 64 (same model; higher throughput).
Raise further if VRAM allows; OOM auto-splits the batch.
ASK_RAG_EMBED_BATCH_GPU
Soft GPU batch when ASK_RAG_EMBED_BATCH is unset (default 64).
ASK_RAG_LOCAL_MODEL
Hub id for forced local load (default BAAI/bge-m3).
ASK_RAG_LOCAL_MODEL_DIR
Directory of a fetched sentence-transformers model (default
third_party/models/embed-bge-m3).
ASK_RAG_DEVICE
Force embed device: cpu, cuda, or cuda:0. Default: cuda when
torch.cuda.is_available() (NVIDIA CUDA or AMD ROCm), else cpu.
Install GPU wheels with ./tools/ensure-torch-gpu.sh.
ASK_RAG_CHAT
auto, ollama, or mistral.
ASK_RAG_OLLAMA_MODEL
Ollama chat model (default qwen2.5:72b).
ASK_RAG_TOP_VEC
Vector retrieval pool size (default 24).
ASK_RAG_TOP_BM25
BM25 retrieval pool size (default 24).
ASK_RAG_TOP_FINAL
Final fused hit count (default 12).
ASK_RAG_MAX_CHARS
Context budget for RAG packing (default 48000). Deep L2+ raise
this unless ASK_RAG_DEEP_MAX_CHARS is set.
ASK_RAG_DEEP_LLM
Set to 1 to add a chat pass for extra lexical synonyms during
--deep (default off; heuristic only).
ASK_RAG_DEEP_ROUNDS
Override deep follow-up rounds (defaults: L1=3, L2=4, L3=5,
L4=6, L5=7).
ASK_RAG_DEEP_FINAL
Override how many diverse hits deep packs (defaults: L1=24,
L2=36, L3=48, L4=56, L5=64).
ASK_RAG_DEEP_MAX_CHARS
Override deep context budget (defaults: L1=64000, L2=80000,
L3=96000, L4=112000, L5=128000).
OCR_LANG
Default OCR/ASR language when -L is omitted (default nld).
ASR_LANG
ASR language override (mapped to Whisper ISO codes; nld→nl).
Falls back to OCR_LANG.
ASR_MODEL
Whisper model size (default large-v3).
ASR_MODEL_DIR
Local faster-whisper model directory (default third_party/mod‐
els/whisper-large-v3 after fetch-whisper.sh).
ASR_TIMESTAMPS
Set to 1 to prefix transcript lines with [mm:ss].
ASR_SIDECAR_SERVE
Set to 0 for oneshot ASR (default: warm JSONL daemon).
ASR_SIDECAR_TIMEOUT
ASR sidecar timeout in seconds (default 600).
MARKDOWNER_ASR_SIDECAR
Path to asr_sidecar script.
OCR_ARCHIVE
Collection material flag (same as --archive; see ARCHIVE). Sets
OCR_SIDECAR_PIPELINE, OCR_HTR, OCR_HTR_ENGINE and OCR_PREPROCESS
to match RapidOCR, Tesseract, Qwen2.5-VL, ARletta/kraken or
TrOCR when not auto.
OCR_EXCLUDE
Skip engines (same as --exclude; see EXCLUDE).
OCR_ENGINE
auto (dynamic: print ensemble and/or handwriting HTR per page;
final pick by text-quality score, not engine name), paddle, or
tesseract.
OCR_QUALITY
Set to fast to skip the Tesseract+sidecar ensemble (single print
path; HTR may still run).
MARKDOWNER_OCR_SIDECAR
Path to tools/ocr_sidecar or ocr_sidecar.py.
OCR_SIDECAR_TIMEOUT
Sidecar wall-clock timeout in seconds (default 1800 when HTR is
enabled; dual historic+modern HTR on CPU is slow; override up to
7200).
OCR_SIDECAR_SERVE
Set to 0 or oneshot to spawn a fresh sidecar per image (default:
keep a warm --serve JSONL daemon).
OCR_SIDECAR_PIPELINE
auto (default: classify print vs handwriting, pick engines),
rapidocr, vlm (Qwen2.5-VL page transcription), htr / arletta /
handwriting / trocr (force HTR), paddleocr, or structurev3.
OCR_VLM
auto (default: Ollama vision if a Qwen2.5-VL tag is loaded, else
Transformers/ROCm), ollama, transformers, or 0 to disable
(TrOCR/ARletta remain as HTR fallback).
OCR_VLM_MODEL
Ollama tag (default qwen2.5vl:7b) or a Transformers directory /
Hugging Face id.
OCR_VLM_MAX_EDGE
Long-edge pixels for the VLM image (default 1536; print OCR
still uses OCR_PDF_DPI).
OCR_HTR
always (default: compare print vs HTR when HTR can help — skip
TrOCR on dense print/newspapers; true handwriting still runs
HTR), auto (same dynamic routing), or 0 / off to disable. When
the sidecar already compared HTR (htr_attempted), Markdowner
does not re-run a second HTR pass. A daemon wall-clock timeout
does not start a oneshot clone. For print-only batches use
OCR_HTR=0 (not the default — mixed archives need HTR).
OCR_HTR_ENGINE
auto (default: historic ARletta/kraken first; modern TrOCR only
if historic is weak or the page has notebook rulings. Ruled
notebooks prefer modern and skip historic when modern is already
usable), ensemble / both (always run historic and modern), ar‐
letta / kraken (historic only), or trocr / modern (contemporary
handwriting only).
OCR_HTR_ARLETTA_DIR
Directory with ARletta *.mlmodel files (default third_party/mod‐
els/arletta; fetch with tools/fetch-arletta.sh).
OCR_HTR_MODEL
Override TrOCR directory for modern handwriting. Otherwise
prefers third_party/models/trocr-base-handwritten on CPU (fin‐
ishes) and trocr-large-handwritten on GPU, then the other if
missing (fetch with tools/fetch-trocr.shbase or
tools/fetch-trocr.sh large). OCR_HTR_PREFER_LARGE=1|0 forces
the order. Historic Dutch uses ARletta under OCR_HTR_AR‐
LETTA_DIR.
OCR_HTR_LINE_ENSEMBLE
auto (default on CPU: also try RapidOCR/Tesseract per line when
TrOCR looks weak), always (default on GPU), or 0. OCR_HTR_BEAMS
defaults to 4 on CPU and 8 on GPU (max 8).
OCR_PREPROCESS
Set to handwriting for gentler preprocess on cursive (optional;
auto path usually enough).
OCR_BRIGHTNESS_AUTO
Set to 0 to disable automatic brightness TRC on dark or bright
scans (default: on).
OCR_CRITICAL
Set to 1 for a longer sidecar timeout (3600 s) on difficult
pages.
OCR_SIDECAR_INIT_TIMEOUT
Seconds for isolated StructureV3 before falling through (default
120).
OCR_PDF_DPI / OCR_PDF_BATCH / OCR_PDF_JOBS
PDF render DPI (default 350), pages per render batch (default
4), and parallel page OCR workers.
MARKDOWNER_PLUGINS
Directory of convert *.json plugins (default plugins/ next to
the binary / tree root).
OMP_NUM_THREADS
Prefer 1 when vault -j > 1.
TESSDATA_PREFIX
Directory that contains *.traineddata (the tessdata directory).
Used when system or bundled paths are insufficient.
EXIT STATUS
0 Success, or all vault inputs skipped as unchanged.
1 Usage error or missing path.
2 Conversion failure, no ask search hits, or ask API failure. In
vault mode, other files may still have been processed.
EXAMPLES
markdowner report.pdf
markdowner sample.eml
markdowner sample.ics
markdowner -L eng scan.tiff
markdowner -o ./out-mail testdata/sample.mbox
markdowner vault -j 2 ./scans
markdowner vault -E ./inbox
markdowner vault ./fixtures/testdata/
markdowner vault -L eng ./archive
markdowner vault --archive handwriting ./letters
markdowner vault --archive handwriting --exclude historic ./notebooks
markdowner --archive print -o newspaper.md scan.jpg
markdowner zettel -v ./vaults/scans
markdowner zettel ./vaults/*
markdowner logseq ./vaults/scans
markdowner logseq ./vaults/*
APPIMAGE_EXTRACT_AND_RUN=1 ./third_party/logseq/logseq
markdowner index -D -j 8 ./vaults/docs
markdowner index --no-rag ./vaults/docs
ASK_RAG_EMBED=hash markdowner index ./vaults/docs
ASK_RAG_CHUNK_CHARS=4000 ASK_RAG_CHUNK_OVERLAP=100 markdowner index ./vaults/docs
ASK_RAG_EMBED_BATCH=64 markdowner index ./vaults/docs
markdowner search -v ./vaults/docs
markdowner search ./vaults/docs common ground
markdowner search ./vaults/* common ground
markdowner ask ./vaults/docs "local question without API"
cp .env.example .env && chmod 600 .env # put MISTRAL_API_KEY=… in .env
markdowner ask ./vaults/docs
markdowner ask -D --dry-run ./vaults/docs "What is the NDD?"
markdowner ask ./vaults/docs "What is the NDD?"
markdowner ask --no-deep ./vaults/docs "short question"
markdowner ask --deep=1 ./vaults/emails "employer timeline"
markdowner ask --deep=2 ./vaults/docs "who is …"
markdowner ask --deep=3 ./vaults/docs "who is …"
markdowner ask --deep=4 ./vaults/docs "who is …"
markdowner ask --deep=5 ./vaults/docs "who is …"
markdowner ask ./vaults/* "What is the NDD?"
OCR_SIDECAR_PIPELINE=rapidocr markdowner -o /tmp/o.md scan.jpg
markdowner -o /tmp/any.md sample-scan.jpg
./tools/fetch-arletta.sh
./tools/fetch-whisper.sh
./tools/fetch-embed.sh
markdowner -L eng -o interview.md interview.mp3
ASR_TIMESTAMPS=1 markdowner -o ts.md clip.wav
markdowner -D -o /tmp/o.md sample-scan.jpg
MARKDOWNER_LOG=verbose MARKDOWNER_LOG_FILE=/tmp/md.log markdowner vault -j 2 ./scans
markdowner ask --dry-run --retrieve smith ./vaults/testdata "who are the authors"
markdowner models -v status
markdowner models update
markdowner metrics -v ./vaults/docs
markdowner status ./vaults/docs
markdowner status ./vaults/*
markdowner webui
markdowner webui --no-browser -p 9000
OCR_CRITICAL=1 markdowner -o /tmp/hard.md hard-scan.tif
Without an FTS index, each search reads every Markdown file under the
vault. Build the index once (or after bulk ingest) for large corpora.
RAG embedding of BGE-M3 on CPU scales with chunk count (large agenda
PDFs → tens/hundreds of thousands of chunks). Prefer larger
ASK_RAG_CHUNK_CHARS, GPU via ./tools/ensure-torch-gpu.sh (NVIDIA CUDA
or AMD ROCm; torch.cuda.is_available() should become true), a lighter
local model dir, or ASK_RAG_EMBED=hash / index --no-rag when wall-clock
matters more than semantic recall. Local embeds checkpoint/resume via
embeddings.partial.f32 (same chunks fingerprint).
Microbenchmark (indexed vs scan) on the sample vault: make bench-search
DEPENDENCIES
Runtime open-source libraries: tesseract, leptonica (OCR fallback),
libsqlite3 (FTS5), and libcurl (optional cloud ask). JSON for the Mis‐
tral API uses nlohmann/json (header-only at build time). Optional doc‐
ument conversion: pandoc (docx/odt/epub/html/rtf, and LibreOffice HTML
for sheets/slides; GPL subprocess) and soffice / libreoffice
(xlsx/ods/pptx/odp → HTML). Optional local graph viewer: Logseq (AGPL;
see tools/fetch-logseq.sh). Optional OCR sidecar (Python 3.10–3.12
venv): paddleocr/paddlepaddle/paddlex and/or rapidocr-onnxruntime (see
tools/requirements-ocr.txt). Optional handwriting VLM: Qwen2.5-VL
(tools/fetch-vlm-ocr.sh) via Ollama or Transformers/ROCm. Optional
handwriting HTR: kraken + ARletta models (tools/fetch-arletta.sh,
tools/requirements-ocr-htr.txt) and/or TrOCR (tools/fetch-trocr.sh).
Optional ASR (audio → Markdown): faster-whisper (see tools/require‐
ments-asr.txt, tools/fetch-whisper.sh) and ffmpeg on PATH. Optional
RAG embeds: sentence-transformers + ./tools/fetch-embed.sh (BGE-M3),
plus chonkie, rank_bm25, numpy, usearch, and python-dotenv (pip install
-r tools/requirements-ask.txt into third_party/ocr-venv). Re-index af‐
ter upgrading the RAG stack (manifest version2; adds vectors.usearch).
Optional spell remarks: hunspell with language dictionaries (else vali‐
dation notes spell: unavailable). PDF page render for OCR uses
pdftoppm / pdftocairo (Poppler) when on PATH. See
third_party/README.md in the source tree for packaging without root
apt.
NOTES
Options follow normal Unix/getopt_long(3) rules: short options may be
clustered; long options use --name or --name=value. A double dash --
ends option parsing. With GNU libc, options may appear after operands
unless POSIXLY_CORRECT is set.
SEE ALSO
getopt(3), pandoc(1), tesseract(1), ffmpeg(1), sqlite3(1)
Dependency setup: third_party/README.md.
AUTHORS
Written by Pascal Greuter.
markdowner 2.1 August 2026 MARKDOWNER(1)
