Why Kapa describes images at index time instead of sending them to LLMs per query

Kapa, which builds RAG assistants over technical documentation, ran the obvious experiment first: retrieve relevant chunks at query time, gather their images, and hand everything to a multimodal model. It does not scale. Raw images added 27% per-query cost on GPT 5.1 and 51% on Claude 4.6 Sonnet, typical questions referenced 20-30 images (long tail past 130) that bumped against Claude’s 30 MB and OpenAI’s 50 MB payload caps, and CLIP-style multimodal retrieval blurred the fine detail in charts and annotated screenshots that users actually need.

The approach that worked moves the vision cost to ingestion. A small vision model writes a caption for each image once, the caption is stored as its own text chunk, and retrieval plus generation run as pure text — the LLM never sees pixels, only cites the original image URL. Per-query overhead drops to 1-6%, and an LLM judge preferred image-aware answers at p<0.05 across three customer projects and two models. Captions handle both illustrative images (a screenshot showing which button to click) and load-bearing ones (a wiring diagram or fire-resistance matrix whose values exist nowhere else in the text).

Production details matter more than model choice. A zero-shot classifier on multimodal embeddings filters logos, avatars, and decorative banners (96.8% accuracy on clear cases, collapsing on genuinely ambiguous images where pixels alone cannot decide). Feeding the captioner the surrounding paragraphs sharply improves grounding, while jumping from GPT 5.4 mini to Claude 4.6 Sonnet barely moves quality — only nano-tier degraded. Storing captions as separate chunks beat inlining them into the document: inline inflated every chunk and shipped on every query, while separate chunks enter context only when the retriever pulls them in.