A mask is an instruction.
The mask identifies where old text should be removed. It guides inpainting; it is not itself the repaired artwork.
The science behind the story
How ComicFluent
translates a page.
Recognize the words. Translate their meaning. Rebuild the background. Fit the new lettering. Different models and a separate layout engine work together to make a comic readable.
Explore the architectureFollow a page through the system
Two paths.
One translated page.
The image goes through vision and cleanup. Recognized text takes a separate language-model path. Both results come together in your reader.
Original page
Image pixelsDetect text
Vision model
Read text
次の駅へ
OCRText mask
Mask generation
Glossary · characters
Recent dialogue
Recent dialogue
Translate
次の駅へ↓NEXT STOP.
LLM translationInpaint
→
Image reconstructionYour reader
Original+Patch+Aa
Compositing, not a flattened replacement
One page. Three layers.
The original stays underneath. A local cleanup patch covers the old glyphs. Editable translated lettering is drawn on top.
The same three layers, aligned.
Blue shows the pixels selected for rebuilding. The patch is transparent everywhere else.
The patch rebuilds locally.
Inpainting estimates the background beneath the glyphs. Alpha coverage applies that repair only inside the affected regions, leaving the rest of the original untouched.
Lettering stays editable.
Translated strings, region bounds, and text style remain separate from the image. Changing a word or font does not require repainting the source page.
Schematic example. The patterned panels make partial coverage visible; actual inpainting quality depends on the page and the text being removed.
Understand each job
The concepts, explained.
Each stage solves a different problem. Better results depend on how well those stages work together.
01
Detection & OCR
Finding text and reading it are different jobs.
A detector locates candidate text areas. Optical character recognition (OCR) turns the pixels inside them into characters. ComicFluent keeps each region’s ID and page geometry so the recognized words can be traced back to the right place.
A bubble outline is a shape. 「カフェ」 is the text recognized inside it.
- Text detection → candidate regions
- OCR → source characters
- Geometry → coordinates and reading order
02
Context & LLMs
A line makes more sense beside the lines before it.
The language model receives ordered OCR text, source and target languages, and a chapter-context snapshot. Glossary entries, character information, and recent dialogue pairs help carry names and meaning into the next batch. Returned translations remain linked to their page and region IDs.
A short line such as 「彼は？」 can depend on who was mentioned in the previous panel.
- Input → recognized text + context
- LLM → target-language dialogue
- Output → text linked to region IDs
03
Masks & inpainting
The mask says where. The model decides what fits.
A text mask selects pixels for replacement. Inpainting predicts a plausible background using the neighboring image. The result is delivered as a cropped RGBA cleanup patch: color holds the reconstructed content, and alpha controls where it is applied.
Text on plain paper needs a different reconstruction from text across a shaded coat.
- Mask → selected replacement area
- Inpainting → reconstructed pixels
- Alpha → patch coverage
04
Typesetting & composition
The translation still needs a place to live.
The app measures text in the selected font, chooses line breaks, and searches for a size that fits. Irregular text regions need polygon-aware containment as well as width and height checks. Original artwork, cleanup, and editable lettering are then composed on the device.
“Thank you” and “ありがとうございます” take different amounts of space.
- Layout → wrapping, font, and size
- Auto-fit → bounds and shape checks
- Composition → original + patch + lettering
Where the hard parts live
Comics make every stage harder.
The technology helps, but recognition, interpretation, reconstruction, and layout can each be imperfect.
Stylized and tiny lettering
Handwriting, furigana, unusual glyphs, and low-resolution images can be hard to recognize. A misread character can change the translation.
Reading order
Columns, overlapping bubbles, and unconventional panels complicate dialogue sequence. Source reading direction helps the page keep its intended flow.
Ambiguous meaning
Names, pronouns, tone, and omitted subjects often depend on context. A glossary and recent dialogue help, but a model can still choose the wrong interpretation.
Hidden background detail
Inpainting estimates what belongs behind the letters. It cannot recover the exact drawing that the original text concealed.
Longer text and font coverage
A translation can need more space or different glyphs. Wrapping, size, available fonts, and the region’s shape all affect readable lettering.
The technology, in your hands
Bring your next chapter.
Add a supported link, images, or a ZIP / CBZ. Choose your languages and reading direction, then read as pages become ready.
Explore the app
