Replacing LayoutPredictor model with SmolDocling

[agodinezmm2007]

Hello,

I have made extensive modifications to these two files:
docling_ibm_models/code_formula_model/code_formula_predictor.py
docling/models/code_formula_model.py

in order to replace the codeformula model with smoldocling and it works well enough for a couple PDF's but if the formula snippet images contain extra text from the cropping not being perfect it will generate nonsense and repeat until reaching max tokens. I figured that instead of feeding it the snippets like the original CodeFormula model, I could modify the LayoutPredictor files so that SmolDocling generates the doctags document for the entire PDF page which. However, i'm not sure if I can simply change the repo folders to use SmolDocling since it seems like the layout model uses a mapping schema not found within the SmolDocling configuration files, though. Any guidance on replacing/integrating SmolDocling into this part of the pipeline?

Also wanted to add: When using the original CodeFormula model within the docling pdf pipeline VRAM usage would swell to 20-30 GB, with smoldocling doing the code formula predictions I havent seen it go past 10,000 MB once

[agodinezmm2007]

nevermind. A problem i have had for weeks is that any stray or poorly cropped text or character that ends up in the formula cluster snippets will cause the model to produce garbage till hitting max tokens or hitting any other generation configuration i could think of to resolve this problem. no matter how insignificant the stray characters were, even like half a dot or something it just completely throws SmolDocling into loops. but yeah getting smoldocling to replaec the codeformula predictor was very difficult, but it had to be done because the code formula predictor model used up like 30-40GB of VRAM, which is not efficient.

I tried for weeks adjusting the expansion factors, bounding box parameters padding in docling/models/code_formula_model.py which was pointless because if i got the images cropped perfecltly to exclude all stray characters, suddenly the padding is now too tight to the top left and the model now can see the formula so it enters another repetitive loop.

I modified docling/models/layout_model.py and docling/datamodel/base_models.py to include logic which turns every cluster rectangle/bounding box object that is not DocItemLabel.FORMULA into a big white rectangle with padding every so sligghtly bigger than the text item bounding box boundaries. The result was page images which have nothing but the formulas on them, so when they got to the the formula prediction all the model had to analyze was the formula, no stray characters. Ran it through 11 academic journal articles, some with over ten formulas, and the model correctly identified maybe 95% of formulas. the SmolDocling model them perfectly formatted into LaTex/markdown, zero stutters or repetitive behavior, the only ones that it "got wrong" was the ones that were not detected by the layout predictor, so accuracy probably like 99% as long as you have a near perfect formula snippet, otherwise good luck getting it to work.