Token optimization

[CsBigDataHub]

Hello @ericdallo ,

I came across this project and I though you might be interested.

https://github.com/8b-is/smart-tree

Maybe we improve few of ECA tools taking inspiration from this project, like for instance

• Smart Filtering & Gitignore Support
• Enhanced eca_grep with Context and line numbers for precise code location, reduced search time

Maybe we optimizing tokens with tool tweak will in-turn improve performance.

Also this - https://www.youtube.com/watch?v=zMM5zqesL1g

[ericdallo]

@CsBigDataHub Interesting project indeed.
I agree there are rooms for improvement in existing tools, I don't think they are bad tho, so we can open a issue for specific improvements in specific tools so we can discuss what can be improved and how, most of the improvements need some testing to check if it's really helping LLM in the end,

[ericdallo]

I heavily improved eca_directory_tree (98k -> 1.6k tokens for eca project) doing:

• removed fancy ascii chars, left only spaces as hierarchy separator. (98 -> 51k)
• considered gitignore on listing files (51k -> 1.6k)

[ericdallo]

I noticed now that eca_directory_tree is now quite similar to repoMap context:

• both return a space hierarchy list of files (but the tool can start from a specifc path + support for max-depth, 10 by default)
• both consider index ignoreFiles config, ignoring gitignored files.

I wonder wherther repoMap context still makes sense nowadays, as LLM can just spawn eca_directory_tree at root and get a similar view (even more precise if needed).

I would love your input here @editor-code-assistant/maintainers

[CsBigDataHub]

Yes repoMap can me deprecated.