Parser enhancements

[Gentle]

Hello, I am very interested in this project, it feels easy and fast to use and I would love to replace hunspell in my codebase since it is a pain to compile for mac or wasm environments :)

I have been exploring the parser by basically installing all hunspell-* packages on debian and then trying to load everything in /usr/share/hunspell

some observations:

• generally, some files start with a BOM, which can be stripped easily dfa77ad
• en_GB
  • the infamous lines zverok used to explain the issue about comments fails to parse

    COMPOUNDRULE 2
    COMPOUNDRULE #*0{
    COMPOUNDRULE #*@}
    

    while there is a test that shows #*0{ can be parsed as flags, the parser errors because everything after the first " #" is considered a comment and filtered out early
    to make this work I followed zverok's suggestion and refactored the comment handling to not strip comments early but instead allow lines to end with arbitrary unparsed data, turning comments into a side-effect since this seems to be how upstream handles it 84b41ea
• da_DK
  • uses "" around words for words that should contain a /
    all occurences: "A/S", "c/o", "I/S", I think this does not warrant adding support for a non-feature one dictionary invented
  • contains one interestingly malformed line
    "FedEx/9 ph:https://denstoredanske.lex.dk/FedEx"Fedkrog/54,9
    I think this was supposed to be

    FedEx/9 ph:https://denstoredanske.lex.dk/FedEx
    Fedkrog/54,9
    

• tr_TR
  • uses FLAG num and 0-based indexing, which breaks the NonZero type.
    can be converted by replacing 0 with u16::MAX or any other unused number

for da_DK and tr_TR I just patched the dictionaries, the only dictionary that is problematic now is hu_HU but those files are actually binary and I think it is better to convert them manually to SET UTF8 FLAG num format so that modern parsers don't need the support for legacy binary files

Do you want those two patches dfa77ad 84b41ea
as one or two PRs? And thank you for the library

[the-mikedavis]

Nice! I have cherry picked both of these over with some tiny style changes (added a comment, inlined that Words type). The parser is a little bit bespoke so the more testing on it the better.

The da_DK case is interesting. I'm pretty sure the proper way of escaping '/' so it's not treated like a separator between word and flags is to put a backslash before it. I wonder if the quoting way works in Hunspell? It does seem like an invented feature.

The tr_TR case is also curious. 0 is valid for Hunspell and Nuspell since they don't use a NonZero, but 0 is used as a null flag that usually triggers warnings or is maybe discarded. I will have to check more thoroughly. I wonder if the 0 flag in tr_TR works as it is intended or if it is just silently discarded.

There isn't a spec or anything for Hunspell so usually the correct way to figure out an edge-case is "go see what Hunspell's doing about it" 🙃. I'll take a deeper look into those two cases if I can find some time for it.

Thanks!