[JEWEL-741] Use TextMate as Fallback for Languages Without Parsers

[DanielSouzaBertoldi]

Disclaimer

This PR can only ever be merged when it's safe to bump the IDEA version to 253

Problem

Our syntax highlighting code for Markdown code blocks currently only highlights code that the IDE has a plugin that supports it. So, if you open PyCharm for instance, it'll render python code blocks but not Kotlin. The idea here is to fallback to using TextMate to highlight code in such cases.

Solution

Instead of relying on the MimeType class, we now use CodeFenceLanguageGuesser.guessLanguageForInjection() that finds a plugin that provides a syntax highlighting for a given language. The cool thing about this class is that it knows that kt, kts and kotlin are all related to the Kotlin language. This means we don't need to keep adding mime types to MimeType just to map the info part of a code block to a given language.

If CodeFenceLanguageGuesser is unable to find a corresponding plugin, then we resort to TextMate. The caveat here is that if the code block is declared as:

\```ruby
puts "Hello, Ruby World!"
\```

TextMate won't be able to highlight the code, since the file we provide to it will have the extension .ruby instead of .rb.

Personally, I don't think this is a big deal since usually code blocks use the file extension instead of the programming language name, but it's valid to know about this current limitation. The plus side is that we don't have to map a bunch of programming languages in MimeType just for such cases.

Changes

• Deprecated any and all code that depends on MimeType to highlight code.
  • Didn't deprecate MimeType itself, though I'm 90% sure it's best to do so.
• Created new functions and classes that should be used instead of the alternative that need MimeType.
  • Created a new class FencedCodeBlockInfo that has code and info as parameters instead of code and mimeType as is the case with FencedCodeBlock. The name however is kinda lame.
• For testing purposes, refactored the LexerBasedCodeHighlighter a bit to delegate the actual highlighting code to a new class called CodeAnnotator.
• Updated LexerBasedCodeHighlighter code to mirror the logic presented in the Solution section of this description.
• Added a bunch of code block examples in ComponentShowcaseTab.kt.
• Created unit tests for LexerBasedCodeHighlighter and CodeAnnotator
• While at it, fixed a bug in MimeType.displayName()
  • It was never returning the correct display name for mime types with values like text/kotlin role=gradle since the normalizeString call never returned the base name (in this case, text/kotlin) to fetch the correct display name in the when block.

Evidences

Syntax Highlighting by Both Plugins and TextMate

Release Notes

⚠️ Important Changes

• [Source breaking change!] The mimeType: MimeType? parameter name from FencedCodeBlock class was changed to language: String?, which takes the raw string for the code block language identifier. This won't introduce binary nor behavioural changes since MimeType is a value class that is seen as String? by the compiler.

New features

• Code blocks now can use both semantic highlighting for languages that are available through a plugin, or fallback to token-based highlighting based on the TextMate bundles

Bug fixes

• Fixed a bug where the mime type string for the Go language was actually returning MimeType.YAML
• Fixed a bug where the displayName wasn't returning the correct display name for mime types that also defined a role

Deprecated API

• MimeType value class has been deprecated, along with all of its functions, variables and extensions.
• CodeHighlighter.highlight(code: String, mimeType: MimeType?) has been deprecated. Please use highlight(code: String, language: String).

[rock3r]

usually code blocks use the file extension instead of the programming language name

I always use the language name personally lol... maybe we should still keep a lang name -> extension that, while it won't cover all cases, at least should cover the ones the IDE supports out of the box?

[DanielSouzaBertoldi]

@rock3r I mean, if that's the case then I think we wouldn't even need to call CodeFenceLanguageGuesser then, we could always delegate to TextMate since we will be mapping the info block to a given file extension 🤔

[rock3r]

That wouldn't work, as we would lose semantic highlighting if we only relied on textmate bundles. I'll finish reviewing asap and will add more context :)

[rock3r]

🚢

[rock3r]

We'll punt this to 0.31, since we have to figure out why the test fails.