[reST] refactor

[jrappen]

Note

This PR is a work-in-progress. If you have suggestions for stuff that
should be fixed, leave a comment below.

---

current TODOs

• compare comments in syntax file
• add tests to prove linked issues can be closed

---

Additions:

• correctly hightlight directive options and their values as
  mapping.pair while checking value format
• default keymap
• symbol index now lists explicit and implicit hyperlink
  targets (sections, footnotes, citations)
• highlight embedded python and raw-code directives (css, js, json,
  jsonc, html, python, toml, yaml), compare [reStructuredText] Code-blocks are not handled #3158
• explicit hyperlink targets and anonymous explicit hyperlink
  targets
• correctly highlight grid tables with optional inline markup
• correctly highlight inline interpreted text, optionally
  with roles
• correctly highlight substitution definitions
• completions with strict scope limits
• indentation settings based upon official docs
• neg. tests for comments
• highlight empty comments and section separators between empty
  lines
• add metadata to enable folding

Fixes:

• line blocks (with optional inline markup)
• footnotes and citations
• pop explicit markup blocks before the next explicit markup block
• correctly highlight (valid) comments as block comments
• do not highlight paragraphs starting with ... as comments
• include inline markup within block quotes
• allowed symbols for heading markers, compare [reStructuredText] Headers using some of the supported characters aren't highlighted #2894
• inline internal targets may contain symbol characters, compare
  [RestructuredText] single quotes break link targets #1204
• inline hyperlink references
• section headings:
  • with overline and underline
  • with underline only
  • optional inset when with overline and underline
• literal blocks:
  • meta scope
  • literal blocks with expanded form
  • literal blocks with partially minimized form
  • literal blocks with fully minimized form
  • literal blocks with quoted form
• enumerated lists
• field lists
• punctuation of inline substitution references
• tests for embedded python
• inline literal for single backtick

Changes:

• the syntax file now uses branching to correctly highlight
  section headings and their punctuation
• scope names have been adjusted with regard to the syntax
  scope naming guide
• match punctuation.definition.end rules for inline items
  before invalid.illegal.newline
• pop line blocks on empty line
• pop explicit markup blocks before next explicit markup block
• moved some rules from block-quote-block context to literal-block
• removed inline markup from literal blocks
• updated list of allowed characters for internal link labels,
  compare [reST] Internal link labels should allow more character types #793
• changed auto_complete_selector default
• re-ordered test file to match order of syntax file
• include linebreak in match for first line of directives
• differentiate known (constant.language) and unknown (constant.other)
  directive names

References:

• fixes [reST] Internal link labels should allow more character types #793
• fixes [RestructuredText] single quotes break link targets #1204
• fixes [reStructuredText] Headers using some of the supported characters aren't highlighted #2894
• fixes [reStructuredText] Code-blocks are not handled #3158
• duplicate the work @deathaxe had done in [Common] Add git conflict marker highlighting #4047, so his changes to the
  syntax don't get lost

Thanks to these contributors:

• deathaxe deathaxe@users.noreply.github.com
• Giampaolo Rodola giampaolo@users.noreply.github.com
• Victor Wheeler vwheeler63@users.noreply.github.com

[giampaolo]

Thanks a lot for taking care of this.

• If possible, could you also add syntax highlighting for these kind of links?

  This is a paragraph `Process.exe()`_.
  
  .. _`Process.exe()`: https://psutil.readthedocs.io/en/latest/#psutil.Process.exe

Example file having lots of these: https://raw.githubusercontent.com/giampaolo/psutil/refs/heads/master/HISTORY.rst.

[vwheeler63]

Hello, @jrappen ! These suggestions might be helpful:

• This happens a lot working on other people's writings when they use (or over-use) unordered lists, or even worse, definition lists, that make what SHOULD be a nice, compact 8-item list of things into a pages-long thing that is more difficult to read. Then I convert these to field lists, which, in the RTD theme, makes a nice, compact, very readable list.

  class RstEncapsulateFieldCommand(sublime_plugin.TextCommand):
      """
      If (conditions applied in keymap):
      - we are in a reStructuredText source file, and
      - all selections have some text selected
  
      then wrap the selected text like this:
  
          :selected text:
      """
      def run(self, edit):
          sel_list = self.view.sel()
  
          # Replace each selection with edited string.
          for rgn in reversed(sel_list):
              selected_text = self.view.substr(rgn)
              new_text = ':' + selected_text + ':'
              self.view.replace(edit, rgn, new_text)
  
          # De-select each selection, leaving cursor on right edge of its
          # previous selection.  This avoids the danger of a stray keystroke
          # wiping out the new text.
          for i, rgn in enumerate(sel_list):
              del sel_list[i]
              sel_list.add(sublime.Region(rgn.end()))

• Occasionally I see something like FILE PATHS in a document are inconsistently marked—some as literals, some not—and I go through, say, 100 pages and make them all literals for consistency.

  class RstEncapsulateLiteralCommand(sublime_plugin.TextCommand):
      """
      If (conditions applied in keymap):
      - we are in a reStructuredText source file, and
      - all selections have some text selected
  
      then wrap the selected text like this:
  
          ``selected text``
      """
      def run(self, edit):
          sel_list = self.view.sel()
  
          # Replace each selection with edited string.
          for rgn in reversed(sel_list):
              selected_text = self.view.substr(rgn)
              new_text = '``' + selected_text + '``'
              self.view.replace(edit, rgn, new_text)
  
          # De-select each selection, leaving cursor on right edge of its
          # previous selection.  This avoids the danger of a stray keystroke
          # wiping out the new text.
          for i, rgn in enumerate(sel_list):
              del sel_list[i]
              sel_list.add(sublime.Region(rgn.end()))

• While building a glossary of a large work recently (over 100 pages), I lost count of how many times rst_encapsulate_interpreted_text_role got used to create

  :term:`word_or_phrase_in_glossary`

  And that's just in 1 "document"!. It saved TONS of otherwise-error-prone keystrokes!

  I expect more of these interpreted-text roles to come in the near future. I know rst_encapsulate_interpreted_text_role is long, but I used it specifically so that people new to reST will be learning the correct terminology and so will know what to look up when they are learning reST

  class RstEncapsulateInterpretedTextRoleCommand(sublime_plugin.TextCommand):
      """
      If (conditions applied in keymap):
      - we are in a reStructuredText source file, and
      - all selections have some text selected
  
      then wrap the selected text like this:
  
          :role_name:`selected text`
      """
      def run(self, edit, role_name):
          sel_list = self.view.sel()
  
          # Replace each selection with edited string.
          for rgn in reversed(sel_list):
              selected_text = self.view.substr(rgn)
              new_text = f':{role_name}:`{selected_text}`'
              self.view.replace(edit, rgn, new_text)
  
          # De-select each selection, leaving cursor on right edge of its
          # previous selection.  This avoids the danger of a stray keystroke
          # wiping out the new text.
          for i, rgn in enumerate(sel_list):
              del sel_list[i]
              sel_list.add(sublime.Region(rgn.end()))

Keymaps for suggestions above:

    // --------------------------------------------------------------------
    // reStructuredText Tools
    // --------------------------------------------------------------------
    {
        "keys": ["ctrl+alt+f"],
        "command": "rst_encapsulate_field",
        "context":
        [
            { "key": "selector", "operand": "text.restructuredtext"},
            { "key": "selection_empty", "operand": false, "match_all": true },
        ]
    },
    {
        "keys": ["ctrl+alt+l"],  // Lower-case 'L'
        "command": "rst_encapsulate_literal",
        "context":
        [
            { "key": "selector", "operand": "text.restructuredtext"},
            { "key": "selection_empty", "operand": false, "match_all": true },
        ]
    },
    {
        "keys": ["ctrl+alt+t"],
        "command": "rst_encapsulate_interpreted_text_role",
        "args": {
            "role_name": "term"
        },
        "context":
        [
            { "key": "selector", "operand": "text.restructuredtext"},
            { "key": "selection_empty", "operand": false, "match_all": true },
        ]
    },

[vwheeler63]

@jrappen More syntax bugs:

• I think I found a good one for you. I just encountered this. I don't understand why that whole section is blue and underlined. I'm guessing a run-away regex?

  .. _sometext:
  
  Subsection heading
  ******************
  
  .. note:
    more text

  

  • Compare the docs in this source: https://github.com/lvgl/lvgl/tree/master/docs/
  • Published here: https://docs.lvgl.io/9.3/

• With inline literals within a paragraph that span across a line break:

  Some text ``inline
  literal`` more text.

  

[vwheeler63]

Hi, @jrappen !

I found an asset for you -- a great "test bed" for reST: https://github.com/sphinx-themes/sphinx-themes.org/tree/master/sample-docs

These are the docs that are used over and over again (under each hosted theme) to generate this website:

https://sphinx-themes.org/

Example for the Futo theme: https://sphinx-themes.org/sample-sites/furo/

And these docs in particular should be a GREAT test for the reST package: https://sphinx-themes.org/sample-sites/furo/kitchen-sink/

Kind regards,
Vic