Capture errors outside examples

[jelaniwoods]

In our efforts to improve RSpec output, we've lacked a way to easily parse errors when they occur outside of examples. Adding an easier way to access that information feels aligned with the goals of this gem.

This PR adds an "errors" key that populates with information (exception class, exception message, path, and line number) about errors that occur outside of examples.

This solution uses Regex to capture the exception details. Since we don't have access to the example I don't think we have native access to the raised errors.

Examples:

(I accidentally excluded "messages" from these results, but it is still always present and contains the same result as ["errors"][0]["message"} during an error outside of examples.)

Syntax Error

{
  "statusCode": 200,
  "body": {
    "statusCode": 200,
    "code_output": "/tmp/main.rb:7:in `require_relative': --> /tmp/code.rb\nsyntax error, unexpected end-of-input\n> 2  pp \"Hello, world!\".\n/tmp/code.rb:2: syntax error, unexpected end-of-input (SyntaxError)\npp \"Hello, world!\".\n                   ^\n\n\tfrom /tmp/main.rb:7:in `<main>'\n",
    "test_output": {
      "errors": [
        {
          "message": "\nWhile loading /tmp/spec.rb a `raise SyntaxError` occurred, RSpec will now quit.\nFailure/Error: load(CODE_PATH)\n\nSyntaxError:\n  --> /tmp/code.rb\n  syntax error, unexpected end-of-input\n  > 2  pp \"Hello, world!\".\n  /tmp/code.rb:2: syntax error, unexpected end-of-input (SyntaxError)\n  pp \"Hello, world!\".\n                     ^\n# /tmp/spec.rb:16:in `load'\n# /tmp/spec.rb:16:in `<top (required)>'\n",
          "path": "  /tmp/code.rb",
          "line_number": "2",
          "exception_class": "SyntaxError",
          "exception_message": "  --> /tmp/code.rb\n  syntax error, unexpected end-of-input\n  > 2  pp \"Hello, world!\".\n  /tmp/code.rb:2: syntax error, unexpected end-of-input (SyntaxError)\n  pp \"Hello, world!\".\n                     ^\n"
        }
      ],
      "examples": [],
      "summary": {
        "duration": 0.000402362,
        "example_count": 0,
        "failure_count": 0,
        "pending_count": 0,
        "errors_outside_of_examples_count": 1
      },
      "summary_line": "0 examples, 0 failures, 1 error occurred outside of examples"
    }
  }
}

NoMethodError

{
  "statusCode": 200,
  "body": {
    "statusCode": 200,
    "code_output": "/tmp/code.rb:2:in `<top (required)>': undefined method `undefined_method' for \"Hello, world!\":String (NoMethodError)\n\npp \"Hello, world!\".undefined_method\n                  ^^^^^^^^^^^^^^^^^\n\tfrom /tmp/main.rb:7:in `require_relative'\n\tfrom /tmp/main.rb:7:in `<main>'\n",
    "test_output": {
      "errors": [
        {
          "message": "\nAn error occurred while loading /tmp/spec.rb.\nFailure/Error: pp \"Hello, world!\".undefined_method\n\nNoMethodError:\n  undefined method `undefined_method' for \"Hello, world!\":String\n# /tmp/code.rb:2:in `<top (required)>'\n# /tmp/spec.rb:16:in `load'\n# /tmp/spec.rb:16:in `<top (required)>'\n",
          "path": " /tmp/code.rb",
          "line_number": "2",
          "exception_class": "NoMethodError",
          "exception_message": "  undefined method `undefined_method' for \"Hello, world!\":String\n"
        }
      ],
      "examples": [],
      "summary": {
        "duration": 6.5066e-05,
        "example_count": 0,
        "failure_count": 0,
        "pending_count": 0,
        "errors_outside_of_examples_count": 1
      },
      "summary_line": "0 examples, 0 failures, 1 error occurred outside of examples"
    }
  }
}

---

Important

Adds error capturing for errors outside of examples in EnrichedJsonFormatter, using regex to extract details, and updates version to 0.8.1.

• Behavior:
  • Adds an "errors" key in EnrichedJsonFormatter to capture errors outside of examples, including exception class, message, path, and line number.
  • Uses regex to detect and extract error details from messages.
• Regex:
  • EXCEPTION_DETECTOR_REGEX to identify error messages.
  • PATH_AND_LINE_NUMBER_REGEX to extract file path and line number.
  • EXCEPTION_CLASS_AND_MESSAGE_REGEX to extract exception class and message.
• Misc:
  • Updates version to 0.8.1 in version.rb.

^{This description was created by for c8586f3. You can customize this summary. It will automatically update as commits are pushed.}

[ellipsis-dev]

Important

Looks good to me! 👍

Reviewed everything up to afc7d42 in 1 minute and 34 seconds. Click for details.

• Reviewed 68 lines of code in 2 files
• Skipped 0 files when reviewing.
• Skipped posting 4 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. lib/rspec/enriched_json/formatters/enriched_json_formatter.rb:10

• Draft comment:
  EXCEPTION_DETECTOR_REGEX is very broad and may match unintended text. Consider refining it to reduce false positives.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% This is an RSpec formatter that processes test output. The regex is specifically used to detect error messages in test output. The patterns it looks for are very specific to Ruby error messages. Looking at how it's used in the message method, it's just a first-pass filter before more detailed parsing happens. False positives would just mean extra processing, not incorrect behavior. The comment might have a point - these terms could appear in normal test output or example descriptions. We don't have enough context about what kind of messages this processes. Given this is an RSpec formatter, these patterns are highly specific to Ruby error messages. Even if there are occasional false positives, they would just trigger extra parsing without causing incorrect behavior. The comment should be deleted. The regex serves its purpose as a basic filter for error messages, and any false positives would not cause actual issues.

2. lib/rspec/enriched_json/formatters/enriched_json_formatter.rb:12

• Draft comment:
  EXCEPTION_CLASS_AND_MESSAGE_REGEX may not reliably capture multi-line exception messages. Consider using multiline modifiers (e.g. /m) or adjusting the anchors to handle varying formats.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% The current regex uses ^ anchors with \n, which means it's already handling multiple lines explicitly. The /m modifier would change how ^ and $ work with newlines, but the current regex is deliberately using line-based anchors. The suggestion might actually make the regex less precise since it would allow matching across lines in unintended ways. I could be wrong about how Ruby handles multiline regexes. Maybe there are edge cases where the current regex fails that the /m modifier would fix. The current regex is carefully constructed to match line-by-line with explicit \n handling. The /m modifier would actually make it behave differently than intended. The comment should be deleted as the current regex implementation appears intentional and the suggested change could potentially break the existing line-based matching behavior.

3. lib/rspec/enriched_json/formatters/enriched_json_formatter.rb:16

• Draft comment:
  Overriding @output_hash in initialize may conflict with the parent class’s structure if it also sets @output_hash. Verify that merging your additions with super's output is safe.
• Reason this comment was not posted:
  Confidence changes required: 33% <= threshold 50% None

4. lib/rspec/enriched_json/formatters/enriched_json_formatter.rb:61

• Draft comment:
  Wrap the assignment in the conditional with parentheses to improve clarity and avoid potential Ruby warnings (e.g., if (match = ...)).
• Reason this comment was not posted:
  Confidence changes required: 33% <= threshold 50% None

Workflow ID: wflow_G7kXzCgwUyrHg3YA

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

[bpurinton]

LGTM

[swaathi]

LGTM.

The CI fails here at the linter.

standard: Use Ruby Standard Style (https://github.com/standardrb/standard)
  lib/rspec/enriched_json/formatters/enriched_json_formatter.rb:12:45: Lint/MixedRegexpCaptureTypes: Do not mix named captures and numbered captures in a Regexp literal.
  lib/rspec/enriched_json/formatters/enriched_json_formatter.rb:61:22: Lint/AssignmentInCondition: Wrap assignment in parentheses if intentional
  lib/rspec/enriched_json/formatters/enriched_json_formatter.rb:66:22: Lint/AssignmentInCondition: Wrap assignment in parentheses if intentional
Error: Process completed with exit code 1.

Specs pass. Though I think new specs should be added to test this behaviour. Would be good to add to demo.rb too.

[swaathi]

Could this regex be made a constant too?

[ellipsis-dev]

Caution

Changes requested ❌

Reviewed c8586f3 in 1 minute and 29 seconds. Click for details.

• Reviewed 40 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 1 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. lib/rspec/enriched_json/formatters/enriched_json_formatter.rb:12

• Draft comment:
  Removing the optional ':in `.*'' segment from PATH_AND_LINE_NUMBER_REGEX might limit matching error messages with extra context. Confirm this change is intended.
• Reason this comment was not posted:
  Confidence changes required: 33% <= threshold 50% None

Workflow ID: wflow_Adg0YMDaLrf3GMzD

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

[ellipsis-dev]

EXCEPTION_CLASS_AND_MESSAGE_REGEX includes an unused nested capture group 'extra'. Remove it to simplify the regex if it's not required.

Suggested change

	EXCEPTION_CLASS_AND_MESSAGE_REGEX = /^(?<exception_class>[A-Z]\w*Error|Exception):$\n(?<exception_message>(?<extra>^\s\s.*\n?)+)/
	EXCEPTION_CLASS_AND_MESSAGE_REGEX = /^(?<exception_class>[A-Z]\w*Error|Exception):$\n(?<exception_message>^\s\s.*\n?)+/

[jelaniwoods]

This group is required to capture multi-line messages in the exception named capture group. The actual result of the extra capture group is not important by itself, but it's presence in the regular expression is. I added a name to it because of Rubocop warnings.