fix: add a pre-scan step to detect the true last non-empty row/column and limit the scan range accordingly

[HuangruiChu]

Fix #2307, Follow the instruction of #2307 (comment).
Add a pre-scan step to detect the true last non-empty row and column (including merged cells), then use those bounds to limit all subsequent scans. This will avoid iterating over millions of empty cells and dramatically improve performance on large, sparse sheets.

Issue resolved by this Pull Request:
Resolves #2307

Checklist:

• Documentation has been updated, if necessary.
• Examples have been added, if necessary.
• Tests have been added, if necessary.

[github-actions]

✅ DCO Check Passed

Thanks @HuangruiChu, all your commits are properly signed off. 🎉

[dosubot]

Related Documentation

Checked 3 published document(s). No updates required.

^{How did I do? Any feedback?}  

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🔴 Require two reviewer for test updates

This rule is failing.

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[HuangruiChu]

Need to fix the "Ruff formatter...........................................................Failed".

[codecov]

Codecov Report

❌ Patch coverage is 4.54545% with 21 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
docling/backend/msexcel_backend.py	4.54%	21 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cau-git]

@HuangruiChu thanks for your contribution. Can you please restore the test_backend_msexcel test unit, which apparently was deleted in this PR?

[ceberam]

Thanks @HuangruiChu for your contribution. Here are some comments:

• Did you test and have any evidence that this PR solves the intended issue #2307 ? I tested the PR against large spreadsheets with empty cells and the processing time was more than double compared to the current implementation.
• The new function _find_true_data_bounds is computationally expensive but it limits the scan range in _find_data_tables and overall it may pay off using it. However, it should also be leveraged in _find_table_bottom and _find_table_right, which still have some unbounded scans. This could explain the increase of processing time in this PR.
• Restore the test_backend_msexcel.py to ensure regression tests on this backend pass, as pointed out by @cau-git .
• Remove some unnecessary comments like # Example integration in MsExcelDocumentBackend._find_data_tables

[HuangruiChu]

Thank you for your reply. Sorry, I mistakenly deleted the content of the "test_backend_msexcel.py". I was planning to add a test for "_find_true_data_bounds" under "test_backend_msexcel.py".

1. "it should also be leveraged in _find_table_bottom and _find_table_right." Thank you for reminding me I am working on updating this.
2. "Remove some unnecessary comments." Sure, I will remove that.
3. "Did you test and have any evidence that this PR solves the intended issue Large spreadsheets - very slow processing #2307 ?" I create a version "test-02.xlsx" where the max_row is close to 1M by copying "test-01.xlsx" and adding a format at A1048496 at datasheet1.

@ceberam The processing time for the whole datasheet1, after I experienced the process 100 times to calculate the real processing time for the corner case.
With find_true_data_boundary Mean: 2.845316 s, Std: 0.364838 s.
Without find_true_data_boundary Mean: 2.635403 s, Std: 0.215835 s.

After I read the whole code of _find_table_bottom , _find_table_right , _find_table_bounds, _find_data_tables, I don't think the "find_true_data_bounds" helps in reducing the actual time the codes spend in finding the table data.

One thing that really going to help is add an Early Stopping of a number like 1000 rows for no data--> Assume the datasheet is empty for the rest. It could be risky for the sheets have tables far away from each other.

[ceberam]

After I read the whole code of _find_table_bottom , _find_table_right , _find_table_bounds, _find_data_tables, I don't think the "find_true_data_bounds" helps in reducing the actual time the codes spend in finding the table data.

Some methods still use sheet.max_row or sheet.max_column instead of the calculated bounds. Let me suggest a small refactoring on those methods. I still believe that having the actual data bounds can reduce the overall time.

One thing that really going to help is add an Early Stopping of a number like 1000 rows for no data--> Assume the datasheet is empty for the rest. It could be risky for the sheets have tables far away from each other.

I would still try to optimize the current code as suggested above, but keeping this heuristic approach as an alternative if that fails.

Would you mind giving me write access to your fork? I would then add a commit with my suggestion.
In the meantime you could please resolve the existing conflict with tests/test_backend_msexcel.py