feat: auto-coerce mixed-type columns to string for Arrow conversion #868
+943
−70
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nversion When DataFrames contain columns with mixed types (e.g., bytes/float/string or list/scalar), PyArrow's from_pandas() fails with ArrowTypeError or ArrowInvalid. This change automatically catches these errors and coerces problematic columns to strings, emitting a warning about which columns were coerced. Changes: - Add _coerce_mixed_type_columns() helper in PlotterBase - Wrap pa.Table.from_pandas() in try/except to catch Arrow errors - Add public to_arrow() method for debugging Arrow conversion issues - Add type stub in Plottable.py - Add 9 new tests for mixed-type handling and to_arrow() - Update CHANGELOG.md Fixes #867 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Extends the auto-coercion feature to cuDF DataFrames: - Added _coerce_mixed_type_columns_cudf() helper method - Wrapped cuDF to_arrow() with try/except for ArrowTypeError/ArrowInvalid - Same pattern as pandas: catch exception → coerce to string → retry 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…lidation Introduces validation modes to support ETL pipelines needing fail-fast behavior: - `validate='autofix'` (default): Auto-coerce mixed-type columns + warn - `validate='strict'`: Fail with ArrowConversionError on mixed types - `validate='strict-fast'`: Same as strict, for type/schema issues Changes: - Added ValidationException and ArrowConversionError to exceptions.py - Modified _table_to_arrow() to check _validate_mode and raise in strict modes - Added 8 new tests for validate mode behavior - Total: 25 Arrow conversion tests pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Thread validate_mode through plot() → _plot_dispatch → _make_dataset → _table_to_arrow - Add validate/warn params to plot() and upload() with backward compat - 'autofix' (default): auto-coerce mixed types with warning - 'strict'/'strict-fast': raise ArrowConversionError - True → 'strict', False → 'autofix' with warn=False - Update Plottable.py type stubs - Fix tests to use validate_mode param instead of self mutation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 8 TDD tests for the validation matrix: Arrow validation through plot() entrypoint: - test_plot_strict_clean_data_passes (scenario 1) - test_plot_strict_mixed_data_raises (scenario 3) - test_plot_autofix_warn_true_clean_no_warning (scenario 10) - test_plot_autofix_warn_true_mixed_warns_and_coerces (scenario 12) - test_plot_autofix_warn_false_mixed_coerces_silently (scenario 16) - test_plot_validate_false_coerces_silently (scenario 19) - test_plot_validate_true_raises_on_mixed (scenario 18) Encoding validation: - test_encoding_validation_strict_invalid_raises (scenario 2) - test_encoding_validation_valid_passes Combined scenarios: - test_strict_mixed_data_fails_before_encoding_check (scenario 4) Also fixed encoding validation wiring: always ON for all validate modes except when explicitly disabled (old validate=True default preserved). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- ArrowUploader.create_dataset() now has strict/warn params - When strict=False, encoding errors warn (if warn=True) or silent - ArrowUploader.post() passes through strict/warn params - plot() wires validate mode to uploader strict param - Added comprehensive encoding warn mode tests 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
F811 flake8 error - Literal was imported from both typing and typing_extensions. Keep typing_extensions version for Python version compatibility. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The inline `for col in table.columns` loops were causing mypy to incorrectly narrow the type of `table_fixed.to_arrow()` to Series. Extracting the column-checking logic into separate helper methods isolates the iteration and prevents type narrowing issues. Added: - _find_bad_arrow_columns_pandas() - finds columns failing Arrow in pandas - _find_bad_arrow_columns_cudf() - finds columns failing Arrow in cudf 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Instead of using pd.DataFrame type (which lacks to_arrow()) or Any (which disables type checking), define CudfDataFrameLike and CudfSeriesLike protocols that properly type cudf operations. This fixes mypy errors about Series[Any] not callable when calling to_arrow() on cudf DataFrames. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add copy(), __setitem__, and astype() to protocols so _coerce_mixed_type_columns_cudf can use proper Protocol types instead of Any, fixing mypy errors on Python 3.8/3.10/3.14. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Create local cudf_table: CudfDataFrameLike binding inside the cudf isinstance guard to properly type cudf operations, avoiding mypy errors from Any parameter type. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…_fixed Mypy was joining the types of table_fixed (pd.DataFrame) from the pandas branch with the cudf branch, causing type errors. Use distinct variable name cudf_fixed for the cudf case. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
ArrowTypeErrororArrowInvalidexceptions, automatically coerce them to string typeto_arrow()public method: New helper for debugging DataFrame→Arrow conversion issuesFixes #867
Changes
graphistry/PlotterBase.py:_coerce_mixed_type_columns()helper method_table_to_arrow()to catch Arrow exceptions and auto-coerceto_arrow()public methodgraphistry/Plottable.py: Added type stub forto_arrow()graphistry/tests/test_plotter.py: Added 9 new tests for Issue Add auto_coerce option to handle mixed-type columns during Arrow conversion #867CHANGELOG.md: Documented the new feature and fixTest plan
🤖 Generated with Claude Code