[Data] Add fillna and dropna functions

[soffer-anyscale]

Why are these changes needed?

Null handling is a basic and common ETL requirement that other data frameworks have. It is important for feature parity and for ML dataset preprocessing to have common null handling features.

This PR adds ds.fillna and ds.dropna functions, modeled after Pandas and PySpark functionalities.

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[Copilot]

Pull Request Overview

This PR adds two essential null handling functions (fillna and dropna) to Ray Data, bringing it to feature parity with pandas and PySpark for basic ETL operations and ML dataset preprocessing.

Key changes include:

• Implementation of fillna method to replace missing values with scalar or column-specific values
• Implementation of dropna method to remove rows containing missing values with flexible filtering options
• Comprehensive test suites covering edge cases and different data types

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
python/ray/data/dataset.py	Adds public API methods fillna and dropna with comprehensive documentation and examples
python/ray/data/_internal/logical/operators/map_operator.py	Implements logical operators FillNa and DropNa for the execution framework
python/ray/data/_internal/planner/plan_udf_map_op.py	Implements planning functions with PyArrow-based transformations for null handling
python/ray/data/_internal/planner/planner.py	Registers the new logical operators with their planning functions
python/ray/data/tests/test_fillna.py	Comprehensive test suite for fillna functionality
python/ray/data/tests/test_dropna.py	Comprehensive test suite for dropna functionality
python/ray/data/BUILD	Adds build targets for the new test files

Comments suppressed due to low confidence (6)

python/ray/data/tests/test_fillna.py:105

• [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.

    for i, (actual, exp) in enumerate(zip(rows, expected)):

python/ray/data/tests/test_dropna.py:53

• [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.

    for i, (actual, exp) in enumerate(zip(rows, expected)):

python/ray/data/tests/test_fillna.py:192

• [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.

    for i, (actual, exp) in enumerate(zip(rows, expected)):

python/ray/data/tests/test_dropna.py:104

• [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.

    for i, (actual, exp) in enumerate(zip(rows, expected)):

python/ray/data/tests/test_fillna.py:220

• [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.

    for i, (actual, exp) in enumerate(zip(rows, expected)):

python/ray/data/tests/test_dropna.py:208

• [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.

    for i, (actual, exp) in enumerate(zip(rows, expected)):