[Data] Ensure Concatenator is deserializable from Ray 2.45 #58319
Conversation
* The Concatenator preprocessor gained new fields between Ray 2.45 and 2.49. * Because Ray Data does not currently support forward-compatible preprocessor serialization, attempting to load serialized data from 2.45 fails in newer versions. * This PR adds support to safely deserialize legacy Concatenator state by assigning defaults for missing fields like flatten. Signed-off-by: cem <cem@anyscale.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request addresses a backward compatibility issue in the Concatenator preprocessor related to deserialization from Ray 2.45. The Concatenator gained a new flatten field between Ray 2.45 and 2.49, and this PR ensures that older serialized data can be safely deserialized in newer versions by assigning a default value to the flatten field if it's missing. The changes include adding type hints to __getstate__ and __setstate__ in preprocessor.py, adding __setstate__ to concatenator.py to handle the missing flatten attribute, and adding a test case to verify backward compatibility in test_concatenator.py.
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
github-actions
bot
added
the
stale
|
This pull request has been automatically closed because there has been no more activity in the 14 days Please feel free to reopen or open a new pull request if you'd still like this to be addressed. Again, you can always ask for help on our discussion forum or Ray's public slack channel. Thanks again for your contribution! |
Description
The Concatenator preprocessor gained new fields between Ray 2.45 and 2.49. Because Ray Data does not currently support forward-compatible preprocessor serialization, attempting to load serialized data from 2.45 fails in newer versions. This PR adds support to safely deserialize legacy Concatenator state by assigning defaults for missing fields like flatten.