Fix o2m id field handling rebased3

[bosd]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @bosd, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly strengthens the data import capabilities by introducing advanced pre-processing and error recovery mechanisms. It addresses common data quality issues, such as incorrect field types and encoding problems in source CSVs, by automatically correcting them or providing robust fallbacks. The changes aim to make the import process more resilient to imperfect input data, reducing manual intervention and improving the success rate of complex data synchronizations with Odoo.

Highlights

• Robust CSV Parsing and Encoding: The CSV reading logic has been refactored into a new helper function, _parse_csv_data, and the main _read_data_file now includes comprehensive error handling for UnicodeDecodeError, attempting to read files with multiple common encodings (UTF-8, Latin-1, CP1252, ISO-8859-1) before failing.
• Intelligent Field Type Correction: A new preflight validation step, _auto_correct_field_types, has been introduced. This function uses the Polars library to detect and automatically correct common type mismatches, specifically converting float-like strings (e.g., "1.0") into integers for Odoo fields defined as integer types, saving a corrected temporary file for import.
• Enhanced Data Type Conversion for Odoo Imports: A new utility function, _safe_convert_field_value, has been added to intelligently convert field values based on their target Odoo field type. This prevents common type-related errors, such as sending non-integer strings to integer fields, by attempting conversion or leaving the value as-is for Odoo to handle if conversion is ambiguous. It also ensures '/id' suffixed fields remain strings.
• Improved Error Handling and Fallback for Batch Operations: The import process now includes more granular error detection for 'tuple index out of range' and 'does not seem to be an integer' errors during Odoo's load and create operations. When such errors occur in bulk load operations, the system will now gracefully fall back to processing records individually using the create method for the problematic batch, providing more detailed error messages and user warnings.
• Refined Relational ID Resolution: The _resolve_related_ids function has been updated to handle both numeric database IDs and XML IDs (e.g., 'module.identifier') more effectively. It now separates and processes these ID types distinctly, resolving XML IDs via ir.model.data and using numeric IDs directly, improving the accuracy and robustness of relational data linking.
• Optimized Many-to-Many (M2M) and One-to-Many (O2M) Updates: The write_tuple strategy for M2M updates now groups commands by parent record to reduce RPC calls, and run_write_tuple_import normalizes external IDs and explicitly handles field/id column naming for better data consistency. The run_write_o2m_tuple_import function has also been updated to correctly identify and process O2M JSON data based on the actual field name.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces significant improvements to the data import process, focusing on robustness and correcting the handling of relational fields. Key changes include:

• Proactive data type conversion for numeric fields to prevent common Odoo errors.
• A more robust method for reading CSV files with fallback encoding support.
• A complete fix for the write_tuple strategy for many-to-many relationships, which was previously using incorrect logic.
• Improved resolution of external IDs, now supporting both database IDs and XML IDs more efficiently.

The changes are well-structured and address important issues. However, I've identified a critical memory regression in the new file reading logic that could impact performance with large files. I've also found a new, potentially useful function that appears to be uncalled. My detailed comments are below.

[bosd]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces significant improvements to the data import functionality, focusing on robustness and correctness. Key changes include a major refactoring of CSV file reading to handle various encodings and fix a bug in skipping header rows. It proactively converts data types before sending them to Odoo, which should prevent many common import errors. The logic for handling many-to-many relationships and resolving external IDs has been rewritten to be more correct, performant, and robust. A new preflight check for auto-correcting data types is also a great addition. My review includes a suggestion to fix a bug in the new error handling logic to prevent double-processing of failed data chunks.

[bosd]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces substantial improvements to the data import process, focusing on robustness and error handling. Key changes include a sophisticated file reading mechanism with encoding fallbacks, proactive data type correction for numeric fields, and a more efficient and correct method for resolving relational IDs. The code is also significantly refactored into smaller, more manageable helper functions, and test coverage has been greatly increased. My review includes a couple of suggestions to address a potential regression in ID sanitization and a minor code redundancy.

[bosd]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and impressive set of improvements to the data import process. The refactoring greatly enhances robustness, correctness, and maintainability. Key highlights include the new pre-flight type correction, safer value conversion to prevent server-side errors, and a complete overhaul of relational ID handling to correctly process both database IDs and various XML ID formats. The addition of comprehensive, focused unit tests is also a major step forward. I've found a critical issue related to batch processing logic and a couple of areas for further improvement, detailed in the comments.

[gemini-code-assist]

This new logic for resolving IDs is a great improvement for handling both database IDs and XML IDs. However, there seems to be a potential regression. The previous implementation handled full XML IDs with module prefixes (e.g., base.res_partner_1) by splitting the string. This new logic treats the entire string as the XML ID name.

If the source data might contain full XML IDs with module prefixes, the lookup in ir.model.data will fail because it only stores the identifier part in the name field. It would be more robust to strip the module prefix if it exists.

Suggested change

	elif isinstance(id_val, str) and len(id_val) > 0:
	# It's a non-empty string that's not purely numeric - treat as XML ID
	xml_ids.append(id_val)
	elif isinstance(id_val, str) and len(id_val) > 0:
	# It's a non-empty string that's not purely numeric - treat as XML ID.
	# Strip module prefix if present, as ir.model.data stores the name without it.
	xml_ids.append(id_val.split(".", 1)[-1])

[gemini-code-assist]

The logic to handle IndexError (specifically 'tuple index out of range') is duplicated here and later in the generic except Exception block at line 1142. While this might be intentional to catch the error from different sources (local vs. RPC), it can be a bit confusing.

Since both blocks trigger the same _handle_fallback_create logic, consider adding a brief comment explaining why two separate handlers are necessary. This would improve clarity for future maintenance.

[bosd]

/gemini review

[gemini-code-assist]

Code Review

This is a substantial and valuable pull request that significantly refactors and hardens the import process. The introduction of helper functions like _parse_csv_data, _safe_convert_field_value, and _handle_fallback_create greatly improves code clarity and maintainability. The more robust error handling, especially for different file encodings and common Odoo RPC errors like 'tuple index out of range', is a major improvement that will make the import tool much more reliable. The addition of a preflight check for data type correction is also a fantastic feature. Finally, the extensive suite of new tests demonstrates a strong commitment to quality. I've left a couple of minor comments on potential issues, but overall this is excellent work.

[bosd]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a major refactoring to improve the robustness and error handling of the data import process. Key improvements include:

• Enhanced CSV Reading: The file reading logic now gracefully handles UnicodeDecodeError by trying a series of fallback encodings.
• Proactive Type Conversion: A new _safe_convert_field_value function and a pre-flight type_correction_check have been added to automatically handle common data type mismatches (e.g., converting "1.0" to an integer), preventing many server-side errors.
• Flexible Relational ID Handling: The logic for resolving relational fields (/id columns) has been significantly improved to support both database IDs (integers) and XML IDs (strings), making it much more versatile.
• Improved Error Handling: Error handling for "tuple index out of range" errors is now centralized and provides more user-friendly feedback. The fallback logic for batch failures has been refactored into a clean helper function, reducing code duplication.
• Code Modularity: The codebase has been broken down into smaller, more focused helper functions, improving readability and maintainability.
• Increased Test Coverage: A substantial number of new tests have been added to cover the new functionality and edge cases.

Overall, this is a very strong pull request that significantly hardens the import engine. I have one suggestion to further improve the robustness of the CSV reading logic for relational imports.

[bosd]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and valuable refactoring of the data import logic. The changes make the import process more robust by improving error handling for data types and encodings, enhancing the flexibility of relational ID resolution, and adding proactive data correction checks. The introduction of helper functions and centralization of logic for error handling and batch fallbacks greatly improves code clarity and maintainability. The addition of extensive tests is also a major plus. I've identified a couple of minor areas for improvement regarding code clarity and logging precision, but overall, this is an excellent set of changes.

[bosd]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces significant improvements to the data import functionality, particularly around error handling, data type conversion, and relational field processing. The refactoring of _read_data_file to handle encoding fallbacks and the introduction of helper functions like _get_model_fields, _safe_convert_field_value, and _handle_fallback_create greatly enhance the robustness and maintainability of the code. The addition of a pre-flight check for type correction is also a valuable feature.

I've identified a few areas for improvement, mainly concerning the simplification of data conversion logic and fixing a bug in ID resolution. Overall, this is a high-quality contribution that addresses important issues in the import process.

[gemini-code-assist]

The logic for converting a string to a float is overly complex and potentially buggy. The check str_value.replace(".", "").replace("-", "").isdigit() can be fooled by strings with multiple dots or misplaced minus signs (e.g., "1.2.3" or "1--2"), which would then cause float() to raise a ValueError. A more direct and Pythonic approach is to just try the conversion and handle the exception.

    elif field_type == "float":
        try:
            # Attempt to convert to float directly. This is more robust.
            return float(str_value)
        except (ValueError, TypeError):
            # If conversion fails, return the original value to let Odoo handle it.
            return field_value

[gemini-code-assist]

This line retrieves a value from the dictionary but does nothing with it. It appears to be a statement with no effect and can be safely removed. The error message from res['messages'] is already handled in other parts of the code (e.g., lines 1051-1060).

[gemini-code-assist]

The logic to separate database IDs from XML IDs does not handle integer values from the external_ids series. If id_list contains integers, they will fall into the else case and be treated as invalid IDs, which is incorrect. The logic should be updated to correctly handle integer database IDs.

    for id_val in id_list:
        if isinstance(id_val, int):
            db_ids.append(id_val)
        elif isinstance(id_val, str):
            if id_val.isdigit():
                db_ids.append(int(id_val))
            elif len(id_val) > 0:
                xml_ids.append(id_val)
            else:  # empty string
                invalid_ids.append(id_val)
        else:
            # Other types like float, or None which was already dropped
            invalid_ids.append(id_val)

[gemini-code-assist]

The logic for integer conversion is more complex than necessary. The elif/else structure to check if a string is a digit before converting is redundant. A try-except block around the conversion is more Pythonic and handles all cases more cleanly and robustly.

        try:
            # Handle float strings like "1.0", "2.0" by converting to int
            if "." in str_value:
                float_val = float(str_value)
                if float_val.is_integer():
                    return int(float_val)
                # It's a non-integer float, return as string for Odoo to handle.
                return str_value

            # For other cases, attempt direct conversion to int.
            return int(str_value)
        except (ValueError, TypeError):
            # If any conversion fails, return the original value.
            return field_value