RuntimeError: dictionary changed size during iteration

[YLivay]

Hello.

I've been getting the following error randomly:

RuntimeError: dictionary changed size during iteration
    at .get_rule ( /app/.venv/lib/python3.12/site-packages/proto/marshal/marshal.py:172 )
    at .to_python ( /app/.venv/lib/python3.12/site-packages/proto/marshal/marshal.py:200 )
    at .__getattr__ ( /app/.venv/lib/python3.12/site-packages/proto/message.py:882 )
    at ._comparator ( /app/.venv/lib/python3.12/site-packages/google/cloud/firestore_v1/base_query.py:1143 )
    at .push ( /app/.venv/lib/python3.12/site-packages/google/cloud/firestore_v1/watch.py:565 )
    at ._on_snapshot_target_change_no_change ( /app/.venv/lib/python3.12/site-packages/google/cloud/firestore_v1/watch.py:387 )
    at .on_snapshot ( /app/.venv/lib/python3.12/site-packages/google/cloud/firestore_v1/watch.py:466 )
    at ._thread_main ( /app/.venv/lib/python3.12/site-packages/google/api_core/bidi.py:667 )

This happens when _instances is being mutated while being iterated on.

proto/marshal/marshal.py:BaseMarshal.get_rule:

class BaseMarshal:

    ....

    def get_rule(self, proto_type):
        # Rules are needed to convert values between proto-plus and pb.
        # Retrieve the rule for the specified proto type.
        # The NoopRule will be used when a rule is not found.
        rule = self._rules.get(proto_type, self._noop)

        # If we don't find a rule, also check under `_instances`
        # in case there is a rule in another package.
        # See https://github.com/googleapis/proto-plus-python/issues/349
        if rule == self._noop and hasattr(self, "_instances"):
            for _, instance in self._instances.items():
                rule = instance._rules.get(proto_type, self._noop)
                if rule != self._noop:
                    break
        return rule

From a quick scan the most likely place _instances is being mutated is when registering a new Marshal instance.

proto/marshal/marshal.py:Marshal.__new__:

class Marshal(BaseMarshal):

    ....

    _instances = {}

    def __new__(cls, *, name: str):
        """Create a marshal instance.

        Args:
            name (str): The name of the marshal. Instantiating multiple
                marshals with the same ``name`` argument will provide the
                same marshal each time.
        """
        klass = cls._instances.get(name)
        if klass is None:
            klass = cls._instances[name] = super().__new__(cls)

        return klass

Environment details

• Programming language: Python
• OS: Linux
• Language runtime version: 3.12
• Package version: 1.26.0

Steps to reproduce

I can't say for sure how to reproduce this, but it seems to happen when a Marshal instance is registered while Marshal.get_rule is busy iterating over self._instances.items() to see which rule can handle a given proto_type.

I'm encountering this randomly when using Firestore (v2.20.1) on_snapshot function to watch for changes on a collection. This on_snapshot is invoked in a separate thread.

To fix

If you're fine with get_rule iterating over potentially slightly stale data, I suggest copying _instances aside in Marshal.new, add the new marshaller to the copy it and set it back to _instances.

If we have to have _instances perfectly in sync, just add a RW lock.