Add `.take_bytes([n])` a zero-copy path from `bytearray` to `bytes`

[cmaloney]

Feature or enhancement

Proposal:

Update bytearray to have its internal buffer always be a PyBytesObject and add a new method, .take_bytes([n]) which extracts that buffer.

.take_byes([n]) adds a way to go from a "mutable" buffer of bytes to an immutable one without requiring a memcpy.

This creates a path to resolve gh-60107

When would I use this?

Any code which makes a ba = bytearray(), modifies it, then calls bytes(ba). Other common patterns are bytes(ba); ba.clear() or bytes(ba[:n]); del ba[:n]. Note that if you want to discard data past a point, the most efficient pattern becomes ba.resize(n); ba.take_bytes() so that take_bytes([n]) doesn’t need to keep around the soon to be discarded extra bytes.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/add-take-bytes-n-to-bytearray-providing-a-zero-copy-path-to-bytes/103804/5
https://discuss.python.org/t/add-zero-copy-conversion-of-bytearray-to-bytes-by-providing-bytes/79164 -- edit: added

Linked PRs

• gh-139871: Add bytearray.take_bytes([n]) to efficiently extract bytes #140128
• gh-139871: Add _PyByteArray_empty_string to the stable ABI #140770

[serhiy-storchaka]

See my arguments against this idea: https://discuss.python.org/t/add-zero-copy-conversion-of-bytearray-to-bytes-by-providing-bytes/79164/15

It will not work because it is not compatible with the C API.

PyByteArray_AS_STRING() cannot return a reference to the bytes object returned by take_bytes(), because it can be mutated. PyByteArray_AS_STRING() cannot allocate a new bytes object, because it never fails, but a memory allocation can fail. The only solution is if take_bytes() allocates a new bytes object, but this kills its purpose.

[cmaloney]

The implementation for this issue is related to that initial discussion thread but iterated on with your feedback incorporated. The final form is in: https://discuss.python.org/t/add-take-bytes-n-to-bytearray-providing-a-zero-copy-path-to-bytes/103804. The initial post there still talks about PyBytesWriter but after further discussion consensus reached using PyBytesObject directly.

In particular with the current design the bytes object inside bytearray is either the empty bytes constant or a uniquely referenced bytes object. That bytes object is only exposed if a user calls .take_bytes([n]) and it returns successfully. There is no __bytes__ function or similar which exposes the internal to bytearray PyObject*.

For the particular points:

1. re: PyByteArray_AS_STRING(): This is equivalent to .clear() and other operations that resize the bytearray. A pointer stashed from PyByteArray_AS_STRING() before a length-mutating operation (ex. .clear()) is not valid after. The implementation of take_bytes([n]) does the same checks as .clear(): Working inside a critical section / lock so only one resizing operation may be in progress at a time and checking ob_exports == 0. There is a free threading test added to help validate correctness.
2. re: Allocation failures: The current implementation changes bytearray to always contain bytes, by default Py_GetConstant(Py_CONSTANT_EMPTY_BYTES);. _PyBytes_Resize's existing implementation handles resizing from empty bytes to larger by allocating a new bytes, resizing to 0 by changing to the empty bytes constant, and resizing in other cases by ensuring the bytes is uniquely referenced before performing any in-place mutation. If a _PyBytes_Resize fails for any reason the bytearray is kept in a good state by re-initializing itself with the empty bytes object. For ba.take_bytes([n]) in particular ba will end in one of these states:
   a. pointing to a new bytes object with all the bytes after n (successfully completed)
   b. It errors before getting to the actual byte taking and is left unmodified (ex. ob_exports != 0 so it can't be resized)
   c. A mid-resizing error occurs and ba is set to the empty bytes constant, an exception is raised, and NULL is returned.

[vstinner]

PyByteArray_AS_STRING() cannot return a reference to the bytes object returned by take_bytes(), because it can be mutated.

I'm not sure that I understand your concern. PyByteArray.ob_bytes_object is private and cannot be accessed (especially in Python). Yes, the ob_bytes_object object is mutated: bytes can be modified and its size can change via bytearray operation (bytearray += bytes, bytearray.resize(), etc.). But I don't see how this is an issue, mutating a bytes object is done by all functions using the soft deprecated PyBytes_FromStringAndSize(NULL, size). PEP 782 implementation uses PyBytesWriter in many cases to avoid that, but PyBytesWriter is implemented internally by ... mutating a bytes object :-)

bytearray.take_bytes() replaces its ob_bytes_object object with a new (empty) one. The returned bytes object should now be treated as immutable.

[vstinner]

Links to previous discussion of this feature:
https://discuss.python.org/t/add-take-bytes-n-to-bytearray-providing-a-zero-copy-path-to-bytes/103804/5

That was in September. There is also the older discussion (February): https://discuss.python.org/t/add-zero-copy-conversion-of-bytearray-to-bytes-by-providing-bytes/79164