[dynamics] cp.array() fails and cudaq.evolve fails with cuDensityMatError

[calumholker]

[dynamics] cp.array() fails and cudaq.evolve fails with cuDensityMatError

I am attempting to run the example code in the 'Quick Start' section of the dynamics documentation (for cudaq 0.11.0).

The example is:

import cudaq
import cupy as cp
from cudaq.dynamics import Schedule

# Set the target to our dynamics simulator
cudaq.set_target("dynamics")

# Dimensions of sub-systems: a single two-level system.
dimensions = {0: 2}

# Initial state of the system (ground state).
rho0 = cudaq.State.from_data(
    cp.array([[1.0, 0.0], [0.0, 0.0]], dtype=cp.complex128))

# Schedule of time steps.
steps = np.linspace(0, t_final, n_steps)
schedule = Schedule(steps, ["t"])

# Run the simulation.
evolution_result = cudaq.evolve(hamiltonian,
                                dimensions,
                                schedule,
                                rho0,
                                observables=[spin.x(0),
                                             spin.y(0),
                                             spin.z(0)],
                                collapse_operators=[],
                                store_intermediate_results=True)

When I run this I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[4], [line 7](vscode-notebook-cell:?execution_count=4&line=7)
      4 dimensions = {0: 2}
      6 # Initial state of the system (ground state).
----> [7](vscode-notebook-cell:?execution_count=4&line=7) rho0 = cudaq.State.from_data(
      8     cp.array([[1.0, 0.0], [0.0, 0.0]], dtype=cp.complex128))
     10 # rho0 = cudaq.State.from_data(
     11 #     np.array([[1.0, 0.0], [0.0, 0.0]], dtype=np.complex128))

File cupy/_core/core.pyx:411, in cupy._core.core._ndarray_base.__getbuffer__()

TypeError: Accessing a CuPy ndarry on CPU is not allowed except when using system memory (on HMM or ATS enabled systems, need to set CUPY_ENABLE_UMP=1) or managed memory

I then set CUPY_ENABLE_UMP=1, but when I do this cp.array fails. This minimal example produces the following error:

import cupy as cp
a = cp.array([1, 2, 3], dtype=cp.float64)

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[2], [line 1](vscode-notebook-cell:?execution_count=2&line=1)
----> [1](vscode-notebook-cell:?execution_count=2&line=1) a = cp.array([1, 2, 3], dtype=cp.float64)

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cupy/_creation/from_data.py:53, in array(obj, dtype, copy, order, subok, ndmin, blocking)
      7 def array(obj, dtype=None, copy=True, order='K', subok=False, ndmin=0, *,
      8           blocking=False):
      9     """Creates an array on the current device.
     10 
     11     This function currently does not support the ``subok`` option.
   (...)     51 
     52     """
---> [53](https://vscode-remote+ssh-002dremote-002bkatana.vscode-resource.vscode-cdn.net/home/z5674748/code/spinphony/examples/notebooks/cudq_tests/~/code/spinphony/.venv/lib64/python3.11/site-packages/cupy/_creation/from_data.py:53)     return _core.array(obj, dtype, copy, order, subok, ndmin, blocking)

File cupy/_core/core.pyx:2509, in cupy._core.core.array()

File cupy/_core/core.pyx:2536, in cupy._core.core.array()

File cupy/_core/core.pyx:2743, in cupy._core.core._array_default()

AssertionError:

How can I fix this?

If I then change the original code snippet to use numpy instead of cupy in rho0 = cudaq.State.from_data(np.array([[1.0, 0.0], [0.0, 0.0]], dtype=np.complex128)) I get the following error from cudaq.evolve:

#ERROR: Exception: Invalid argument: #ERROR(cudensitymatCreateElementaryOperator): Argument elemOperator is NULL!
---------------------------------------------------------------------------
cuDensityMatError                         Traceback (most recent call last)
Cell In[8], [line 6](vscode-notebook-cell:?execution_count=8&line=6)
      3 schedule = Schedule(steps, ["t"])
      5 # Run the simulation.
----> [6](vscode-notebook-cell:?execution_count=8&line=6) evolution_result = cudaq.evolve(hamiltonian,
      7                                 dimensions,
      8                                 schedule,
      9                                 rho0,
     10                                 observables=[spin.x(0), spin.y(0), spin.z(0)],
     11                                 collapse_operators=[],
     12                                 store_intermediate_results=True)

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cudaq/dynamics/evolution.py:415, in evolve(hamiltonian, dimensions, schedule, initial_state, collapse_operators, observables, store_intermediate_results, integrator, shots_count)
    408     return [
    409         evolve_single(hamiltonian, dimensions, schedule, state,
    410                       collapse_operators, observables,
    411                       store_intermediate_results, integrator, shots_count)
    412         for state in initial_state
    413     ]
    414 else:
--> [415](https://vscode-remote+ssh-002dremote-002bkatana.vscode-resource.vscode-cdn.net/home/z5674748/code/spinphony/examples/notebooks/cudq_tests/~/code/spinphony/.venv/lib64/python3.11/site-packages/cudaq/dynamics/evolution.py:415)     return evolve_single(hamiltonian, dimensions, schedule, initial_state,
    416                          collapse_operators, observables,
    417                          store_intermediate_results, integrator,
    418                          shots_count)

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cudaq/dynamics/evolution.py:219, in evolve_single(hamiltonian, dimensions, schedule, initial_state, collapse_operators, observables, store_intermediate_results, integrator, shots_count)
    215     except:
    216         raise ImportError(
    217             "Failed to load dynamics solver. Please check your installation"
    218         )
--> [219](https://vscode-remote+ssh-002dremote-002bkatana.vscode-resource.vscode-cdn.net/home/z5674748/code/spinphony/examples/notebooks/cudq_tests/~/code/spinphony/.venv/lib64/python3.11/site-packages/cudaq/dynamics/evolution.py:219)     return evolve_dynamics(hamiltonian, dimensions, schedule, initial_state,
    220                            collapse_operators, observables,
    221                            store_intermediate_results, integrator)
    223 if target_name in analog_targets:
    224     ## TODO: Convert result from `sample_result` to `evolve_result`
    225     return _launch_analog_hamiltonian_kernel(target_name, hamiltonian,
    226                                              schedule, shots_count)

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cudaq/dynamics/cudm_solver.py:123, in evolve_dynamics(hamiltonian, dimensions, schedule, initial_state, collapse_operators, observables, store_intermediate_results, integrator)
    121 _, state = integrator.get_state()
    122 with ScopeTimer("evolve.prepare_expectation") as timer:
--> [123](https://vscode-remote+ssh-002dremote-002bkatana.vscode-resource.vscode-cdn.net/home/z5674748/code/spinphony/examples/notebooks/cudq_tests/~/code/spinphony/.venv/lib64/python3.11/site-packages/cudaq/dynamics/cudm_solver.py:123)     obs.prepare_expectation(cudm_ctx, state)
    124 with ScopeTimer("evolve.compute_expectation") as timer:
    125     exp_val = obs.compute_expectation(schedule.current_step, (),
    126                                       state)

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/operators.py:976, in Operator.prepare_expectation(self, ctx, state, compute_type)
    964 """
    965 Prepare the computation of an expectation value of this :class:`Operator` on a state.
    966 
   (...)    973     The ``compute_type`` argument is currently not used and will default to the data type.
    974 """
    975 if not self._valid_state:
--> [976](https://vscode-remote+ssh-002dremote-002bkatana.vscode-resource.vscode-cdn.net/home/z5674748/code/spinphony/examples/notebooks/cudq_tests/~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/operators.py:976)     self._maybe_instantiate(ctx)
    977 else:
    978     if self._ctx != ctx:

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/operators.py:836, in Operator._maybe_instantiate(self, ctx)
    833 utils.register_with(self, self._ctx, self._ctx.logger)
    835 for term, coeff, dual in zip(self.terms, self._coefficients, self.dualities):
--> [836](https://vscode-remote+ssh-002dremote-002bkatana.vscode-resource.vscode-cdn.net/home/z5674748/code/spinphony/examples/notebooks/cudq_tests/~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/operators.py:836)     self._append_internal(term, coeff, dual)

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/operators.py:764, in Operator._append_internal(self, term, coeff, dual)
    762     raise TypeError("Can only append instances of OperatorTerm to Operator.")
    763 # side effect on entries of self.terms
--> [764](https://vscode-remote+ssh-002dremote-002bkatana.vscode-resource.vscode-cdn.net/home/z5674748/code/spinphony/examples/notebooks/cudq_tests/~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/operators.py:764) term._maybe_instantiate(self._ctx, self.hilbert_space_dims)
    765 utils.register_with(self, term, self._ctx.logger)
    766 self._using_terms.add(term)

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/operators.py:406, in OperatorTerm._maybe_instantiate(self, ctx, hilbert_space_dims)
    397 for term, term_type, modes, conjugations, duals, coeff in zip(
    398     self.terms,
    399     self._term_types,
   (...)    403     self._coefficients,
    404 ):
    405     if term_type == ElementaryOperator:
--> [406](https://vscode-remote+ssh-002dremote-002bkatana.vscode-resource.vscode-cdn.net/home/z5674748/code/spinphony/examples/notebooks/cudq_tests/~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/operators.py:406)         self._append_elementary_product(term, modes, duals, coeff)
    407     elif term_type == MatrixOperator:
    408         self._append_matrix_product(term, conjugations, duals, coeff, hilbert_space_dims)

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/operators.py:332, in OperatorTerm._append_elementary_product(self, elem_ops, modes, duals, coeff)
    330 with cutn_utils.device_ctx(self._ctx.device_id):
    331     for elem_op, _modes, _duals in zip(elem_ops, modes, duals):
--> [332](https://vscode-remote+ssh-002dremote-002bkatana.vscode-resource.vscode-cdn.net/home/z5674748/code/spinphony/examples/notebooks/cudq_tests/~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/operators.py:332)         elem_op._maybe_instantiate(self._ctx)
    333         _batch_size = check_and_get_batchsize(elem_op.batch_size, batch_size)
    334         _ = check_and_get_batchsize(self._batch_size, _batch_size)

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/elementary_operator.py:207, in ElementaryOperator._maybe_instantiate(self, ctx)
    205     except AssertionError as e:
    206         raise RuntimeError("Device id of input array does not match device id of library context.") from e
--> [207](https://vscode-remote+ssh-002dremote-002bkatana.vscode-resource.vscode-cdn.net/home/z5674748/code/spinphony/examples/notebooks/cudq_tests/~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/elementary_operator.py:207) self._instantiate()
    208 register_with(self, self._ctx, self._ctx.logger)

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/elementary_operator.py:235, in ElementaryOperator._instantiate(self)
    221     self._ptr = cudm.create_elementary_operator_batch(
    222         self._ctx._handle._validated_ptr,
    223         self.num_modes,
   (...)    232         (self.callback._get_internal_gradient_wrapper("tensor") if (self.callback is not None and self.callback.has_gradient) else None),
    233     )
    234 elif self.batch_size == 1:
--> [235](https://vscode-remote+ssh-002dremote-002bkatana.vscode-resource.vscode-cdn.net/home/z5674748/code/spinphony/examples/notebooks/cudq_tests/~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/densitymat/elementary_operator.py:235)     self._ptr = cudm.create_elementary_operator(
    236         self._ctx._handle._validated_ptr,
    237         self.num_modes,
    238         self.mode_dims,
    239         sparsity,
    240         num_diagonals,
    241         diagonal_offsets,
    242         self._dtype,
    243         self._data.data_ptr,
    244         (self.callback._get_internal_wrapper("tensor") if self.callback is not None else None),
    245         (self.callback._get_internal_gradient_wrapper("tensor") if (self.callback is not None and self.callback.has_gradient) else None),
    246     )
    247 else:
    248     raise RuntimeError(f"Unsupported batch size {self.batch_size} of ElementaryOperator.")

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/bindings/cudensitymat.pyx:455, in cuquantum.bindings.cudensitymat.create_elementary_operator()

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/bindings/cudensitymat.pyx:492, in cuquantum.bindings.cudensitymat.create_elementary_operator()

File ~/code/spinphony/.venv/lib64/python3.11/site-packages/cuquantum/bindings/cudensitymat.pyx:120, in cuquantum.bindings.cudensitymat.check_status()

cuDensityMatError: INVALID_VALUE (7):

Are these issues related? I presume they are. Is there a simple fix?

My system and versions are the following:

=== OS & Kernel ===
Linux-4.18.0-553.51.1.el8_10.x86_64-x86_64-with-glibc2.28

=== Python ===
/home/z5674748/code/test-cudaq/.env/bin/python
3.11.11 (main, Dec 12 2024, 16:28:23) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)]

=== ENV VARS ===
CUPY_ENABLE_UMP = 1

=== GPU & CuPy ===
CuPy version : 13.5.1
CUDA runtime version : 12090
CUDA driver version : 12090
Device count : 4
Device 0: NVIDIA L40S
Total memory (bytes): 47929425920
UnifiedAddressing : True
ManagedMemory : True
Device 1: NVIDIA L40S
Total memory (bytes): 47929425920
UnifiedAddressing : True
ManagedMemory : True
Device 2: NVIDIA L40S
Total memory (bytes): 47929425920
UnifiedAddressing : True
ManagedMemory : True
Device 3: NVIDIA L40S
Total memory (bytes): 47929425920
UnifiedAddressing : True
ManagedMemory : True

=== nvidia-smi ===
Fri Aug 1 14:36:00 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.51.03 Driver Version: 575.51.03 CUDA Version: 12.9 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA L40S On | 00000000:49:00.0 Off | 0 |
| N/A 41C P0 140W / 350W | 16833MiB / 46068MiB | 25% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA L40S On | 00000000:61:00.0 Off | 0 |
| N/A 57C P0 319W / 350W | 43905MiB / 46068MiB | 100% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA L40S On | 00000000:C9:00.0 Off | 0 |
| N/A 25C P8 31W / 350W | 3MiB / 46068MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA L40S On | 00000000:E1:00.0 Off | 0 |
| N/A 48C P0 91W / 350W | 43861MiB / 46068MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+

=== Package Versions ===
cudaq version: CUDA-Q Version 0.11.0 (https://github.com/NVIDIA/cuda-quantum)
cupy version: 13.5.1
numpy version: 2.3.2
pip version: pip 25.2 from /home/z5674748/code/test-cudaq/.env/lib64/python3.11/site-packages/pip (python 3.11)

=== pip freeze (filtered) ===
cudaq==0.11.0
cupy-cuda12x==13.5.1
numpy==2.3.2

[1tnguyen]

Hi @calumholker,

CUDA-Q is not compatible with the latest cupy version (v13.5+). In the upcoming CUDA-Q release, we constrain the cupy version in the dependency list.

For CUDA-Q v0.11, we would need to manually downgrade the cupy version if it's 13.5+, e.g., with pip install cupy-cuda12x==13.4.1

[calumholker]

Ah I see, thank you!