cli_eval: Multiple AsyncSqliteSaver instances operating on a single file

[andrei-rusu]

_runtime.py: async with AsyncSqliteSaver.from_conn_string(...) as memory is executed for every LangGraphRuntime. This creates new aiosqlite connections, each of which being assigned a separate asyncio.Lock. The problem is that each of these connections operates on a single shared file __uipath/state.db. As such, when running with multiple workers, racing conditions can occur at write time (signalled via sqlite3.OperationalError: database is locked).

Possible solutions:

• Create separate DB files for each run item. For example in uipath._cli._runtime._contracts>>UiPathBaseRuntime.state_file_path, we can change the return to something like os.path.join(base_dir, f"eval_run_{id}.db") if is_eval_run is True.
• Create the lock ahead of the eval set loop. We could pass the lock (or the memory instance) from outside the loop, but make it Optional such that we can create a new connection with its own lock for uipath run etc.

[akshaylive]

There's a simple fix here.

This needs to go away, and should be maintained in the runtime state

            async with AsyncSqliteSaver.from_conn_string(
                self.state_file_path
            ) as memory

Initializer:

self.memory = AsyncSqliteSaver(aiosqlite.connect(conn_string))

Cleanup:

self.memory.conn.close()

[DragosBobolea]

I'd implement my own AsyncSqliteSaver wrapper, like a singleton, such that if a saver exists, you re-use it, but when they're released, you can close the connection

the manual cleanup is sketchy

[mjnovice]

Attaching the stack trace

  Error: database is locked
Traceback (most recent call last):
  File "/Users/mayankjha/Code/uipath-langchain-python/samples/gym-sample/.venv/lib/python3.13/site-packages/uipath_langchain/_cli/_runtime/_runtime.py", line 124, in execute
    self.context.output = await graph.ainvoke(
                          ^^^^^^^^^^^^^^^^^^^^
        processed_input, graph_config
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/mayankjha/Code/uipath-langchain-python/samples/gym-sample/.venv/lib/python3.13/site-packages/langgraph/pregel/main.py", line 3171, in ainvoke
    async for chunk in self.astream(
    ...<29 lines>...
            chunks.append(chunk)
  File "/Users/mayankjha/Code/uipath-langchain-python/samples/gym-sample/.venv/lib/python3.13/site-packages/langgraph/pregel/main.py", line 2910, in astream
    async with AsyncPregelLoop(
               ~~~~~~~~~~~~~~~^
        input,
        ^^^^^^
    ...<17 lines>...
        cache_policy=self.cache_policy,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as loop:
    ^
  File "/Users/mayankjha/Code/uipath-langchain-python/samples/gym-sample/.venv/lib/python3.13/site-packages/langgraph/pregel/_loop.py", line 1228, in __aenter__
    saved = await self.checkpointer.aget_tuple(self.checkpoint_config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mayankjha/Code/uipath-langchain-python/samples/gym-sample/.venv/lib/python3.13/site-packages/langgraph/checkpoint/sqlite/aio.py", line 329, in aget_tuple
    await self.setup()
  File "/Users/mayankjha/Code/uipath-langchain-python/samples/gym-sample/.venv/lib/python3.13/site-packages/langgraph/checkpoint/sqlite/aio.py", line 285, in setup
    async with self.conn.executescript(
               ~~~~~~~~~~~~~~~~~~~~~~~^
        """
        ^^^
    ...<22 lines>...
        """
        ^^^
    ):
    ^
  File "/Users/mayankjha/Code/uipath-langchain-python/samples/gym-sample/.venv/lib/python3.13/site-packages/aiosqlite/context.py", line 41, in __aenter__
    self._obj = await self._coro
                ^^^^^^^^^^^^^^^^
  File "/Users/mayankjha/Code/uipath-langchain-python/samples/gym-sample/.venv/lib/python3.13/site-packages/aiosqlite/core.py", line 215, in executescript
    cursor = await self._execute(self._conn.executescript, sql_script)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mayankjha/Code/uipath-langchain-python/samples/gym-sample/.venv/lib/python3.13/site-packages/aiosqlite/core.py", line 122, in _execute
    return await future
           ^^^^^^^^^^^^
  File "/Users/mayankjha/Code/uipath-langchain-python/samples/gym-sample/.venv/lib/python3.13/site-packages/aiosqlite/core.py", line 105, in run
    result = function()
sqlite3.OperationalError: database is locked

[mjnovice]

Correction (I commented earlier without re-compiling - was using the old langchain): Thanks @akshaylive for the suggestion.

I tried again, this time re-compiling langchain. It seems it doesn't work. I was able to repro it #217 using ./export_evals.sh; ./eval.sh calculator calculator 10 after trying it few times.

I would second @andrei-rusu in using separate files/databases.

Thoughts @DragosBobolea @akshaylive @andrei-rusu ?

[cristipufu]

This should fix it: #235