BUG: OSError: [Errno 24] Too many open files #769
Description
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
To help us to reproduce this bug, please provide information below:
- Your Python version
- The version of Xorbits you use
- Versions of crucial packages, such as numpy, scipy and pandas
- Full stack of the error.
- Minimized code to reproduce the error.
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.
I met this issue again. Please help to solve.
===================================
I create a dataframe
| N1 | N2 |
|---|---|
| query | ori_txt |
| text len 10-50 | 300-5000 |
running code is as below:
import xorbits.pandas as pd
from xorbits.experimental import dedup
data_lst = [{'query': 'xxx', 'ori_txt': 'xxx'}, {'query': 'xxx', 'ori_txt': 'xxx'}]
df = pd.DataFrame(data_lst])
res = dedup(df, col="query", method="minhash", threshold=threshold,
num_perm=num_perm, min_length=min_length, ngrams=ngrams, seed=seed,
verbose=True)
print('ori len: ', len(data_lst))
print('dedup len: ', len(res['query'].tolist()))data_lst contains 10k+ data.
===================================
I got error as below:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/forkserver.py", line 258, in main
fds = reduction.recvfds(s, MAXFDS_TO_SEND + 1)
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/reduction.py", line 159, in recvfds
raise EOFError
EOFError
Traceback (most recent call last):
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xorbits/_mars/deploy/oscar/session.py", line 1954, in get_default_or_create
session = new_session("127.0.0.1", init_local=True, **kwargs)
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xorbits/_mars/deploy/oscar/session.py", line 1924, in new_session
session = SyncSession.init(
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xorbits/_mars/deploy/oscar/session.py", line 1550, in init
isolated_session = fut.result()
File "/home//miniconda3/envs/train_py310/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/home//miniconda3/envs/train_py310/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xorbits/_mars/deploy/oscar/session.py", line 775, in init
await new_cluster_in_isolation(
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xorbits/_mars/deploy/oscar/local.py", line 101, in new_cluster_in_isolation
await cluster.start()
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xorbits/_mars/deploy/oscar/local.py", line 344, in start
await self._start_worker_pools()
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xorbits/_mars/deploy/oscar/local.py", line 386, in _start_worker_pools
worker_pool = await create_worker_actor_pool(
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xorbits/_mars/deploy/oscar/pool.py", line 310, in create_worker_actor_pool
return await create_actor_pool(
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xoscar/api.py", line 179, in create_actor_pool
return await get_backend(scheme).create_actor_pool(
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xoscar/backends/indigen/backend.py", line 49, in create_actor_pool
return await create_actor_pool(
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xoscar/backends/pool.py", line 1585, in create_actor_pool
pool: MainActorPoolType = await pool_cls.create(
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xoscar/backends/pool.py", line 1282, in create
processes, ext_addresses = await cls.wait_sub_pools_ready(tasks)
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xoscar/backends/indigen/pool.py", line 221, in wait_sub_pools_ready
process, status = await task
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xoscar/backends/indigen/pool.py", line 213, in start_sub_pool
return await create_pool_task
File "/home//miniconda3/envs/train_py310/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xoscar/backends/indigen/pool.py", line 203, in start_pool_in_process
process.start()
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/context.py", line 300, in _Popen
return Popen(process_obj)
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/popen_forkserver.py", line 58, in _launch
f.write(buf.getbuffer())
BrokenPipeError: [Errno 32] Broken pipe
2024-04-12 13:59:09,343 asyncio 671836 ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-10' coro=<MainActorPoolBase.monitor_sub_pools() running at /home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xoscar/backends/pool.py:1458> wait_for=<Future pending cb=[Task.task_wakeup()]>>
2024-04-12 13:59:09,344 asyncio 671836 ERROR Task exception was never retrieved
future: <Task finished name='Task-402' coro=<MainActorPool.start_sub_pool() done, defined at /home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xoscar/backends/indigen/pool.py:180> exception=OSError(24, 'Too many open files')>
Traceback (most recent call last):
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xoscar/backends/indigen/pool.py", line 213, in start_sub_pool
return await create_pool_task
File "/home//miniconda3/envs/train_py310/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home//miniconda3/envs/train_py310/lib/python3.10/site-packages/xoscar/backends/indigen/pool.py", line 203, in start_pool_in_process
process.start()
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/context.py", line 300, in _Popen
return Popen(process_obj)
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/home//miniconda3/envs/train_py310/lib/python3.10/multiprocessing/forkserver.py", line 87, in connect_to_new_process
with socket.socket(socket.AF_UNIX) as client:
File "/home//miniconda3/envs/train_py310/lib/python3.10/socket.py", line 232, in __init__
_socket.socket.__init__(self, family, type, proto, fileno)
OSError: [Errno 24] Too many open files===================================
I did operations as below:
sudo sysctl -w fs.file-max=100000
ulimit -S -n 1048576It still gives out error. Error comes from len(res['query'].tolist()), so how to parse the result.
Thanks for your reply.