exp run --run-all: checks on templated input data can't pass since folder is not copied in tmp/exps

[Gwenn-LR]

Bug Report

Description

Hi, first of all, thanks to all your team for the amazing tool you've provided to the community !

While working on multiple versions of one dataset, I've tried to get metrics for both versions and I've stumbled upon an issue specific to my setup : when I execute dvc repro on each value of my templated dataset_dirpath, all my checks pass since it execution space is my workspace ; however, when I use dvc exp run --queue, since my workspace is copied in tmp/exps WITHOUT the gitignored files (including data tracked with dvc), the pipeline breaks.

Reproduce

I've created a minimal repository to reproduce the issue :

git clone https://github.com/Gwenn-LR/dvc_exp_run_with_templated_input_dataset.git
cd dvc_exp_run_with_templated_input_dataset

dvc init

mkdir data/dataset_1 data/dataset_2
dvc get https://github.com/iterative/dataset-registry get-started/data.xml -o data/dataset_1/data.xml
cp data/dataset_1 data/dataset_2

dvc add data/dataset_1 data/dataset_2

dvc repro pipelines/default/dvc.yaml
# Works.

dvc exp run --queue -S "pipelines/default/params.yaml:dataset_dirpath=dataset_1, dataset_2" pipelines/default/dvc.yaml
dvc exp run --run-all
# Does not work !

Expected

After what I've found on multiple discussions on this repository, I know that those gitignored files are not copied since it could interfere with the Git status of the associated experiment. However, I think that this behavior should not be ignored and integrated or clearly mentioned in the docs.

I know that the linked repository is not the simplest one, but its structure is meant to reproduce a more complex one which I am working on. So my structure might be improved and I also think that composition with hydra might be a solution, but I would like a quick solution focusing on the specific point of my issue.

It could "only" be an environment variable that might help me introduce a different behavior between an execution with dvc repro and another with dvc exp run --queue or even an answer like "It is not possible as it is". I'll be really glad to discuss all those improvements after a clear answer to this specific issue !

Thank you for your consideration.

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 3.59.1 (pip)
-------------------------
Platform: Python 3.10.16 on Linux-6.11.0-25-generic-x86_64-with-glibc2.39
Subprojects:
        dvc_data = 3.16.10
        dvc_objects = 5.1.0
        dvc_render = 1.0.2
        dvc_task = 0.40.2
        scmrepo = 3.3.11
Supports:
        http (aiohttp = 3.11.18, aiohttp-retry = 2.9.1),
        https (aiohttp = 3.11.18, aiohttp-retry = 2.9.1)
Config:
        Global: /home/leroch/.config/dvc
        System: /etc/xdg/xdg-ubuntu/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/ubuntu--vg-ubuntu--lv
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/mapper/ubuntu--vg-ubuntu--lv
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/30f0eb604189e8e37e4eb38ec5c8d890

Additional Information (if any):

[xingxing1024]

it seems that i met the same question too.

output of dvc doctor

DVC version: 3.59.2 (pip)
-------------------------
Platform: Python 3.13.3 on Linux-6.12.25-amd64-x86_64-with-glibc2.41
Subprojects:
	dvc_data = 3.16.10
	dvc_objects = 5.1.0
	dvc_render = 1.0.2
	dvc_task = 0.40.2
	scmrepo = 3.3.11
Supports:
	http (aiohttp = 3.11.18, aiohttp-retry = 2.9.1),
	https (aiohttp = 3.11.18, aiohttp-retry = 2.9.1),
	s3 (s3fs = 2025.3.2, boto3 = 1.37.3)
Config:
	Global: /home/tiger/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/sdb
Repo: dvc, git

question

when use command dvc exp run, the pipeline runs very well，but when i use dvc exp run --queue and then execute dvc queue start，the experiment failed, error messages are listed below.

Following logs for all queued experiments. Use Ctrl+C to stop following logs (experiment execution will continue).

Reproducing experiment 'first-rins'
Running stage 'raw_data':
> python src/pipeline/data/raw_data.py
Traceback (most recent call last):
  File "/data/sdb/project/tiger-quant/.dvc/tmp/exps/tmp50ko2iz4/src/pipeline/data/raw_data.py", line 7, in <module>
    shutil.copy('./data/原始数据/all_instrument_info.par', 'data/raw_data/all_instrument_info.par')
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/shutil.py", line 428, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/shutil.py", line 260, in copyfile
    with open(src, 'rb') as fsrc:
         ~~~~^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './data/原始数据/all_instrument_info.par'

project structure

├── data
│   ├── 原始数据
│   │   ├── all_instrument_info.par
│   │   ├── bar_1h.par
│   │   ├── daily_bar.par
│   │   └── main_contract.par
│   ├── raw_data
│   │   ├── all_instrument_info.par
│   │   ├── bar_1h.par
│   │   ├── daily_bar.par
│   │   └── main_contract.par
├── dvc.lock
├── dvc.yaml
├── metric.yaml
├── params.yaml
├── poetry.lock
├── pyproject.toml
├── pytest.ini
├── README.md
└── src
    ├── constant
    │   ├── bar_period.py
    │   ├── __init__.py
    │   └── underlying_symbol.py

[xingxing1024]

excuse me, the reason why experiment failed is that i misuse the command dvc exp run

wrong command:
dvc exp run <dag node> --queue

correct command:
dvc exp run --queue it works well