[bug] ERROR - Benchmark directory not found: benchmarks/DPBench-e2e/eval_dataset/DPBench/layout

[msmygit]

I'm trying to follow this page https://github.com/docling-project/docling-eval/blob/main/docs/DP-Bench_benchmarks.md and hit with this at the Create the evaluation report: step,

$ docling-eval evaluate \
  --modality layout \
  --benchmark DPBench \
  --output-dir ./benchmarks/DPBench-e2e
2025-08-20 13:28:13,211 - ERROR - Benchmark directory not found: benchmarks/DPBench-e2e/eval_dataset/DPBench/layout

I tried to create the below directories manually,

• mkdir -p benchmarks/DPBench-e2e/eval_dataset/DPBench
• mkdir -p benchmarks/DPBench-e2e/eval_dataset/DPBench/layout
  and it fails with below,

(docling-eval) ubuntu@ip-172-31-90-179:~/docling-eval$ mkdir -p benchmarks/DPBench-e2e/eval_dataset/DPBench
(docling-eval) ubuntu@ip-172-31-90-179:~/docling-eval$ mkdir -p benchmarks/DPBench-e2e/eval_dataset/DPBench/layout

(docling-eval) ubuntu@ip-172-31-90-179:~/docling-eval$ docling-eval evaluate   --modality layout   --benchmark DPBench   --output-dir ./benchmarks/DPBench-e2e
2025-08-20 14:09:25,358 - INFO - Loading the split 'test' from: 'benchmarks/DPBench-e2e/eval_dataset/DPBench/layout'
2025-08-20 14:09:25,358 - INFO - #-files: 0
Generating test split: 0 examples [00:00, ? examples/s]
Traceback (most recent call last):
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/datasets/builder.py", line 1887, in _prepare_split_single
    num_examples, num_bytes = writer.finalize()
                              ^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/datasets/arrow_writer.py", line 669, in finalize
    raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
datasets.arrow_writer.SchemaInferenceError: Please pass `features` or at least one example when writing data

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/docling-eval/.venv/bin/docling-eval", line 10, in <module>
    sys.exit(app())
             ^^^^^
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/typer/main.py", line 341, in __call__
    raise e
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/typer/main.py", line 324, in __call__
    return get_command(self)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/typer/core.py", line 757, in main
    return _main(
           ^^^^^^
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/typer/core.py", line 195, in _main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/typer/main.py", line 699, in wrapper
    return callback(**use_params)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/docling-eval/docling_eval/cli/main.py", line 1310, in evaluate_cmd
    evaluate(
  File "/home/ubuntu/docling-eval/docling_eval/cli/main.py", line 577, in evaluate
    evaluation = layout_evaluator(  # type: ignore
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/docling-eval/docling_eval/evaluators/layout_evaluator.py", line 192, in __call__
    ds = load_dataset("parquet", data_files={split: split_files})
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/datasets/load.py", line 2084, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/datasets/builder.py", line 925, in download_and_prepare
    self._download_and_prepare(
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/datasets/builder.py", line 1001, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/datasets/builder.py", line 1742, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/ubuntu/docling-eval/.venv/lib/python3.12/site-packages/datasets/builder.py", line 1898, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset

[nikos-livathinos]

@msmygit thanks for bringing up this issue. We need to update our documentation and check the default paths for input/output. You can pass the parameter: --input-dir ./benchmarks/DPBench-e2e/eval_dataset to overcome this issue.

Here is the end to end example for DP-Bench with the correct parameters:

# Make the ground-truth
docling-eval create-gt --benchmark DPBench --output-dir ./benchmarks/DPBench-gt/

# Make predictions for different modalities
docling-eval create-eval \
  --benchmark DPBench \
  --gt-dir ./benchmarks/DPBench-gt/gt_dataset/ \
  --output-dir ./benchmarks/DPBench-e2e/ \
  --prediction-provider Docling # use full-document predictions from docling

# This is not really needed for the next steps, exists only as an example
docling-eval create-eval \
  --benchmark DPBench \
  --gt-dir ./benchmarks/DPBench-gt/gt_dataset/ \
  --output-dir ./benchmarks/DPBench-tables/ \
  --prediction-provider TableFormer # use tableformer predictions only


# Create the evaluation report
docling-eval evaluate \
  --modality layout \
  --benchmark DPBench \
  --input-dir ./benchmarks/DPBench-e2e/eval_dataset \
  --output-dir ./benchmarks/DPBench-e2e/


# Visualize the report
docling-eval visualize \
  --modality layout \
  --benchmark DPBench \
  --input-dir ./benchmarks/DPBench-e2e/eval_dataset \
  --output-dir ./benchmarks/DPBench-e2e/

[msmygit]

Thank you; I'm unblocked now!