FileNotFoundError while using own data

[ikmMaierBTUCS]

I cloned the repo and created a jsonfile that I saved in the repo. When I tried to python src/run.py --train_on SGnumlist.json,
it gave me an error:

File "C:\Users\ikm\miniconda3\Lib\site-packages\pandas\io\common.py", line 873, in get_handle
handle = open(
^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'data\SGnumlist.json\train.tsv'

It seems as if it want the json to be in a folder data, but when i move SGnumlist.json to a folder data, it said:
FileNotFoundError: [Errno 2] No such file or directory: 'data\data\SGnumlist.json\train.tsv'

What can I do about it?

[danfriedman0]

The code is expected to work if the argument to --train_on is the path to a JSON file with your own training data (following this section of the README). So if your file is data/SGnumlist.json, I would expect python src/run.py --train_on data/SGnumlist.json to work.

I'm not exactly sure what's going wrong in your case. The relevant code is here:

ShortcutGrammar/src/utils/data_utils.py

Line 197 in 759127a

if name.endswith(".json"):

It looks like for some reason the code isn't recognizing that name (which should be "data/SGnumlist.json") ends in ".json". So my guess is the file name isn't getting parsed correctly from the command line.

Can you share the logs from this run? The program should log to the console and to a file in the output directory (by default "output/scratch/").

[ikmMaierBTUCS]

When I literally follow your instructions, I get:
(base) C:\Users\ikm\ShortcutGrammar>python src/run.py --train_on data/SGnumlist.json
Traceback (most recent call last):
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 11, in
from src.utils import data_utils, logging, metrics, model_utils, tokenizers
ModuleNotFoundError: No module named 'src'

It seems like a deeper problem that I cannot recognize a folder as a module. I had tried python -m src.run --train_on data/SGnumlist.json before which led to the error mentioned before

[danfriedman0]

To deal with the ModuleNotFoundError, try running pip install -e . from the base directory to install the local src package. If that doesn't work, you can try adding the directory to the python path (https://stackoverflow.com/a/57164633). E.g.: set PYTHONPATH=%PYTHONPATH%;C:\Users\ikm\ShortcutGrammar\. If you could share any other logging output from the program, that would help a lot for debugging.

[ikmMaierBTUCS]

I Successfully installed src-0.1.

I confirmed that SGnumlist.json is in ShortcutGrammar/data, but still...

(base) C:\Users\ikm\ShortcutGrammar>python src/run.py --train_on data/SGnumlist.json
I 2025-02-26T00:05:23 main:279: logging to output/scratch
I 2025-02-26T00:05:23 main:280: args: {'output_dir': 'output/scratch', 'seed': 13, 'data_seed': 13, 'model_type': None, 'load_from': None, 'save': False, 'vocab_size': 20000, 'min_frequency': 3, 'load_tokenizer_from': None, 'data_dir': 'data', 'cache_dir': 'cache', 'overwrite_cache': False, 'train_on': 'data/SGnumlist.json', 'validate_on': 'data/SGnumlist.json', 'eval_on': ['data/SGnumlist.json'], 'eval_on_train_datasets': [], 'max_train_examples': None, 'max_dev_examples': None, 'max_length': 512, 'max_eval_length': None, 'min_length': 4, 'use_product_length': 0, 'epochs': 0, 'steps': 0, 'eval_every': 1024, 'patience': 3, 'criterion': 'loss', 'train_batch_size': 4, 'gradient_accumulation_steps': 1, 'eval_batch_size': 16, 'eval_before_training': False, 'optimizer': 'adamw', 'learning_rate': 0.001, 'adam_epsilon': 1e-08, 'weight_decay': 0.01, 'max_grad_norm': 1.0, 'dropout': 0.1, 'preterminals': 64, 'nonterminals': 32, 'state_dim': 256, 'use_grad_eval': 0, 'resume': False, 'remove_special': False}
Traceback (most recent call last):
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 289, in
run(args)
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 253, in run
tokenizer = tokenizers.get_tokenizer(**vars(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\tokenizers.py", line 161, in get_tokenizer
return get_word_tokenizer(
^^^^^^^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\tokenizers.py", line 129, in get_word_tokenizer
word_tokenizer = build_word_tokenizer(
^^^^^^^^^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\tokenizers.py", line 88, in build_word_tokenizer
examples = data_utils.load_tsv(Path(data_dir) / name / "train")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\data_utils.py", line 127, in load_tsv
data = pd.read_csv(fn.with_suffix(".tsv"), delimiter="\t")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ikm\miniconda3\Lib\site-packages\pandas\io\parsers\readers.py", line 1026, in read_csv
return _read(filepath_or_buffer, kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ikm\miniconda3\Lib\site-packages\pandas\io\parsers\readers.py", line 620, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ikm\miniconda3\Lib\site-packages\pandas\io\parsers\readers.py", line 1620, in init
self._engine = self._make_engine(f, self.engine)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ikm\miniconda3\Lib\site-packages\pandas\io\parsers\readers.py", line 1880, in _make_engine
self.handles = get_handle(
^^^^^^^^^^^
File "C:\Users\ikm\miniconda3\Lib\site-packages\pandas\io\common.py", line 873, in get_handle
handle = open(
^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'data\data\SGnumlist.json\train.tsv'

[danfriedman0]

Thanks for posting the log! I've pushed a commit that I think should fix the problem (fb2d41e). Let me know if this works.

[ikmMaierBTUCS]

The initial problem appears to be solved :)
But now I ran into a KeyError: 'text'.
I confirmed that all entries of my json have a key "text".
I got the following log:
(base) C:\Users\ikm\ShortcutGrammar>python src/run.py --train_on SGnumlist.json
I 2025-02-26T14:43:19 main:275: logging to output/scratch
I 2025-02-26T14:43:19 main:276: args: {'output_dir': 'output/scratch', 'seed': 13, 'data_seed': 13, 'model_type': None, 'load_from': None, 'save': False, 'vocab_size': 20000, 'min_frequency': 3, 'load_tokenizer_from': None, 'data_dir': 'data', 'cache_dir': 'cache', 'overwrite_cache': False, 'train_on': 'SGnumlist.json', 'validate_on': 'SGnumlist.json', 'eval_on': ['SGnumlist.json'], 'eval_on_train_datasets': [], 'max_train_examples': None, 'max_dev_examples': None, 'max_length': 512, 'max_eval_length': None, 'min_length': 4, 'use_product_length': 0, 'epochs': 0, 'steps': 0, 'eval_every': 1024, 'patience': 3, 'criterion': 'loss', 'train_batch_size': 4, 'gradient_accumulation_steps': 1, 'eval_batch_size': 16, 'eval_before_training': False, 'optimizer': 'adamw', 'learning_rate': 0.001, 'adam_epsilon': 1e-08, 'weight_decay': 0.01, 'max_grad_norm': 1.0, 'dropout': 0.1, 'preterminals': 64, 'nonterminals': 32, 'state_dim': 256, 'use_grad_eval': 0, 'resume': False, 'remove_special': False}
Traceback (most recent call last):
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 285, in
run(args)
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 249, in run
tokenizer = tokenizers.get_tokenizer(**vars(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\tokenizers.py", line 161, in get_tokenizer
return get_word_tokenizer(
^^^^^^^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\tokenizers.py", line 129, in get_word_tokenizer
word_tokenizer = build_word_tokenizer(
^^^^^^^^^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\tokenizers.py", line 88, in build_word_tokenizer
examples = data_utils.load_examples(name, "train", data_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\data_utils.py", line 185, in load_examples
return load_json(fn)
^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\data_utils.py", line 97, in load_json
if remove_br and "a" not in e and e["text"].startswith("<br"):
~^^^^^^^^
KeyError: 'text'

[danfriedman0]

That's interesting. Are you sure the json files are formatted as a list of objects? Can you share what the json file looks like (maybe just the beginning)? The format should be:

[{"text": "First sentence."}, {"text": "Second sentence."}]

[ikmMaierBTUCS]

No, I used the format [{"a": {"text": "First sentence"}, "b": {"text": "Second sentence"}}] as mentioned in the README. This format does not look equivalent to yours, since it uses an extra {} around everything.

Now I copy+pasted [{"text": "First sentence."}, {"text": "Second sentence."}] into a test.json and tried that:

(base) C:\Users\ikm\ShortcutGrammar>python src/run.py --train_on test.json
I 2025-02-26T15:48:13 main:275: logging to output/scratch
I 2025-02-26T15:48:13 main:276: args: {'output_dir': 'output/scratch', 'seed': 13, 'data_seed': 13, 'model_type': None, 'load_from': None, 'save': False, 'vocab_size': 20000, 'min_frequency': 3, 'load_tokenizer_from': None, 'data_dir': 'data', 'cache_dir': 'cache', 'overwrite_cache': False, 'train_on': 'test.json', 'validate_on': 'test.json', 'eval_on': ['test.json'], 'eval_on_train_datasets': [], 'max_train_examples': None, 'max_dev_examples': None, 'max_length': 512, 'max_eval_length': None, 'min_length': 4, 'use_product_length': 0, 'epochs': 0, 'steps': 0, 'eval_every': 1024, 'patience': 3, 'criterion': 'loss', 'train_batch_size': 4, 'gradient_accumulation_steps': 1, 'eval_batch_size': 16, 'eval_before_training': False, 'optimizer': 'adamw', 'learning_rate': 0.001, 'adam_epsilon': 1e-08, 'weight_decay': 0.01, 'max_grad_norm': 1.0, 'dropout': 0.1, 'preterminals': 64, 'nonterminals': 32, 'state_dim': 256, 'use_grad_eval': 0, 'resume': False, 'remove_special': False}
I 2025-02-26T15:48:13 src.utils.tokenizers:89: building word tokenizer from 2 test.json examples
[00:00:00] Pre-processing sequences ██████████████████████████████████████████████████████████ 0 / 0I 2025-02-26T15:48:13 src.utils.tokenizers:135: caching tokenizer to data\test.json\test.json
Traceback (most recent call last):
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 285, in
run(args)
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 249, in run
tokenizer = tokenizers.get_tokenizer(**vars(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\tokenizers.py", line 161, in get_tokenizer
return get_word_tokenizer(
^^^^^^^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\tokenizers.py", line 136, in get_word_tokenizer
word_tokenizer.save(str(cache_fn))
Exception: Das System kann den angegebenen Pfad nicht finden. (os error 3)

[danfriedman0]

I pushed another commit that I think will address this issue (83b723f).

It seems that in general there are some inconsistencies here and there in the tokenizers file that lead to bugs if you're using your own data. Let me know if any more issues come up. Please also feel free to try to fix any of these bugs yourself and submit a pull request :).

[ikmMaierBTUCS]

I get errors that appear essentially different, depending on whether I input a file with content
[{"a": {"text": "First sentence"}, "b": {"text": "Second sentence"}}] (->KeyError: 'text')
or [{"text": "First sentence."}, {"text": "Second sentence."}] (->NameError: name 'dataset_name' is not defined).
Let us first clarify, which input we want to refer to, before I look into the .py files.

[danfriedman0]

Apologies for the bugs. I've pushed another commit that should address the remaining problems (#eea07c6). I generated a synthetic dataset from a toy CFG (scripts/generate_test_data.py) and tested the following command:

python src/run.py \
  --train_on data/toy_cfg_sentences.json \
  --epochs 5 \
  --min_length 3  \
  --output_dir output/toy_cfg \
  --model_type pcfg \
  --preterminals 6 \
  --nonterminals 6 \
  --overwrite_cache;

This ran successfully.

Regarding the format of the file: If you are training a PCFG on a dataset of individual sentences, the data should be formatted as:

[{"text": "Sentence 1"}, {"text": "Sentence 2"}]

If you are training a synchronous CFG on a dataset of sentence pairs, the data should be formatted as:

[{"a": "Source sentence 1", "b": "Target sentence 1"}, {"a": "Source sentence 2", "b": "Target sentence 2"}]

To my understanding, you are training a PCFG on single-sentence data, so you should use the first format. (Please see the relevant section of the README.)

[ikmMaierBTUCS]

Thank you very much for your help. Unfortunately I am still running in an error, after I double-checked that I had cloned the repo again. Next week I can take some time to look into the bug from myself.
This is what it looks like:
python src/run.py --train_on test.json
I 2025-03-03T17:06:15 main:267: logging to output/scratch
I 2025-03-03T17:06:15 main:268: args: {'output_dir': 'output/scratch', 'seed': 13, 'data_seed': 13, 'model_type': None, 'load_from': None, 'save': False, 'vocab_size': 20000, 'min_frequency': 3, 'load_tokenizer_from': None, 'data_dir': 'data', 'cache_dir': 'cache', 'overwrite_cache': False, 'train_on': 'test.json', 'validate_on': 'test.json', 'eval_on': ['test.json'], 'eval_on_train_datasets': [], 'max_train_examples': None, 'max_dev_examples': None, 'max_length': 512, 'max_eval_length': None, 'min_length': 4, 'use_product_length': 0, 'epochs': 0, 'steps': 0, 'eval_every': 1024, 'patience': 3, 'criterion': 'loss', 'train_batch_size': 4, 'gradient_accumulation_steps': 1, 'eval_batch_size': 16, 'eval_before_training': False, 'optimizer': 'adamw', 'learning_rate': 0.001, 'adam_epsilon': 1e-08, 'weight_decay': 0.01, 'max_grad_norm': 1.0, 'dropout': 0.1, 'preterminals': 64, 'nonterminals': 32, 'state_dim': 256, 'use_grad_eval': 0, 'resume': False, 'remove_special': False}
I 2025-03-03T17:06:15 src.utils.data_utils:61: tokenizing 2 sentences
I 2025-03-03T17:06:15 src.utils.data_utils:215: caching processed examples to cache\test.json_train_testWordTokenizer_v20000_f3.pkl
I 2025-03-03T17:06:15 src.utils.data_utils:61: tokenizing 2 sentences
I 2025-03-03T17:06:15 src.utils.data_utils:215: caching processed examples to cache\test.json_dev_testWordTokenizer_v20000_f3.pkl
I 2025-03-03T17:06:15 src.utils.data_utils:61: tokenizing 2 sentences
I 2025-03-03T17:06:15 src.utils.data_utils:215: caching processed examples to cache\test.json__testWordTokenizer_v20000_f3.pkl
Traceback (most recent call last):
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 277, in
run(args)
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 247, in run
raise NotImplementedError(args.model_type)
NotImplementedError: None

[danfriedman0]

The error here is because you need to pass an argument for --model_type. In your case, I believe this should be --model_type "pcfg".

[ikmMaierBTUCS]

Thank you. Now the code works, when i use the test.json with content

[{"text": "Sentence 1"}, {"text": "Sentence 2"}]

However, when I try to train on a file with content

[{"text": "zero"}, {"text": "one"}, {"text": "two"}, {"text": "three"}, {"text": "four"}, {"text": "five"}, {"text": "six"}, {"text": "seven"}, {"text": "eight"}, {"text": "nine"}, {"text": "ten"}, {"text": "eleven"}, {"text": "twelve"}, {"text": "thirteen"}, {"text": "fourteen"}, {"text": "fifteen"}, {"text": "sixteen"}, {"text": "seventeen"}, {"text": "eighteen"}, {"text": "nineteen"}, {"text": "twenty"}, {"text": "twenty-one"}, {"text": "twenty-two"}, {"text": "twenty-three"}, {"text": "twenty-four"}, {"text": "twenty-five"}, {"text": "twenty-six"}, {"text": "twenty-seven"}, {"text": "twenty-eight"}, {"text": "twenty-nine"}, {"text": "thirty"}, {"text": "thirty-one"}, {"text": "thirty-two"}, {"text": "thirty-three"}, {"text": "thirty-four"}, {"text": "thirty-five"}, {"text": "thirty-six"}, {"text": "thirty-seven"}, {"text": "thirty-eight"}, {"text": "thirty-nine"}]

then i get:
python src/run.py --train_on SGnumlist.json --model_type "pcfg"
I 2025-03-04T12:00:19 main:267: logging to output/scratch
I 2025-03-04T12:00:19 main:268: args: {'output_dir': 'output/scratch', 'seed': 13, 'data_seed': 13, 'model_type': 'pcfg', 'load_from': None, 'save': False, 'vocab_size': 20000, 'min_frequency': 3, 'load_tokenizer_from': None, 'data_dir': 'data', 'cache_dir': 'cache', 'overwrite_cache': False, 'train_on': 'SGnumlist.json', 'validate_on': 'SGnumlist.json', 'eval_on': ['SGnumlist.json'], 'eval_on_train_datasets': [], 'max_train_examples': None, 'max_dev_examples': None, 'max_length': 512, 'max_eval_length': None, 'min_length': 4, 'use_product_length': 0, 'epochs': 0, 'steps': 0, 'eval_every': 1024, 'patience': 3, 'criterion': 'loss', 'train_batch_size': 4, 'gradient_accumulation_steps': 1, 'eval_batch_size': 16, 'eval_before_training': False, 'optimizer': 'adamw', 'learning_rate': 0.001, 'adam_epsilon': 1e-08, 'weight_decay': 0.01, 'max_grad_norm': 1.0, 'dropout': 0.1, 'preterminals': 64, 'nonterminals': 32, 'state_dim': 256, 'use_grad_eval': 1, 'resume': False, 'remove_special': True}
Traceback (most recent call last):
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 277, in
run(args)
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 244, in run
train_dataset, dev_datasets, eval_datasets = get_datasets(args, tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ikm\ShortcutGrammar\src\run.py", line 211, in get_datasets
train_dataset = data_utils.load_dataset(
^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\users\ikm\shortcutgrammar\src\utils\data_utils.py", line 230, in load_dataset
examples, processed = zip(*keep)
^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 0)