Skip to content

adding option for spacy large tokenizer #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

adding option for spacy large tokenizer #17

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5571a57
adding option for spacy large tokenizer
yenniejun Mar 10, 2022
293c799
Update generate_data.py
yenniejun Mar 16, 2022
2522fd8
Moving spacy_lg elif statement up
yenniejun Mar 16, 2022
b8f6bef
Update req.txt to download spacy_lg model
yenniejun Mar 16, 2022
fe4151b
Merge branch 'master' of https://github.com/yenniejun/ehr-relation-ex…
yenniejun Mar 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions generate_data.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ def parse_arguments():
default=0.1)

parser.add_argument("--tokenizer", type=str,
help="The tokenizer to use. 'scispacy', 'scispacy_plus', 'biobert-base', 'biobert-large', 'default'.",
help="The tokenizer to use. 'scispacy', 'scispacy_plus', 'biobert-base', 'biobert-large', 'spacy_lg', 'default'.",
default="scispacy")

parser.add_argument("--ext", type=str,
Expand Down Expand Up @@ -171,11 +171,17 @@ def main():
args.max_seq_len -= biobert.num_special_tokens_to_add()
tokenizer = biobert.tokenize
is_bert_tokenizer = True

elif args.tokenizer == 'spacy_lg':
import spacy
tokenizer = spacy.load("en_core_web_lg")
is_bert_tokenizer = False


else:
warnings.warn("Tokenizer named " + args.tokenizer + " not found."
"Using default tokenizer instead. Acceptable values"
"include 'scispacy', 'biobert-base', 'biobert-large',"
"include 'scispacy', 'spacy_lg', 'biobert-base', 'biobert-large',"
"and 'default'.")
tokenizer = default_tokenizer
is_bert_tokenizer = False
Expand Down