Is it possible to provide tokenized input?

[qiuwei]

I would like to use an already tokenized text as input. Is it possible?

[ljm625]

Hi there, could you give a example about the tokenized text? For my understanding of the tokenized text, currently you cant, but feel free to give me a example and i can try it out

[qiuwei]

Instead of providing a string, I would like to provide an array of words as the input, assuming the tokenzition is already done by other tools.

…

On Wed, Nov 1, 2017, 2:39 AM Jiaming Li ***@***.***> wrote: Hi there, could you give a example about the tokenized text? For my understanding of the tokenized text, currently you cant, but feel free to give me a example and i can try it out — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABPnuZ0iFXqBCUNBPh9Z-u8_xZKCGXe4ks5sx8uzgaJpZM4QLZeE> .

[ljm625]

Hi there,

the above image is how syntaxnet works, it use the whole sentence for the input of the model, so I believe it might not possible to use the tokenized text as input, but if you can recombine them into a sentence, it works.

btw, if you are using syntaxnet directly with tokenized text and it works, pls give me a example so I can add an api for it.

[qiuwei]

Hi, I am a beginner of syntaxnet and dragnn. I don't know much about how syntaxnet works internally.
However, your code did use a segmenter before running the parser.

Considering the fact that for the Chinese, Korean or Japanese model, the input is actually required to be segmented before passed in, I think tokenized text being used as the input should be possible.

I will look into it and get back to this issue if I find out something.

[ljm625]

ok, thanks, will check further as well when I have time ;)