Should elisions be tokenized?
Posted: March 2nd, 2013, 7:51 pm
I found when tokenizing the SBLGNT with NLTK that elisions are split off as separate tokens - e.g. μεθ, ’ instead of μεθ’. This is due to how the NLTK default parser works, and I've found a way around it.
But it raised the question for me: should Greek elisions be considered separate tokens or not? I know that in English the possessive use of the apostrophe is tokenized separately (e.g. Mark, 's) because it indicates the possession, and I believe contractions are handled similarly.
For Greek elisions my feeling is that they should be not be separate tokens because they do not add anything semantically to the sentence and are a function of pronunciation, essentially. Someone correct me if I am making a bad assumption here.
But it raised the question for me: should Greek elisions be considered separate tokens or not? I know that in English the possessive use of the apostrophe is tokenized separately (e.g. Mark, 's) because it indicates the possession, and I believe contractions are handled similarly.
For Greek elisions my feeling is that they should be not be separate tokens because they do not add anything semantically to the sentence and are a function of pronunciation, essentially. Someone correct me if I am making a bad assumption here.