Should elisions be tokenized?

Should elisions be tokenized?

Postby NathanSmith » March 2nd, 2013, 7:51 pm

I found when tokenizing the SBLGNT with NLTK that elisions are split off as separate tokens - e.g. μεθ, ’ instead of μεθ’. This is due to how the NLTK default parser works, and I've found a way around it.

But it raised the question for me: should Greek elisions be considered separate tokens or not? I know that in English the possessive use of the apostrophe is tokenized separately (e.g. Mark, 's) because it indicates the possession, and I believe contractions are handled similarly.

For Greek elisions my feeling is that they should be not be separate tokens because they do not add anything semantically to the sentence and are a function of pronunciation, essentially. Someone correct me if I am making a bad assumption here.
NathanSmith
 
Posts: 49
Joined: June 10th, 2011, 12:38 am
Location: Portland, OR, USA

Re: Should elisions be tokenized?

Postby Stephen Carlson » March 2nd, 2013, 9:52 pm

I'm not exactly sure what your goal is, but based on your blog it seems that not tokenizing elisions is what you want. In fact, I would normalize μεθ to μετα. The NA28 changed its editorial practice about about ἀλλά vs. ἀλλ' before a vowel, and that is one area where editions may differ.
Stephen C. Carlson, Ph.D. (Duke)
Post-Doctoral Fellow, Faculty of Theology, Uppsala
Stephen Carlson
 
Posts: 1857
Joined: May 11th, 2011, 10:51 am
Location: Uppsala University

Re: Should elisions be tokenized?

Postby Ken M. Penner » March 2nd, 2013, 11:17 pm

NathanSmith wrote:For Greek elisions my feeling is that they should be not be separate tokens because they do not add anything semantically to the sentence and are a function of pronunciation, essentially. Someone correct me if I am making a bad assumption here.

I agree.
Ken M. Penner
St. Francis Xavier University
Ken M. Penner
 
Posts: 617
Joined: May 12th, 2011, 7:50 am
Location: Antigonish, NS, Canada


Return to Syntax and Grammar

Who is online

Users browsing this forum: No registered users and 1 guest