Page 1 of 2

Unicode Greek friendly Unix tools

Posted: March 19th, 2016, 11:11 pm
by Stephen Carlson
I recently got an account on a Unix machine (thanks to my son) and I'm interesting in playing around with doing a little computing with Unicode Greek texts. I git clone'd Tauber's SBLGNT morph files to start off with. I'm finding that my awk | sort isn't working so well because the standard util 'sort' doesn't grok Unicode Greek (it treats the precomposed characters as if they don't exist).

Any advice for basic Unicode Greek compliant tools, etc.?

Re: Unicode Greek friendly Unix tools

Posted: March 20th, 2016, 8:20 am
by Alan Bunning
Unfortunately, many computer languages still don’t support Unicode at all and for the few that do (i.e. Java, Javascript, Python, etc.) you still have to be very careful (particularly in dealing with strings and string lengths). I often use Javascript if I have to deal with Unicode characters. Sometimes I use older tools with Unicode and deal with them as individual bytes, and in this regard using a Unicode character format with a fixed number of bytes (UCS-2, UCS-4) is usually easier to work with (you also have to know the byte order). I have done this with awk on occasion, but some tools may treat a byte as an ASCII character and strip off the 8th bit which causes problems. If you aren’t going to actually manipulate the Unicode text but merely deal with it as a field, many of the Unix tools will still work (UTF-8 is a good choice that is sort of “backwards compatible” with ASCII). What I end up doing a lot of the time for Greek, is simply convert the Unicode to an ASCII format such as Betacode and then all the languages and UNIX tools work wonderfully. I do all of my processing in ASCII and then convert back to Unicode characters when it comes time for output. Spreadsheet programs like Excel and LibreOffice also handle Unicode pretty well and are easy to use when manipulating column delimited text.

Re: Unicode Greek friendly Unix tools

Posted: March 20th, 2016, 2:40 pm
by Jonathan Robie
Stephen Carlson wrote:I recently got an account on a Unix machine (thanks to my son) and I'm interesting in playing around with doing a little computing with Unicode Greek texts. I git clone'd Tauber's SBLGNT morph files to start off with. I'm finding that my awk | sort isn't working so well because the standard util 'sort' doesn't grok Unicode Greek (it treats the precomposed characters as if they don't exist).

Any advice for basic Unicode Greek compliant tools, etc.?
I generally use an XML stack that has pretty good support for Unicode ... but I think both James and Ulrik use awk | sort.

Start here, make sure your locale is set up for Unicode. What distribution are you using?

Re: Unicode Greek friendly Unix tools

Posted: March 20th, 2016, 3:32 pm
by Stephen Carlson
Jonathan Robie wrote:Start here, make sure your locale is set up for Unicode. What distribution are you using?
Thanks, I'll take a look. I'm using Ubuntu 15 and Perl 5.20. I've already cut some scripts to generate KWICs (a concordance format some linguistics like). I'll see if I can start a repo on GitHub with them. Baby steps.

ETA: On my system the monotonic Greek sorts just fine. It's the polytonic Greek (pre-composted) stuff that one. I doubt it's a locale issue, or more precisely, I doubt someone's implemented a locale that works for polytonic Greek. (Converting to Betacode won't sort right without additional hackery.)

Re: Unicode Greek friendly Unix tools

Posted: March 20th, 2016, 6:37 pm
by jtauber
Stephen Carlson wrote:I'm finding that my awk | sort isn't working so well because the standard util 'sort' doesn't grok Unicode Greek (it treats the precomposed characters as if they don't exist).

Any advice for basic Unicode Greek compliant tools, etc.?
While I do basic stuff on the Unix command line, I switch to Python for anything involving sorting. I have a Python implementation of the Unicode Collation Algorithm https://github.com/jtauber/pyuca which, while not fast, sorts Greek correctly out of the box.

I have a work in progress article discussing Python, Unicode and Greek: http://jktauber.com/articles/python-uni ... ent-greek/

Re: Unicode Greek friendly Unix tools

Posted: March 20th, 2016, 6:50 pm
by Stephen Carlson
jtauber wrote:While I do basic stuff on the Unix command line, I switch to Python for anything involving sorting.
Thanks for the pointer to your code. I'm more of Perl person, but I'll see what I can learn.

Re: Unicode Greek friendly Unix tools

Posted: March 21st, 2016, 9:13 am
by Jonathan Robie
You might also want to look into some of the natural language libraries like kwic.py and NLTK. Take a look at this too.

Re: Unicode Greek friendly Unix tools

Posted: March 21st, 2016, 11:32 am
by caseyperkins
Stephen Carlson wrote:
jtauber wrote:While I do basic stuff on the Unix command line, I switch to Python for anything involving sorting.
Thanks for the pointer to your code. I'm more of Perl person, but I'll see what I can learn.
I would say stick with Perl. This kind of thing is where Perl really excels. I was able to work with Perl for processing Unicode Greek texts with much greater ease than would have been possible with other programming languages.

Re: Unicode Greek friendly Unix tools

Posted: March 21st, 2016, 12:07 pm
by Jonathan Robie
caseyperkins wrote:I would say stick with Perl. This kind of thing is where Perl really excels. I was able to work with Perl for processing Unicode Greek texts with much greater ease than would have been possible with other programming languages.
Compared to Python or Ruby? What advantages do you see?

My impression is that these languages are broadly similar in what they can do.

Re: Unicode Greek friendly Unix tools

Posted: March 21st, 2016, 5:08 pm
by Stephen Carlson
Jonathan Robie wrote:You might also want to look into some of the natural language libraries like kwic.py and NLTK. Take a look at this too.
Last Sunday afternoon I just slapped together a quickie set of Perl utilities that slurps in the SBLGNT morph files and burps out an n-gram KWIC concordance for a given lemma or inflected word. It's really easy to do this kind of stuff in Perl. (And part of the time was (re)learning Perl.)