Unicode Greek friendly Unix tools

Tell us about interesting projects involving biblical Greek. Collaborative projects involving biblical Greek may use this forum for their communication - please contact jonathan.robie@ibiblio.org if you want to use this forum for your project.
Stephen Carlson
Posts: 2566
Joined: May 11th, 2011, 10:51 am
Location: Melbourne
Contact:

Unicode Greek friendly Unix tools

Post by Stephen Carlson » March 19th, 2016, 11:11 pm

I recently got an account on a Unix machine (thanks to my son) and I'm interesting in playing around with doing a little computing with Unicode Greek texts. I git clone'd Tauber's SBLGNT morph files to start off with. I'm finding that my awk | sort isn't working so well because the standard util 'sort' doesn't grok Unicode Greek (it treats the precomposed characters as if they don't exist).

Any advice for basic Unicode Greek compliant tools, etc.?
Stephen C. Carlson, Ph.D.
Melbourne, Australia

Alan Bunning
Posts: 218
Joined: June 5th, 2011, 7:31 am
Contact:

Re: Unicode Greek friendly Unix tools

Post by Alan Bunning » March 20th, 2016, 8:20 am

Unfortunately, many computer languages still don’t support Unicode at all and for the few that do (i.e. Java, Javascript, Python, etc.) you still have to be very careful (particularly in dealing with strings and string lengths). I often use Javascript if I have to deal with Unicode characters. Sometimes I use older tools with Unicode and deal with them as individual bytes, and in this regard using a Unicode character format with a fixed number of bytes (UCS-2, UCS-4) is usually easier to work with (you also have to know the byte order). I have done this with awk on occasion, but some tools may treat a byte as an ASCII character and strip off the 8th bit which causes problems. If you aren’t going to actually manipulate the Unicode text but merely deal with it as a field, many of the Unix tools will still work (UTF-8 is a good choice that is sort of “backwards compatible” with ASCII). What I end up doing a lot of the time for Greek, is simply convert the Unicode to an ASCII format such as Betacode and then all the languages and UNIX tools work wonderfully. I do all of my processing in ASCII and then convert back to Unicode characters when it comes time for output. Spreadsheet programs like Excel and LibreOffice also handle Unicode pretty well and are easy to use when manipulating column delimited text.

Jonathan Robie
Posts: 3101
Joined: May 5th, 2011, 5:34 pm
Location: Durham, NC
Contact:

Re: Unicode Greek friendly Unix tools

Post by Jonathan Robie » March 20th, 2016, 2:40 pm

Stephen Carlson wrote:I recently got an account on a Unix machine (thanks to my son) and I'm interesting in playing around with doing a little computing with Unicode Greek texts. I git clone'd Tauber's SBLGNT morph files to start off with. I'm finding that my awk | sort isn't working so well because the standard util 'sort' doesn't grok Unicode Greek (it treats the precomposed characters as if they don't exist).

Any advice for basic Unicode Greek compliant tools, etc.?
I generally use an XML stack that has pretty good support for Unicode ... but I think both James and Ulrik use awk | sort.

Start here, make sure your locale is set up for Unicode. What distribution are you using?
ἐξίσταντο δὲ πάντες καὶ διηποροῦντο, ἄλλος πρὸς ἄλλον λέγοντες, τί θέλει τοῦτο εἶναι;
http://jonathanrobie.biblicalhumanities.org/

Stephen Carlson
Posts: 2566
Joined: May 11th, 2011, 10:51 am
Location: Melbourne
Contact:

Re: Unicode Greek friendly Unix tools

Post by Stephen Carlson » March 20th, 2016, 3:32 pm

Jonathan Robie wrote:Start here, make sure your locale is set up for Unicode. What distribution are you using?
Thanks, I'll take a look. I'm using Ubuntu 15 and Perl 5.20. I've already cut some scripts to generate KWICs (a concordance format some linguistics like). I'll see if I can start a repo on GitHub with them. Baby steps.

ETA: On my system the monotonic Greek sorts just fine. It's the polytonic Greek (pre-composted) stuff that one. I doubt it's a locale issue, or more precisely, I doubt someone's implemented a locale that works for polytonic Greek. (Converting to Betacode won't sort right without additional hackery.)
Stephen C. Carlson, Ph.D.
Melbourne, Australia

jtauber
Posts: 53
Joined: May 31st, 2011, 11:34 am
Location: Burlington, MA, USA
Contact:

Re: Unicode Greek friendly Unix tools

Post by jtauber » March 20th, 2016, 6:37 pm

Stephen Carlson wrote:I'm finding that my awk | sort isn't working so well because the standard util 'sort' doesn't grok Unicode Greek (it treats the precomposed characters as if they don't exist).

Any advice for basic Unicode Greek compliant tools, etc.?
While I do basic stuff on the Unix command line, I switch to Python for anything involving sorting. I have a Python implementation of the Unicode Collation Algorithm https://github.com/jtauber/pyuca which, while not fast, sorts Greek correctly out of the box.

I have a work in progress article discussing Python, Unicode and Greek: http://jktauber.com/articles/python-uni ... ent-greek/
James Tauber
http://jktauber.com/

Stephen Carlson
Posts: 2566
Joined: May 11th, 2011, 10:51 am
Location: Melbourne
Contact:

Re: Unicode Greek friendly Unix tools

Post by Stephen Carlson » March 20th, 2016, 6:50 pm

jtauber wrote:While I do basic stuff on the Unix command line, I switch to Python for anything involving sorting.
Thanks for the pointer to your code. I'm more of Perl person, but I'll see what I can learn.
Stephen C. Carlson, Ph.D.
Melbourne, Australia

Jonathan Robie
Posts: 3101
Joined: May 5th, 2011, 5:34 pm
Location: Durham, NC
Contact:

Re: Unicode Greek friendly Unix tools

Post by Jonathan Robie » March 21st, 2016, 9:13 am

You might also want to look into some of the natural language libraries like kwic.py and NLTK. Take a look at this too.
ἐξίσταντο δὲ πάντες καὶ διηποροῦντο, ἄλλος πρὸς ἄλλον λέγοντες, τί θέλει τοῦτο εἶναι;
http://jonathanrobie.biblicalhumanities.org/

caseyperkins
Posts: 9
Joined: January 23rd, 2015, 3:26 pm

Re: Unicode Greek friendly Unix tools

Post by caseyperkins » March 21st, 2016, 11:32 am

Stephen Carlson wrote:
jtauber wrote:While I do basic stuff on the Unix command line, I switch to Python for anything involving sorting.
Thanks for the pointer to your code. I'm more of Perl person, but I'll see what I can learn.
I would say stick with Perl. This kind of thing is where Perl really excels. I was able to work with Perl for processing Unicode Greek texts with much greater ease than would have been possible with other programming languages.

Jonathan Robie
Posts: 3101
Joined: May 5th, 2011, 5:34 pm
Location: Durham, NC
Contact:

Re: Unicode Greek friendly Unix tools

Post by Jonathan Robie » March 21st, 2016, 12:07 pm

caseyperkins wrote:I would say stick with Perl. This kind of thing is where Perl really excels. I was able to work with Perl for processing Unicode Greek texts with much greater ease than would have been possible with other programming languages.
Compared to Python or Ruby? What advantages do you see?

My impression is that these languages are broadly similar in what they can do.
ἐξίσταντο δὲ πάντες καὶ διηποροῦντο, ἄλλος πρὸς ἄλλον λέγοντες, τί θέλει τοῦτο εἶναι;
http://jonathanrobie.biblicalhumanities.org/

Stephen Carlson
Posts: 2566
Joined: May 11th, 2011, 10:51 am
Location: Melbourne
Contact:

Re: Unicode Greek friendly Unix tools

Post by Stephen Carlson » March 21st, 2016, 5:08 pm

Jonathan Robie wrote:You might also want to look into some of the natural language libraries like kwic.py and NLTK. Take a look at this too.
Last Sunday afternoon I just slapped together a quickie set of Perl utilities that slurps in the SBLGNT morph files and burps out an n-gram KWIC concordance for a given lemma or inflected word. It's really easy to do this kind of stuff in Perl. (And part of the time was (re)learning Perl.)
Stephen C. Carlson, Ph.D.
Melbourne, Australia

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest