cf.
https://en.wikipedia.org/wiki/Greek_to_me
It may have been a direct translation of a similar phrase in Latin: "Graecum est; non legitur" ("it is Greek, [therefore] it cannot be read"). This phrase was increasingly used by monk scribes in the Middle Ages, as knowledge of the Greek alphabet and language was dwindling among those who were copying manuscripts in monastic libraries.
The usage of the metaphor in English traces back to early modern times, and it is used in 1599 in Shakespeare's play Julius Caesar, as spoken by Servilius Casca to Cassius after a festival in which Caesar was offered a crown:
CASSIUS: Did Cicero say any thing?
CASCA: Ay, he spoke Greek.
CASSIUS: To what effect?
CASCA: Nay, an I tell you that, I'll ne'er look you i' the face again: but those that understood him smiled at one another and shook their heads; but, for mine own part, it was Greek to me. I could tell you more news too: Marullus and Flavius, for pulling scarfs off Caesar's images, are put to silence. Fare you well. There was more foolery yet, if I could remember it.
(William Shakespeare, The Tragedy of Julius Caesar (1599))
Here, Casca's literal ignorance of Greek is the source of the phrase, using its common meaning to play on the uncertainty among the conspirators about Cicero's attitude to Caesar's increasingly regal behaviour. Shakespeare was not the only author to use the expression.
The Neo-Greek version of this, by the way, as the above article reports it: Αυτά μου φαίνονται κινέζικα. i.e. "This strikes me as Chinese." On the other hand, Cypriot Greek employs Εν τούρτζικα που μιλάς; i.e. "Are you speaking Turkish?"