I think the first thing that those who listen to Greek audio of the New Testament is that they pronounce the accents correctly and consistently, and that they produce the vowels, diphthongs and consonants consistently. (I keep a resource page on audio recordings of the Greek New Testament at http://www.letsreadgreek.com/resources/greekntaudio.htm
. Each audio is rated according to its consistency of pronunciation, correct use of accents, pronunciation scheme, pathos, smoothness of reading, speed, etc. If the accents are not correct, then people cannot talk in a group about the same word, as sometimes the connection between the word and the sound of the word cannot be made.
A very good summary of the different pronunciation schemes can be found on John Schwandt's site, Institute of Biblical Greek, under the page titled Guide to Greek Pronunciation Conventions: How we pronounce ancient Greek, biblical (Koine) Greek, and modern Greek http://www.biblicalgreek.org/links/pronunciation.php
Those that use the Restored versus the traditional western academic Erasmian system (I'm included in that crowd) are doing so for several reasons:
1) They want to be as close to the New Testament mileau as possible.
2) They are sensitive that historical language pronunciation is the natural outcome of tendencies in any language. The phonetic changes that happened in a language happened for a reason. At any given point in time, there are certain sounds which would be heard or not heard. Those sounds often go hand in hand with other sounds. The system happened for a reason.
3) There are homonyms which in one dialect/pronunciation scheme are not homonyms in another.
4) Many consider that the books of the Greek New Testament were meant to be heard, not read. Hearing is important, and you need a consistent system for that.
5) Modern Greek considers Erasmian to be 'painful' to the Greeks' sense of prosody, harmony, euphony.
6) Erasmian is a contrived (mixed system) of pronunciation not representing the language as it appeared at any given historical period.
But these reasons hint at, do not really answer the question, "Why does it matter?"
It especially matters if you want to speak the language to others. Those who are able to speak and hear Greek at a conversational speed have strong internalization of the language that others who cannot do that. Latinists have an avid community of restored language speakers (I don't think they run against the tide of academia where in Greek circles, 90% of western academics use a non-historical pronunciation. Once you start speaking Greek and hearing it spoken, you want to get on board with others who are speaking it, and like human nature, you become what you surround yourself with.
On the other hand, if you only read it silently, you can do that anyway you want. I constantly remember reading classical English literature in high school. When we started talking in class about the book, I often found that I verbalized the names incorrectly. This happened many times, at times, I would omit whole syllables, at other times transpose characters, at other times, pronounce a name phonetically OK but not representative of the correct way that name was pronounced by everyone who knew except me.