Well, it does seem to be rare outside of Christian writings, but it's used right along side of the nominative for the vocative, so that's not determinative of Hebrews 1:18. cf.
ὁ Φαρισαῖος σταθεὶςa πρὸς ἑαυτὸν ταῦτα προσηύχετο,
Ὁ θεός, εὐχαριστῶ σοι ὅτι οὐκ εἰμὶ ὥσπερ οἱ λοιποὶ τῶν ἀνθρώπων, ἅρπαγες, ἄδικοι, μοιχοί, ἢ καὶ ὡς οὗτος ὁ τελώνης...
Where θεός clearly used in a vocative sense.
I have always understood it as vocative based on the original context. The translation above is theoretically viable, but...
Since I'm busy with other things today, I'll cite the NET note:
Or possibly, “Your throne is God forever and ever.” This translation is quite doubtful, however, since (1) in the context the Son is being contrasted to the angels and is presented as far better than they. The imagery of God being the Son’s throne would seem to be of God being his authority. If so, in what sense could this not be said of the angels? In what sense is the Son thus contrasted with the angels? (2) The μέν … δέ (men … de) construction that connects v. 7 with v. 8 clearly lays out this contrast: “On the one hand, he says of the angels … on the other hand, he says of the Son.” Thus, although it is grammatically possible that θεός (theos) in v. 8 should be taken as a predicate nominative, the context and the correlative conjunctions are decidedly against it. Hebrews 1:8 is thus a strong affirmation of the deity of Christ.
Biblical Studies Press. (2006). The NET Bible First Edition Notes (Heb 1:8). Biblical Studies Press.