The original Museum was the “hall of muses” in Alexandria, and the original Mausoleum was a memorial to the Persian satrap Mausoleus at Halicarnassus, which was so impressive that it was designated one of the Seven Wonders of the World. The Lyceum was a school founded by Aristotle in the grove of Apollo Lykeos, while the Athenaeum was a school in Rome that was named after a nearby temple to Athena. Finally, the Colosseum (also spelled Coliseum) was a venue for gladiatorial combat in Rome that took its name from a colossal statue of Nero.

All of these “-eum” words have become general words in English. Museums and mausoleums are all over the place, and a lot of cities have coliseums (although lyceum, as “lycée,” is much more common in French*).

Plenty of other such words have not become general ones. Either they still refer to specific buildings, or specify types of buildings, in the ancient world only. I jotted down a few:

Ramesseum – the memorial temple of Ramesses the Great
Mithraeum – a temple to the god Mithras
pyreum – a Zoroastrian fire-altar (from Greek pura = fire)
Serapeum – a temple to Serapis in Egypt

And -eum is simply the Latin equivalent of the Greek -eon, which we also see from time to time, as in Odeon (a venue for the singing of odes) or Pantheon (a temple for worshiping all gods).

* It’s interesting how “lycée” is common in French but “academy” is common in English. The Academy, of course, was Plato’s school, in opposition to which Aristotle founded the Lyceum. But I’ve always considered the French to be far more Platonic than Aristotelian, and the English more Aristotelian than Platonic.