By definition an Observatory is a large building usually housing one or two large Scopes. Traditionally these have been optical telescopes, but in recent years Radioscopes have been used.
But you can also have an observatory! By simply getting a chair, Star map, and some Binoculars or a Telescope and sit in a clear spot on a good night and you can have your own personal Observatory! However, it won't be in a building!