Found this remarkable, tight quote near the end of Umberto Eco's fabulously inventive Faucault's Pendulum. Apparently, Eco saw this quote in a way as an inspiration for his book - see a great discussion of this quote, from G.K. Chesterton, at http://www.chesterton.org/ceases-to-worship/.
Thinking back on Eco's fantastic story, I can see how this Chesterton saying fits - Eco creates a sense of inquiry, both among Belbo and Casaubon and Diotallevi in which their imagination can range everywhere over everything, as well as in the Diabolicals. This inquisitiveness, which delves into anything and everything, is diametrically opposed to a mere belief in the stories and beliefs related to God. They leave behind understanding rooted only in Biblical verse, and roam into magical and irrational specters beyond the realm of ancient text. Imagination fuses ancients texts into a faux-reality that encompasses everything.
In the same way, I too attribute Chesterton's saying to the belief of what we call Atheism (not the remainder of that philosophers works however). To the initiate of religious belief, to the notion of a God or gods directing and overseeing the creation and existence of the universe and all living beings on Earth, the Atheist believes nothing. If he or she doesn't believe in God, there is a vacancy, an erroneous absence, a deprivation of belief, of belief in anything.
I remember speaking to a new initiate into Christianity, who passionately believed in God and Jesus and all their teachings, who came from a culture which has been converted, in the late 19th/early 20th centuries, to Christianity. And I asked her, what did her ancestors believe before this conversion. "They believed in nothing." This was wrong, naturally, for an historical sense - surely her ancestors had beliefs, envisioned a theistic world as well, one no less worthy than the Christianity they had been forced to accept. They did not believe in nothing.
And neither do Atheists today or yesterday. Take the Biblical notion of creation, for instance. God, they say, created the world in seven days. Simple, eh? So Atheists who don't believe in the Genesis story believe, instead, in nothing? On the contrary, the scientific explorations of the Cosmos, of the Big Bang theory, of Evolution, have ranged over all possibilities, all avenues of scientific thought and reasoned experimentation. - ie. EVERYTHING. The world of an Atheist is not overseen by a god. It is overseen by a set of laws of physical and chemical and biological science that is, in fact, far more expansive and amazing and capable of inspiring wonder than a mere seven days creation. The Atheist believes in EVERYTHING.