My first thoughts on seeing this was: Wait, that's not true! And then it got me thinking, "What is a temple?"
To begin answering this question, we have to understand what "religion" is. For us, today, our thinking about these things is corrupted by one of the most potent mind viruses of our time - Christianity. More specifically, the version of Christianity that survived the Protestant Reformation. Religion for us is a private affair, a thing that we do on top of living our lives. We go to our work every day, we go to movies once in a while, we go on vacation, we retire, we spend time with our loved ones...and sometimes we pray to our Gods. There's life, and then there's practicing our religion. There are things that we owe to our loved ones, to ourselves, to the government and sometimes, if you are religious, there are things you owe to your Gods. The ancient ones saw religion completely differently. Rather, it was such innate part of their life, they didn't see it at all. When being religious involves everything from getting educated, getting married, earning your bread and butter, raising kids, and eventually getting old and dying with honor, would they know that they're being religious?
(To expand: This is even more pronounced in polytheistic societies where lack of belief in one God, can always be substituted by belief in other God which could be easily be included into the existing pantheon.)
This changes with Christianity, or the flavour of Christianity that we know and has taken over the minds of almost everyone in the world. The separation of the Church and the State, also defines the religion to be a personal voluntary practice. It allowed people to have a life outside of religion, which would've been inconceivable for the ancients.
I digress, the main issue is about temples.
So, let's assume you (your civilization) has created Gods. As it happened in those ancient civilizations like Hindus or the Greeks, the divine stood for abstract ideas and natural elements - rain, wind, fire, thunder, seas, rivers, big trees, etc. Any adjective you can think of for describing these elements, becomes an attribute of the divine. Any utility or nuisance value that these entities hold becomes their power. Having made these Gods, how do you make them manifest? Then came the idols and the temples.
The Hindus and the Greeks, I think understood temples in a very similar way. The temple is the "devalaya" or the "oikos tou theou", both essentially meaning "the house of god". Why does the god have to be housed, you ask? Because abstract entities aren't digestable until they are "pointable". One needs to be able to point at a thing that exists in the physical world as a proxy for what they are talking about. This is how the human minds works. We make things in the physical world to convey abstract ideas from one mind to another.
Temples are one of these things. You tie the abstract to a defines space in the physical world. "This right here is where God is." Only then can the idea be put to downstream use. Even if we assume that the pre-Vedic people lived a near-nomadic life without temples, the fact that they conducted ritual sacrifices, with sacrificial altars, the sites for which were chosen carefully can be assumed as an act of grounding the divine. The sacrificial site itself becomes the temple in such a scenario.