Energy for millions of generations could come from several sources.
The sun is expected to maintain roughly the same output for at least hundreds of million of years, probably several billion years. As long as the earth maintains its orbit and weather patterns, the sun would deposit many thousand times more energy on the earth's surface than is consumed by current civilizations. It could be harnessed directly or indirectly via biomass, wind or water.
Tidal forces caused by the moon are also expected to change little in strength over millions of years and could also serve as energy for long periods. The moon moves slowly away from the earth, its distance would be expected to increase by about 10% in a billion years due to tidal forces.
Geothermal energy from nuclear decays of radioactive elements in the earth is also expected to be only slowly reduced over the course of billions of years.
Energy from nuclear fission would not play a major role in civilizations lasting for millions of generations, as fuel would only last for a few tens of generations. Such fuels may e.g., be beneficial for extended space travel and existing sources of such nuclear fission fuels might better be saved for applications where its use becomes indispensable.
Energy from nuclear fusion might last for millions of years at current usage if lithium can be won from the sea or billions of years if fusion using only deuterium would become feasible. Harnessing nuclear fusion for energy consumption with currently conceivable technologies seems considerably more expensive than collecting energy from the sun.