Previous research has suggested that near the end of their existence some 40,000 years ago, only about 1,000 Neanderthals were left on Earth.
But the new study shows their population was far larger -- likely numbering in the tens of thousands -- though they existed in isolated groups across Europe, said the report in the Proceedings of the National Academy of Sciences.
The genetic clues include Neanderthal DNA that contains mutations that usually occur in small populations with little genetic diversity.
Also, Neanderthal remains -- found in various locations -- are genetically different from each other.
"The idea is that there are these small, geographically isolated populations, like islands, that sometimes interact, but it's a pain to move from island to island," said co-author Ryan Bohlender, a post-doctoral fellow at the MD Anderson Cancer Center at the University of Texas.
"So, they tend to stay with their own populations."
Using a new method to analyze DNA sequence data, researchers also found that Neanderthals split from another mysterious lineage, known as the Denisovans, about 744,000 years ago, much earlier than any other estimation of the split.
After that, the global Neanderthal population grew to tens of thousands.
"This hypothesis is against conventional wisdom, but it makes more sense than the conventional wisdom," lead author Alan Rogers, professor in the department of anthropology at the University of Utah.
"There's a rich Neanderthal fossil record. There are lots of Neanderthal sites," he said.
"It's hard to imagine that there would be so many of them if there were only 1,000 individuals in the whole world."
Very little is known about the Denisovans, sometimes described as the Eastern cousins of Neanderthals.
Only a few pieces of their remains -- including some teeth and a pinkie bone -- have ever been found.
Both Denisovans and Neanderthals mated with the ancestors of modern humans, who emerged from Africa about 60,000 years ago.
Researchers are not sure exactly why Neanderthals or Denisovans eventually died out, but it could have been due to harsh climate, or competition for scare resources with modern humans.
The study was based on comparing the genomes of four human populations: Modern Eurasians, modern Africans, Neanderthals and Denisovans.
This improved statistical method, called legofit, helped researchers estimate the percentage of Neanderthal genes flowing into modern Eurasian populations -- which they confirmed was about two percent.
It method revealed the date at which these ancestral populations diverged from each other, and their population sizes.
