It would have to be for a very large number of us wouldn't it? It simply doesn't add up numerically, and I have yet too see this disrepancy successfully addressed by any believer in "traditional" reincarnation.
On the other hand, a lot of people report on experiences, links and flashbacks, and I'd like to think there is "something" going on here. So what could it be?
Being fascinated by the phenomena of consciousness I continuously work on a metaphysical model that can encompass all of humanity's experiences and beliefs. Admittedly this, suffice to call it notion, changes ever so often as I get some new ideas. But there is a body, and simply put (always the best) it goes like this:
Consciousness is, for lack of better terms, a force or dimension linked somehow to what we call matter (or energy, depending on what quantum mechanics problem one wishes to explain). This force is therefore in high concentrations around the heavenly bodies, and I believe the planets with organic life-forms are where it is "crystallized" in its purest form. There seems to be some sort of evolution going on here, and I think it can be witnessed in humanity's history as we progress from more barbaric to gentler beings (albeit it slowly and with a lot of setbacks).
On this wonderful planet we, the humans, are the most advanced nodes or vehicles for the force of consciousness, and we naturally develop a special relationship with one another. The Common Consciousness, which is a term I see being more and more used today among us Westerners is our local planetary area of concentration, and we all contribute to its growth as well as nourish from it.
The brain, our biologic electromagnetic computer, is a sender / receiver apparatus to this field as well as the organic vessel's CPU. The ongoing synapses shapes it to an unique form for every one of us, and the "I" "ego", "self" is simply the interface between this unit and the emanate field of consciousness.
The Common Consciousness is obviously storing all uploaded data, and when we "feel reincarnated" it really is a retrieval, for some reason, of some of this information.
Take no offence if this model affronts your own reality tunnel. Yours is as "good" as mine. It might just be the clumsy rambling of a very, very young soul. :-)