Egyptologists have suggested as few as 4,000 men would be needed early on for this job. It would be a tight squeeze but they would fit in the workers village. This would last only three or four seasons before needs would begin decreasing.

I believe this estimate is too low but this should well be within the capabilities of the port and village. What they didn't have room for was an army of stone draggers and water haulers. I believe they could easily accommodate 7 or 8000 men, women, and children in the village. The first few seasons would be a little cramped.