It did seem like a pretty small area of beach for 400 K to be there without being jam packed.
They didn't gather all of the troops on the beach at once. A lot of the soldiers were kept in the rear to protect those on the beach. They apparently only queued up as they were receiving notices to do so, as the larger ships were coming in to the mole.
The Brits/French actually thought the Nazi's were going to keep up the blitzkrieg attack and were expecting tanks to come rolling through the town. Fortunately, the tanks had advanced well beyond the German ground troops and supply lines. Rommel was apparently concerned about losing tanks if they pushed on before the supply lines and fuel caught up with them.
That's what actually bought the 5 to 6 days for the Brits/French/Belgians to evacuate as many as they did. There really was only the mole in place that the larger naval ships could tie up and take on passengers. It was only when the smaller fishing boats/trawlers, passenger boats, sail boats, etc. started arriving en masse could they start taking advantage of the larger beach area and the troops could start moving away from the mole to catch a ride.