The connection speed itself isn't dropped when more users connect, but the throughput of data is. I.e. you can have 100 people connect to an access point at 54Mbit, but when they start using data that's when you'll see the difference.

The thing is not everyone will be using the network 100% of the time. I.e. you'll never have 100 people trying to download as fast as they can. You may have a couple of people doing this and the others downloading a web page or something.

Hopefully this answers your question