Really server side as long as you're not rendering the actions of players you could have a huge number of people in one spot from what I know. It's just making them appear on screen that is the issue.
Not really, more players in an area = more events occuring and more players that you need to send each piece of data to which cause outgoing bandwidth requirements to scale exponentially as you increase the number of players in an area.
When we hit MMO scale (10.000+ players) we can do a bit of fun math:
lets say we use 2 8 bit values for x,y (delta) , don't use z and orientation as a 8 bit value and send each player the position and rotation of all other players 10 times per second then we would send 9999*10000*10*24 bits per second or ~24Gbps,
by splitting those same players up into 1000 instances/areas with 10 players each we would only use 1000*10*9*10*24 = ~2.16Mbps upstream bandwidth for the same number of players and the same update frequency. (most games will require you to send for more data per player but the difference in magnitude stays the same, 1 room with 10.000 players is roughly 1000 times more expensive to host than 1000 rooms with 10 players each if you use a basic approach to networking.
MMOs have to work around this problem by throttling updates based on distance and importance and designing things in such a way that players will spread out (you can't effectively throttle updates based on distance and importance if everyone is having an all out brawl in a small room) and if there is content that alot of players will be doing at the same time you may have to use instancing for that content or make sure players will naturally spread out within that content(There may also be gameplay reasons to use instancing (smaller battles make each players individual performance more important). I would strongly recommend against moving players between instances on the fly (it sounds like it can get incredibly messy as even though it is fairly easy to keep players who are in a group together it will be a nightmare to keep track of interactions between groups (if group A is fighting group B and group C is moving in to pick off whatever remains after the fight you can't really separate those groups either without disrupting gameplay), assign groups to an instance as they enter the content (i.e, as they pass through the dungeon entrance) or design the content in such a way that players will spread out by themselves and add safeguards to deal with high player concentrations.
For pvp dungeons where you want competing groups to fight over the price i would either create dungeons with 2 or more starting points(depending on how many groups I want it) and then treat them like WoW battlegrounds (players queue up for it and start when all groups are ready) or have people join an existing instance when they enter unless all currently running instances are either full or close to finished (probably having the enemy respawn rate and/or enemy power/numbers get reduced in sections the previous groups have pushed through to give those who enter late a good shot at catching up and fighting the ones who started before them without giving them a clear path)