How do dungeon designers tend to approach designing a dungeon? Is it room by room where you start with designing entrance, then the room(s) off the entrance, and so on? Do you create the entire topology of the dungeon first and then go to the room level? Do you come up with a series of challenges first and go from there? Or do you take some other approaches?

A dungeon, or any level, has to have a purpose. Even if the purpose is as simple as "an abandoned cave that some scorpions have made their home," it has to have a purpose. Once you have the general purpose of the level in mind, you can shape it out appropriately.

There are a variety of ways to create the actual map. You can start with a blank sheet of graphing paper, pick a starting point near the middle or outer edge, and then go step by step and "carve" your path through the dungeon, including extra corridors and rooms as needed.

For above-ground areas, such as houses or towers, I'll generally start with the shape of the area and then draw in some walls and doors to guide the player through the level. If I don't already have a general idea of where I want the important action points to be, then I'll usually get a good idea of where they can fit as I'm designing the layout. I've found that when I complete a level layout without placing down any areas of interest, I end up scrapping it. So I'd suggest having a better-than-vague idea of where you want the points of interest to lie in your level.

The best way to design any area, and that is especially easy to do for in-door dungeons such as caves, is in my opinion to simply make the map of the area and place elements on that map.
Then have your system automatically create the area with some style you defined. You could eventually modify it afterwards to add specific things.

Some people actually use that kind of thing to generate whole cities using a city map and architectural rules to set up buildings.

I DM a lot in D&D, and my habit is to make the outlines for size first, then lay down the entrance and the 'exit' or boss chamber. After that, it really depends on style. The dungeon is still closed world, but some are not much more than passages with rooms hanging off, while others are underground cities.

I've never actually made a dungeon, so take this with a grain of salt, but I would go about it like this:

Any area of an environment has two purposes, the one the designer has for it as part of the game, and the one the in game characters are fauna would use it for. Therefore an in game house needs a bedroom and a kitchen and some living space because that's what the NPC that lives there needs out of it. If the designer needs a dank basement for some monsters to inhabit or a closet with a secret passage, then that needs to be there too, but that doesn't make the kitchen unimportant.

In the case of a cave, its in game purpose is more akin to keeping in mind how real caves are formed and how they are shaped. Caves are not 8ft high and level all the way through. There are crawl spaces, high rooms, many rooms you duck through, inaccessible areas, etc. Don't add anything in that the player could never see of course, but add in some things he can see but maybe not get to, or through.

For large buildings, I might rough out some ideas for how they would be designed for their game world purpose first, then figure out how to open areas and close off areas to make it an appropriate level design for what I need.

I think designing areas like this would give them a more realistic touch while not compromising their viability as a dungeon. This means the player isn't left feeling like the area he's in is just an arena for the monsters to fight him in.

Hope that helps.