So, if it is really like developers building their own gaming computer (Buying and putting together their selected video card, motherboard, processor etc) then putting it inside of a cabinet connecting to arcade parts like the joysticks and running their own arcade game off of the hard drive...
Does that mean the games in those modern arcade cabinets are programs running on Windows? Or other Operating system like Linux, or maybe the developers own made distro of Linux?
Yes. There's a whole industry behind it. The Taito Type X2 is widely used today (BlazBlue fighting series comes to mind) and runs on Windows XP Embedded.
Basically what devs gain from this is a friendly known environment (i.e. x86 processor, Windows XP, DirectX, Visual Studio), stable known hardware specs (like in console development); and has a few customizations (most notably the hard drive is encrypted and all data in and out is decrypted/encrypted on the fly to prevent people from just cloning the drive and distributing the game online in pirate sites)