Portability is the big one for C#, but as for why it's considered 'slower', a VM compiled language like C# has to be translated at runtime, so an operation in native machine code will always have a significant advantage in speed (although a good VM can overcome that with caching). The simple reason is that a simple CPU instruction will execute at least twice as fast as a VM language instruction that has to be translated and then executed. There's more work to do for every single instruction, so you don't get the same speed unless the VM uses tricks to get that speed for you.
That said, modern machines, as was mentioned, can execute horrifically suboptimal code at near instantaneous speeds, so there's no reason to avoid C# unless you're doing something very low-level that needs to be blazing fast for some reason. Even in those cases you can find ways of bending C# to your will if you must. My only real gripe with C# is that it isn't Ruby. If I'm using a VM language I want it to JIT compile from raw scripts that read like Terry Pratchett novels and I want it do some of the crazy stuff that Ruby can get away with such as dynamically redefining classes at runtime or that beautiful, horrible monster known as eval().
C# is taken a lot more seriously than Ruby, though. <sad panda>