If you're talking about assets (player/enemy sprites, tilesets, etc.), you can either harvest some of the free stuff floating around the internet, or throw together your own "programmer art." It doesn't have to be pretty if you're just using it for testing. I've gone with the latter thus far, although I do have somewhat of an art background (Protip: don't get a degree in architecture. Ever.).
My biggest concern is graphics. I have absolutely no concept of graphics and am not exactly an artistic guy. What do you do for graphics?
C#: XNA exists. Microsoft may have dropped their official support, but it's still great. There's also MonoGame, an open source XNA implementation, which is cross-platform if you're into that sort of thing. As far as I know, the guys behind it intend to keep going despite Microsoft's bizarre loss of interest in XNA.
Of C# and Java, which would be best? (I know, I know.. the generic no language is best answer). What's positive/not-so positive about each of these languages in game development terms?
Java: I can't think of a reason to recommend Java over C# for game programming if you're equally comfortable with both. Err... Minecraft is Java. So... there's that, I guess.