When you're talking to a machine across the network, you have no idea what the code is that's running on the other end. It may be your server. It may be a Python script. It may be an alien that's really good at typing hex code into a telnet window :-)

Thus, the question of "what /could/ a cheater do if I let users host games" is "anything that's possible using your network protocol."

If the network protocol downloads maps from the hosting server, for example, then a malicious hoster could download whatever files he want. A very very large file that takes up all disk space? Files containing all kinds of contraband, terrorist threads, and child porn to make the player vulnerable to law enforcement search? Carefully crafted images that root the machine if opened with a vulnerable image decoder? That kind of thing. You can address this by, for example, setting an upper limit on the size of maps, and enforcing this on the downloaded side, and also enforcing that all assets are "baked into" the map file. You probably also want to ensure that the map file actually conforms to the map file format (header bytes, internal structure) by carefully validating it with code that won't thrash the stack or allocate too much memory if some internal data field is wrong.

If the network protocol allows the affecting of game entities during play (which it most likely does) then the attacker can make whatever entities do whatever he wants, within the limit of what entities can do in your game.

The physical implementation of an attack may be as simple as running a second program on the same machine that runs the server, which injects itself into the server address space and mutates data. Or it may be as complex as a network gateway that intercepts the data packets and re-writes them outside the server machine, totally undetectable to the server process. Or it may not be using your server code at all, instead emulating it using some other program.

Don't get me wrong, though: It's important that you write your networking code to be robust. It should never trust a size field that it hasn't verified is "sane;" it should never trust a piece of data that doesn't have the right header bytes; it should be prepared to deal with reads/writes being "short." Once you do that, the impact of a sophisticated cheater is basically validation: it's a great problem to have, because it means that people care enough about your game to spend the time to do that! Once you have that problem, you can probably figure out a way to make enough money on the game to move the server onto machines you can trust.

papalazaru,
thanks for answer! I would like to use some authetification system but I need to know how hacks work. How someone can change my server's content if server is ran on his own machine? Does he somehow stick another program to my server? Or he's just sending false values to / from server via for example windows command line?

If the server runs on the users machine he can change anything he wants, (the code it runs, the values it sends/recieves, the data stored in RAM, etc), modifying software is a fairly simple process. (Pirates do it all the time to remove copyprotection from games)

With a matchmaking system your best bet is probably to allow players to rate hosts, highly rated hosts would then get matches quicker while really bad hosts could get their right to act as hosts suspended. (a host doesn't have to cheat to be bad)

Alright, I think I've made up my mind eventually, thanks to everybody who has helped here! Thanks for your time guys!
So I'll have a lobby server which will be ran on localhost on some specially designed for server machine. Do you think that for about 100$ I could buy a machine for lobby server that should handle 100 people online?
Lobby server will handle all the match making and after a match has been created and started one of players will host the game to others. Here's one more question: Should the server be implemented in the same .exe as client or it should be a separate program? or it doesn't matter at all?
And if my game occurs to be so cool that hackers will get interested then I'll think about some secure remote hosting ^^

How you separate the client and server is up to you, personally i'd put the server in its own library/class and build it into the client .exe aswell (This makes it fairly straightforward to move it out to a separate binary if you want dedicated servers later on).

The lobby server shouldn't really have to care about how many players you got online, it only has to work when players are trying to find a match. (The longer each game lasts the more players your lobby will be able to handle), $100 is pretty hard to get anything for these days, you might be able to find an old used machine at that price though. (If you buy new hardware it becomes very difficult to get a working system for less than $200-$300) (If the computer runs, is capable of running a decent server OS and can connect to the internet its good enough for a small lobby)

A Raspberry Pi is $35 and can probably be a lobby server for a thousand players. The hardware cist is not necessarily the problem.

The question is more where the machine is located. If you're thinking of running the machine on your home internet connection, there are usually lots of problems with that set-up.

$100/month lets you rent a basic online self-managed dedicated server that would be sufficient for all matchmaking needs for any typical indie game. In fact, it might be able to run the forums for your game, too :-) The rock bottom dedicated servers start at about $50/month. Cheaper can be had through "virtual private servers" which are fractions of a server, starting as low as $20/month. Make sure you get enough bandwidth for your needs, though -- if you host your own downloads, it may be expensive otherwise. A VPS is fine for a matchmaker and forums machine, but probably not for actual game server processes, because of the scheduling jitter of virtualization.