I look for some sort of robotic race for a 4X game. I want them to be at least a bit unusual, they should be hostile and dangerous, but they can't have a goal of galactic conquest. Generally, they could/should have some weird patterns of behaviour (like not taking some planets, being limited to a part of space, etc).

I think insects (real ones) will help you. For an unusual robotic appearance, just Google "unusual insects" images. For unusual behavior that is hostile and dangerous, stay with the insects. By that I mean consider what motivates an insect - search for specific resources (especially food/fuel), things needed for reproduction (certain kind of land, plants, minerals, etc), shelter/defense/compulsion to protect territory against intrusions.

Since these are robots, you could also consider another motive - a damaged or warped compulsion to "repair" everything... and humans break a lot of stuff, so....

What if the robots were low on resources that were used to build them.

Their only way of reproducing is to craft more of themselves, but since they are low on resource x, this is unfeasible, and in the last 100 years they have been using all of their last resources to sustain themselves. They have a lot of weaponry that in the past they used primarily for defense, but they are now motivated to go and use it to acquire those resources wherever they can in order to ensure their racial survival.

Unfortunately, what they need is an ore, which happens to exist most plentifully beneath the major cities on planets of some other races.

Then let's say that the resource is extraordinarily heavy and can only be moved short distances unless broken down which takes a ton of time and energy.

So every time they run out (last time they peacefully took this current planet, which had been empty) they up and move to the new place (and currently in full military force).

So why can't they negotiate at all? Sure they need something the natives of the cities can't give up, but why no communication?

Simple. They can't talk. Their form of communication in neural and only on a frequency that one another can pick up, so they would need to learn another language from scratch and still then be mute.

Strategically: From a native (human or otherwise) perspective, these robots are attacking without proper declarations of war (think pearl harbor, ignoring the attempt to send one that failed-- more the backlash) and hitting target cities/planets where there are loads of civilians but do not appear to have massive strategic value.

(The humans don't know about the ore).

This would give them an odd strike pattern and seemingly unintelligent migratory patterns that have perfectly logical yet incommunicable reasons that would alter gameplay accordingly.

So what you are not looking for is The Borg, but an interesting way to go would be this. Hostile robots who only favor alien races and not humans and vice versa. Then there could be robots fighting robots for various purposes. Another would be robots with strict moral codes for others, and they become aggressive when they see people breaking the rules. I will get back to you on this, as it is interesting.