Are there any larger scale games that include the possibility to program behavior of in-game objects?
What I mean is something like http://robocode.sourceforge.net/ but with a more complex strategic gameplay.

If there is not, do you guys think there could be an audience for something like that. What I have in mind is a RTS game where it is optional to modify behavior of units.

Here is a list of some games that use programming as the core gameplay.

http://programminggames.org/

Personally if it is well written and uses good concepts then yes, but an RTS game I'm a little skeptical if that could be used, but you never know!

I am developing a browser-based online RPG where the characters are 'always online' and can be scripted to perform tasks while the player is offline.

Are there any larger scale games that include the possibility to program behavior of in-game objects?
What I mean is something like http://robocode.sourceforge.net/ but with a more complex strategic gameplay.

If there is not, do you guys think there could be an audience for something like that. What I have in mind is a RTS game where it is optional to modify behavior of units.

I can't imagine a case where one has the time to program behavior for units, at least in multiplayer. Depends on how you define RTS I suppose. I know there are some RTS games that allow limited control of unit behavior with things like toggles and what not.

Could you be more specific as to the capabilities you are looking at?

If you hide the actual programming stuff like zachtronics has in space chem: http://www.spacechemthegame.com/ and Even optimization score boards. To be honest this is probably one of my favorite programming games just because of how well it hides the fact that it is one.

I was not thinking of a classical RTS like starcraft but rather something more tactical. Actually it would be probably closer to an RPG than an RTS game. Maybe the last sentence in the original post was a bit misleading.

So you are controlling a guy which is some kind of hacker and you can build/upgrade robots which follow you (or not, depending on how you program them . You find resources to upgrade/specialize the robots etc. There could be both single player missions and multiplayer skirmishes. Of course you would not code during the match but have a set of programs you can plug into the robots.

To make it more accessible I had the idea to allow ingame trading of code (closed source) between players and maybe add ingame repositories for open source projects.

Well I guess I should start working on the game and get a prototype going.

One thing i was wondering is, if it is a better idea to implement my own very simple scripting language, (I was thinking of something like glsl, where you have access to some predefined variables, but only allowing a main function) or just go with something like lua (which i have never used).

While not exactly what you're looking for, Final Fantasy XII, and the Dragon Age series both have similar systems that allow you to script your party members' behavior when in combat. I'd say those are beyond niche titles.

The interface is pretty simple (and is pretty much identical for both games), where you essentially build a big "if, else if, else if, else if..." block using pre-defined conditions and actions. This makes it more accessible than having to code the behavior, but limits what the end user can do.

The final fantasy system was dubbed the "Gambit system" if you wanted to look it up. The Dragon Age system wasn't named but there weren't many differences between the two.

Perhaps a similar system with the ability to script your own conditions might make it accessible enough, but interesting enough for programmers to poke around.