Your use of the term "coordinate system" is somewhat confusing; even a tile-based game is going to use a coordinate system. But, as frob indicated, if what you want is a continuous coordinate space not limited to tile boundaries, then any 3D engine will do. However, Baldur's Gate and Planescape are not examples of this kind of game. BG and PS:T were Infinity Engine games. IE used fully pre-rendered backgrounds with encoded depth information, and 2D sprites super-imposed upon it. In their own way, they were every bit as limited as a tile-based game; it's just that they traded one set of limitations (grid-based-ness) for another (pre-rendered, memory-hogging gigantic bitmaps). If you're interested in actual IE game creation, you could check out GemRB for an open-source recreation of the Infinity Engine. You could implement similar using modern shader-based 3D engines as well. (I don't know the internals of GemRB, but I suspect they do something like this.)

Note that the content pipeline for an IE-like game is actually quite hefty. The tiles that you dislike in other games aren't perfect by a long shot, but they do aid in drastically reducing the amount of unique artwork that must be created, allowing for re-use of pieces to construct larger worlds or to construct levels procedurally from rule sets at run-time. IE levels are constructed whole-sale and pre-rendered in order to eliminate any appearance of tiling or repetition. This means that re-use of assets is fairly limited, and while procedural generation can be used in the pipeline for a lot of tasks, it is less useful at run-time meaning that it is more difficult to do randomly-generated areas and there is still a lot of hand-tweaking that has to be done, and a lot of hand-detailing that must be performed, in order to justify the decision to go with pre-rendered. If you have an art team, that's fine; you can put them to work churning out thousands of props and textures and composing scenes from those props. If you are a lone wolf or a very small team, the task actually becomes quite monumental. Some years back, I took a stab at generating some BG-ish scenes, and the labor involved in building the scenes to harness the advantages that pre-rendered offers (including a large amount of tiny detail) was daunting. Granted, modern tools do make things a bit easier but it is still a fairly huge task. There is a reason that art-related expenses consume the vast majority of game development budgets these days. Personally, in light of these facts, statements such as "tiling is for bathrooms, not games" simply do not belong in any serious discussion of game design, and serve only as tongue-in-cheek braggadoccio meant to hype a given project. If you don't like tiles, that's fine, but that doesn't make them useless or irrelevant as the quote seems to indicate.

Edit: Funnily enough, even the Infinity Engine uses tiles after a fashion. It's just that the "tiles" are a lot more complicated.

You may want to check out the game Phantom Brave for PS2 (there are also Wii and PSP ports of it). It's a radically different game than BG or PS:T, but it uses randomly generated terrain on which the player can freely roam (ie, not tile based). It uses 2d sprite assets in a 3D world, therefore the view is restricted to a more or less isometric view (with predefined rotations allowed).

At the very least it might give you more to think about.

Looks tile-based to me. Tile-based doesn't necessarily mean you are restricted to tile-by-tile movement; just that the world is built up from tiles, or building blocks, that are usually aligned on a grid.

True enough. I was more thinking in terms of tile (grid) based movement, that's why it popped into my head. However, the OP was clearly talking graphics, not movement, therefore I stand correct. :)