Wouldn't the size problem be reduced if you avoid putting continuously updated precompiled binary files into git/mercurial and dont add all kinds of external stuff you depend on into a single repository but instead have some sub-repositories?
You could also split it up to put the source art files, the compiled art files, the compiled code into 3 svn repositories, to reduce the load in the dvcs further by only having source code inside. Or add some custom merge tool for some binary files or convert them when storing into the repository to make them mergeable.
Maybe on bigger projects people have some buildserver already that compiles everything for testing commits, that could also update the separate svn repositories for compiled things, for the artists to avoid having to compile code and the programmers avoiding to convert art files themself?
Compiled binary files have no reason to be in SOURCE control. Important ones, such as public packages of release or beta test versions, should be archived to Dropbox, network shared drives, or other types of "dumb" storage by build managers to let others download them as needed.
Old compiled binary files, as opposed to compiling the game by themselves, are needed by developers only for unusual reference purposes (e.g. testing that building the appropriate revision of the sources reproduces what's been released), not to be routinely updated, compared, and be copied to or from their personal workspaces like sources.
Correctly managed binary source assets are unlikely to be troublesome: they should be available to everybody (enabling all developers to build the game), which is a good reason to make all revisions easily available, and they should change only rarely and in meaningful fine-grained increments (for example, repainting a 3D model should change that model's texture maps, not a big texture atlas containing lots of unrelated images; the big texture atlas can be kept outside source control and rebuilt automatically).
A sane organization of assets and build tools is an opportunity, not a cost incurred because of source control; setting up an easy, automated and effective workflow using source control should be compared to skipping the initial effort, sinking into progressive complication and confusion, and throwing the towel (or wasting a lot of time) because of errors in manual builds.