My two cents: I honestly can't imagine making a better bespoke build system than what already exists. In my experience, a build system is to be leveraged and extended, but not replaced -- that is, when you feel limited by it, its usually because you don't fully understand how to use or extend it with custom steps.
But its absolutely the norm that a complex project *does* extend its build system, it won't every last thing you need out of the box. And they're built to be extended with custom steps -- there are hooks all over to allow you to interject into the build process.
But fully custom, or even heavily modified (beyond easy recognition) or just plain complicated builds, are to be avoided. I had a gig once where my job was to perform static analysis of Xbox 360 titles and report back where there were dangerous practices or performance anti-patterns, so we'd get projects from various AAA studios. Once there was one with a very large and complicated build system. The normal process was that there was a Wisual Studio project and it was just good to go, but this one with the complicated build system was crazy -- in the end, what they ended up doing was having their build engineers create us a Hyper-V image on a physical hard drive and literally shipping it to us overnight. Even then, we had one of their build engineers on-call who would remote into the virtual machine to fix things when our tools changed how things worked (which was usually not an issue) -- they also were doing things that broke our tools, like issuing *single compiler commands* that spanned larger than 4096 characters in length (which exposed an incorrectly handled buffer copy in our tool, and took us a week to figure out).
Long story short, bespoke systems are usually brittle and often complicated, almost by definition. Existing build systems do a good job at the job they're meant to handle, otherwise they would not be so widely spread. Attempting a bespoke system on your own means you have to do all that mundane stuff just as well as the others before you might even get to gain any ground with your unique vision -- unless its a requirement, you're probably better of extending one of the existing systems with what you need.