The first "commercial game engine" that comes to my mind that supported multiple platforms and used by several AAA titles that remotely gets close to the modern concept of game engines was RenderWare. It wasn't even a game engine, it was a rendering engine. And it wasn't for PS1 / N64 gen.
Licensing it costed several tens of thousands of dollars AFAIK. Wikipedia has a list of its competitors (Unreal Engine & Frostbite). Doesn't matter who came first, none of them were for the era you're looking for because, like others already explained, it was all handcrafted and kept in house; occasionally having their stuff licensed to other studios.