and one another thing i hardly can imagine 800 MB for animations consider 3 joints per finger * 5 fingers * 2 + 2 joints per foot + 2*anckle * 4 for legs + 8 others gives us
48 matrices per frame * 16*4 bytes = 3 Kilobytes per frame, considering that:
hitting ~800 MB of VRAM due to animations.
this sounds like gamemaker exporst each frame to different model, instead of using one vbuff for one character. or maybe you did the animtion wrong, there is a way to export triangle set to tri_strip, but i wonder how are you supposed to go through vertice data, if this program stores animation set in 800 MB seriously
i may calculated that wrong but it seems: 2.53 hours of pure animation at 30 fps. for one vertex buffer set.
anyway to the answer...
how exactly it would work i don't know but finding shared vertices between triangles and then converting these shared ones (you group by 2 triangles, and convert that to tri_strip) next would be to find shared vertice of tri strip quad but i mean ' 'the' 'next' ' shared vertice i reckon for first quad tri strip is 4th vert and for second quad tri strip is first vert.
so first step group shared verts and connect by 2 triangles , then you will have almost everywhere quads made out of triangle strips,, you could find shared verts there and connect them accordingly. but i wish you could say how did you do the animation on this game maker. coz it seems pretty unreasonable,
but still if you have way too much gemoetry you could consider using dynamic level of detail, or at least precompute models to lower LODs, or make lower polygon models,
anyway if you plan really to release the game that uses such amount of vram onyl to draw a menu, then so be it, i have seen way too much unity games that were drawing 2 quads and it ran at 1 fps... so why would you care?