So I've written a simplistic 3D file format that suits my needs for now and I was wondering how I could utilize the plug-in system of a 3D modeler to export to my file format. I did a considerable amount of debating on whether this should go here or GP&T, so if it's wrong, please forgive me. Thanks.

The 3dsmax Maxscript help comes with a tutorial on writing an exporter. Likely you will write it in a scripting language or SDK for an app, its up to you, it is possible in all of them (though I woudlnt' suggest using 3dsmax... maxscript is, uh, not good).