My experience with video manipulation is limited.
Are you targeting windows? Or a specific OS?

You would think there would be a ton of libraries that let you pull video into an application

There are quite a few.

(how complex can it be?).

Very complex, actually. Videos come in all sorts of pixel formats, container formats, video codecs, and audio codecs.

I just need to stream a video file one frame at a time, convert each frame to a directx/opengl texture, run my shader on it, then write it back out to a video file. But I am having trouble finding projects that will allow me to do all that...

You could use FFmpeg do to this. It's LGPL and cross platform. It's got a bit of a learning curve, to be honest, but you could just write a wrapper around it and use your simplified wrapper. I wrote a blog post once about using FFmpeg to decode video in a Visual Studio 2010 project. I should probably write a couple follow up posts going into more detail and a post about how to encode video too. Anyway, check it out and let me know if there's any questions you've got that I might be able to help you with.