So I've got a fairly interesting project that I might have a chance to work on, and I was wondering if I could get some feedback from the great minds here :-) In a nutshell, I would need to produce a semi-interactive scene where a grid of videos (about 60 of them) would be playing concurrently ... sort of like the beginning parts of this, but with videos. Now, the videos don't have to be very high resolution, maybe webcam quality ... but when selected (maybe with the mouse), they need to kind of pop out and clear up (ie. play at a higher resolution). Just to get some stats out of the way, assume that this will run on a top of the line computer, with a high end video card, and as big of a hard drive as needed. Also, I'm planning on doing this in managed code ... the specific technology is still up in the air though.

• Getting Video onto a 3D object in WPF is ridiculously easy, but I fear that the ease of implementation may result in a lack of scalability (unverified ATM).
• Rendering a video to a texture in MDX seems to be a possible route, but MDX feels icky and dead now :-P
• I'd love to do it in XNA, but I wonder if the MDX/Interop examples out there would work with XNA on windows since there is no built in video decoding support. I could potentially just write a custom content pipeline for video files which decodes said video files and sets them up to be rendered to a texture at runtime ... but that would likely result in huge storage requirements and long IO times (which may not matter if the only issue is storage space).
• If doing it fully real-time doesn't turn out to be possible, I'm thinking that I'm going to explore some alternative methods. One possible way would be to simply render all of the above (slowly) to a video so that playback would result in a nice framerate. Then I could simply play that pre-rendered video, and when the mouse goes over a given video, just load it up, and overlay that. Doing it this way would result in only two videos playing at any given time, but it'll probably be a pain to synchronize.

I know a lot of this is still rather pie-in-the-sky ... I'm just in the research stages and trying to find out if it is even feasible to play 60 videos at the same time while rendering each one to a textured quad. Anyways, I'd appreciate any comments that I could get about all this craziness ... thanks :-)

joelmartinez,

I have worked on a similar project, although it was different in that we were limited to about four or five videos, but it was to run on a mobile device.

Now, you say you would run this on a top of the line computer, so of course you have far far more hardware to work with, yet I think 60 videos is a goal ambitious enough to where you might run into the same kind of problem as I did.

Something I ran into pretty much right off the bat was the fact that actually rendering the videos is the easy bit. ;) What takes a lot of time is decoding the video from the file, so I would suggest dedicating some time to research in this area. (Google seems to have plenty of answers for the kind of stats you would need.)

Of course the videos would necessarily be pretty small. 60 videos at a time means about a 8 x 8 grid of pictures, so on 1600 x 1200 monitor, that would be the picture size roughly of 160 x 120, if you want them on screen all at the same time. That's pretty tiny, and if the original video you are reading from is that resolution, then maybe decoding won't be that bad. You'd need to run some tests I guess.

Also, taking in all those videos all at the same time might be a bit hard. I am no HCI expert, but I would *guess* when you are looking at a grid that big, you probably can't see more than the 3x3 pictures around the point you are looking at. So maybe one curious experiment to run would be to track mouse motion, and only play the 9 videos in a square around the one the mouse is pointing at. Then, the user would follow where he's looking with the mouse, and only the videos he can take in at a time will be playing - the others would just be still images. That might look pretty cool. Of course, then, it might look like ass. ;)

well, on the decoding subject, only the "active" video is to be proper resolution ... so everything else can be pre-processed to be nice and tiny. I'm fairly certain that I'll have enough RAM in the machine to just put all of the tiny videos into memory :-P

for the HCI, I've heard the term "video wall" thrown around ... don't know much details on that just yet, but the small size of each individual clip might not be an issue.

At this point, I think my main concern is whether the video card will be able to handle the bandwidth of all those videos being sent in real-time. I think I've sort of thrown out the pre-rendered video of videos idea at this point. If the whole 60 vids at a time doesn't work out, I'm going to see if I can sell them on having each tile just be a texture, then playing the video on request.

Of course, all of this is based on uninformed guesses so far :-P I'm going to do some tests here shortly and decide a direction. I do appreciate the responses so far ... keep em comin' if you got em :-)