LockRect and Video RGB

Graphics and GPU Programming Programming

Started by Shael February 13, 2008 09:29 PM

6 comments, last by Shael 16 years, 2 months ago

286

Author

February 13, 2008 09:29 PM

Hi all, I'm having some trouble getting video to render to a texture in my engine. I'm using ffmpeg to open the video file then using it to convert the frames into RGB format. I'm then trying to put that data into a DX created texture. I can't seem to get it to work properly at all. Heres some of the code I currently have:



...

pSWSCtx = sws_getContext(pCodecCtx->width, pCodecCtx->height, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height, PIX_FMT_RGB24, SWS_FAST_BILINEAR, NULL, NULL, NULL);

if ( FAILED(D3DXCreateTexture(CDXMgr::Get().GetD3DDevice(), pCodecCtx->width, pCodecCtx->height, 1, D3DUSAGE_DYNAMIC, D3DFMT_R8G8B8, D3DPOOL_DEFAULT, &g_pTexture)))
return false;

...

// Convert the image from its native format to RGB
sws_scale(pSWSCtx, pFrame->data, pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);


D3DLOCKED_RECT lockedRect;
HRESULT hResult = g_pTexture->LockRect(0, &lockedRect, NULL, D3DLOCK_DISCARD);
if(FAILED(hResult))
	return false;

unsigned int uHeight = pCodecCtx->height;
unsigned int uWidth = pCodecCtx->width;
unsigned char *pTextureMemory = static_cast<unsigned char*>(lockedRect.pBits);

for( unsigned int uRow = 0; uRow < uHeight; uRow++ )
{
	unsigned int uRowStart = uRow * lockedRect.Pitch;
	for( unsigned int uCol = 0; uCol < uWidth; uCol++ )
	{
		unsigned int uTexelStart = uRowStart + (uCol*3);
		pTextureMemory[uTexelStart + 0] = pFrame->data[0][uRow*uWidth + uCol*3 +0]; // Red
		pTextureMemory[uTexelStart + 1] = pFrame->data[1][uRow*uWidth + uCol*3 +1]; // Green
		pTextureMemory[uTexelStart + 2] = pFrame->data[2][uRow*uWidth + uCol*3 +2]; // Blue
	}
}
g_pTexture->UnlockRect(0);

Any help is appreciated, thanks.

2EZ Studios

Games4TheFuture

121

February 13, 2008 10:24 PM

If you're trying to use a video for a texture, I would use DirectShow.

-games4thefuture-

Namethatnobodyelsetook

1,260

February 13, 2008 10:31 PM

Nearly no cards currently in existance support R8G8B8. CreateTexture in the device will fail if you don't specify exactly valid values. D3DXCreateTexture on the other hand is a wrapper around that, which tries to manipulate your request into something that will succeed. It's probably making an X8R8G8B8 texture, but the only way to be sure is to ask D3D for the surface descriptor once the texture is made.

Also, because you don't know what format you're actually getting, "unsigned int uTexelStart = uRowStart + (uCol*3);" is not correct. The *3 will vary depending on the texture actually made. Since all cards support X8R8G8B8, it's best to just ask for that format directly, and change the *3 to *4. I think you'll also need to change the write order to blue, green, red, as Intel chips store the lowest byte first so the DWORD XXRRGGBB is stored as BB, GG, RR, XX in memory. You're using Pitch correctly.

If you like tons of overkill and want to do what's technically necessary to be 100% right all the time (remember you can't technically assume you'll get X8R8G8B8), you'll need to handle conversions to any format, of whatever bitdepth and byte order D3D happens to give you. I'd just assume X8R8G8B8 is valid though.

Shael

286

Author

February 13, 2008 11:41 PM

Thanks for the replies. I dont want to use DirectShow because its seems so confusing and is apparantly not very good for use in a game engine.

Namethatnobodyelsetook:

I'm starting to think half of my problem is how to get the data from the ffmpeg frame and put it into a texture. I found an example of how to save frames as .PPM format. Here's how they do it:

void SaveFrame(AVFrame *pFrame, int width, int height, int iFrame) {  FILE *pFile;  char szFilename[32];  int  y;    // Open file  sprintf(szFilename, "frame%d.ppm", iFrame);  pFile=fopen(szFilename, "wb");  if(pFile==NULL)    return;    // Write header  fprintf(pFile, "P6\n%d %d\n255\n", width, height);    // Write pixel data  for(y=0; y<height; y++)    fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, width*3, pFile);    // Close file  fclose(pFile);}

The output of that is fine and loads in Photoshop. So how would I use this way of accessing the data and copy it into my DirectX texture?

Cheers!

2EZ Studios

sirlemonhead

227

February 14, 2008 05:20 AM

You could do the reverse of the write pixel data loops to read back the information and write it into a d3d texture (using LockRect)

but loading each frame from a file is a bad way of doing it..is there any way to read frame by frame from the video file directly?

Shael

286

Author

February 14, 2008 07:25 AM

Yeah you can read it frame by frame, thats what I'm doing at the moment. But my problem is, getting the image data of the frame and putting it into a DirectX texture. Using ffmpeg I can convert the video format from the video format YUV into RGB, which I've tested and works because that SaveFrame function I posted spits out valid .PPM files which contain RGB data that I can open in photoshop.

So my problem is, how do I read the RGB data of the frame correctly, and how do I copy it into a DirectX texture correctly?

I've been messing around with it for over a day now and can't seem to work it out. Maybe someone who has knowledge with LockRect and image formats could come save me from this torment hehe.

Cheers. :)

2EZ Studios

sirlemonhead

227

February 14, 2008 08:08 AM

I'm at work at the moment but i'll throw up some code when I get home if i remember to do it..

how do you get the frame data back from the YUV -> RGB conver function? just raw data stored somewhere?

Shael

286

Author

February 14, 2008 08:30 AM

In my original post theres a function called "sws_scale" which takes the original video frame that is in whatever format, usually YUV I think, and then converts it into another frame with a format I specify. As you can see I specified PIX_FMT_RGB24 as my format to convert to. Which according to the comments in the ffmpeg header it is this.. "// Packed RGB 8:8:8, 24bpp, RGBRGB..."

The raw RGB data is stored in pFrameRGB->data and.. pFrameRGB->linesize I think is the "pitch". Theres not much documentation unfortunately, only this shoddy doxygen:

http://cekirdek.pardus.org.tr/~ismail/ffmpeg-docs/structAVPicture.html#92fce3ff225b920a33c1ad866f4008cb

Btw I've tried doing a similar approach to the SaveFrame function but it doesnt seem to produce the right results.. just a random garbled image..

BYTE* pTextureData = (BYTE *)lockedRect.pBits;for(int y=0; y < pCodecCtx->height; y++)memcpy(pTextureData + y*lockedRect.Pitch, pFrameRGB->data[0]+y*pFrameRGB->linesize[0], pCodecCtx->width*4);

EDIT:

I've managed to get it working using the code above. It took a bit but I had to go through and get all the formats from ffmpeg matching the texture formats of directx. My only problem now is the image is upside down heh.

Cheers for your help guys.

[Edited by - Shael on February 14, 2008 8:30:52 PM]

2EZ Studios

LockRect and Video RGB

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

LockRect and Video RGB

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines