codecProcess

• Opening a Video File

　　First things first - let's look at how to open a video file and get at the streams contained in it. The first thing we need to do is to initialize libavformat/libavcodec:  

 av_register_all(); 

　　This registers all available file formats and codecs with the library so they will be used automatically when a file with the corresponding format/codec is opened. Note that you only need to call av_register_all() once, so it's probably best to do this somewhere in your startup code. If you like, it's possible to register only certain individual file formats and codecs, but there's usually no reason why you would have to do that.  
　　Next off, opening the file:  

AVFormatContext *pFormatCtx; 
const char  *filename="myvideo.mpg"; // Open video file 
if(av_open_input_file(&pFormatCtx, filename, NULL, 0, NULL)!=0)     
    handle_error(); //Couldn't open file

　　The last three parameters specify the file format, buffer size and format parameters; by simply specifying NULL or 0 we ask libavformat to auto-detect the format and use a default buffer size. Replace handle_error() with appropriate error handling code for your application.  
　　Next, we need to retrieve information about the streams contained in the file:  // Retrieve stream information 

 if(av_find_stream_info(pFormatCtx)<0) handle_error();// Couldn't find stream information  

　　This fills the streams field of the AVFormatContext with valid information. As a debugging aid, we'll dump this information onto standard error, but of course you don't have to do this in a production application: 

 dump_format(pFormatCtx, 0, filename, false);  

　　As mentioned in the introduction, we'll handle only video streams, not audio streams. To make things nice and easy, we simply use the first video stream we find:  

int i, videoStream; 
AVCodecContext *pCodecCtx; // Find the first video stream 
videoStream=-1; 
for(i=0; i<pFormatCtx->nb_streams; i++) {
    if(pFormatCtx->streams[i]->codec.codec_type==CODEC_TYPE_VIDEO) { 
        videoStream=i;         
        break;     
    } 
    if(videoStream==-1) 
    handle_error(); // Didn't find a video stream 
}    // Get a pointer to the codec context for the video stream pCodecCtx=&pFormatCtx->streams[videoStream]->codec; 

　　OK, so now we've got a pointer to the so-called codec context for our video stream, but we still have to find the actual codec and open it:  AVCodec *pCodec; 

// Find the decoder for the video stream 
pCodec=avcodec_find_decoder(pCodecCtx->codec_id); 
if(pCodec==NULL) 
    handle_error(); // Codec not found 

　　Inform the codec that we can handle truncated bitstreams -- i.e., bitstreams where frame boundaries can fall in the middle of packets 

if(pCodec->capabilities & CODEC_CAP_TRUNCATED)     
    pCodecCtx->flags|=CODEC_FLAG_TRUNCATED; // Open codec 
if(avcodec_open(pCodecCtx, pCodec)<0) 
    handle_error(); // Could not open codec

　　So what's up with those "truncated bitstreams"? Well, as we'll see in a moment, the data in a video stream is split up into packets. Since the amount of data per video frame can vary, the boundary between two video frames need not coincide with a packet boundary. Here, we're telling the codec that we can handle this situation.
　　One important piece of information that is stored in the AVCodecContext structure is the frame rate of the video. To allow for non-integer frame rates (like NTSC's 29.97 fps), the rate is stored as a fraction, with the numerator in pCodecCtx->frame_rate and the denominator in pCodecCtx->frame_rate_base. While testing the library with different video files, I noticed that some codecs (notably ASF) seem to fill these fields incorrectly (frame_rate_base contains 1 instead of 1000). The following hack fixes this:  

// Hack to correct wrong frame rates that seem to be generated by some  // codecs 
if(pCodecCtx->frame_rate>1000 && pCodecCtx->frame_rate_base==1)     
    pCodecCtx->frame_rate_base=1000; 

　　Note that it shouldn't be a problem to leave this fix in place even if the bug is corrected some day - it's unlikely that a video would have a frame rate of more than 1000 fps.  
　　One more thing left to do: 
　　Allocate a video frame to store the decoded images in:  

 AVFrame *pFrame;  2 pFrame=avcodec_alloc_frame();  

　　That's it! Now let's start decoding some video.

 

• Decoding Video Frames

 

　　As I've already mentioned, a video file can contain several audio and video streams, and each of those streams is split up into packets of a particular size. Our job is to read these packets one by one using libavformat, filter out all those that aren't part of the video stream we're interested in, and hand them on to libavcodec for decoding. In doing this, we'll have to take care of the fact that the boundary between two frames can occur in the middle of a packet.  
　　Sound complicated? Lucikly, we can encapsulate this whole process in a routine that simply returns the next video frame:  

bool GetNextFrame(AVFormatContext *pFormatCtx, AVCodecContext *pCodecCtx, int videoStream, AVFrame *pFrame) { 
    
    static AVPacket packet; 
    static int      bytesRemaining=0;     
    static uint8_t  *rawData; 
    static bool     fFirstTime=true;
    int bytesDecoded;     
    int frameFinished; 
    // First time we're called, set packet.data to NULL to indicate it     
    // doesn't have to be freed     
    if(fFirstTime) { 
        fFirstTime=false;         
        packet.data=NULL;     
    }
     
    // Decode packets until we have decoded a complete frame     
    while(true) { 
    // Work on the current packet until we have decoded all of it         
        while(bytesRemaining > 0) { 
            // Decode the next chunk of data 
            bytesDecoded = avcodec_decode_video(pCodecCtx, pFrame, &frameFinished, rawData, bytesRemaining);             
            // Was there an error?             
            if(bytesDecoded < 0) { 
                fprintf(stderr, "Error while decoding frame\n");                 
                return false;             
            } 

            bytesRemaining-=bytesDecoded;             
            rawData+=bytesDecoded; 
            // Did we finish the current frame? Then we can return             
            if(frameFinished)                 
                return true;         
        } 

        do { // Read the next packet, skipping all packets that aren't for this         
            // stream                                  
            // Free old packet             
            if(packet.data!=NULL) 
                av_free_packet(&packet);             // Read new packet 

            if(av_read_packet(pFormatCtx, &packet)<0)                 
                goto loop_exit; 
        } while(packet.stream_index!=videoStream);  
       
        bytesRemaining=packet.size;         
        rawData=packet.data;     
    } 

loop_exit: 
    // Decode the rest of the last frame 
    bytesDecoded=avcodec_decode_video(pCodecCtx, pFrame, &frameFinished, rawData, bytesRemaining);     // Free last packet     

    if(packet.data!=NULL) 
        av_free_packet(&packet);     
    return frameFinished!=0; 
} 

　　Now, all we have to do is sit in a loop, calling GetNextFrame() until it returns false. Just one more thing to take care of: Most codecs return images in YUV 420 format (one luminance and two chrominance channels, with the chrominance channels samples at half the spatial resolution of the luminance channel). Depending on what you want to do with the video data, you may want to convert this to RGB. (Note, though, that this is not necessary if all you want to do is display the video data; take a look at the X11 Xvideo extension, which does YUV-to-RGB and scaling in hardware.) Fortunately, libavcodec provides a conversion routine called img_convert, which does conversion between YUV and RGB as well as a variety of other image formats. The loop that decodes the video thus becomes:  

while(GetNextFrame(pFormatCtx, pCodecCtx, videoStream, pFrame)){ 
    img_convert((AVPicture *)pFrameRGB, PIX_FMT_RGB24, (AVPicture*)pFrame,          
    pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height);     
    // Process the video frame (save to disk etc.)     
    DoSomethingWithTheImage(pFrameRGB); 
}

　　The RGB image pFrameRGB (of type AVFrame *) is allocated like this:  

 AVFrame *pFrameRGB;  2 int numBytes;  3 uint8_t *buffer;  

　　Allocate an AVFrame structure pFrameRGB=avcodec_alloc_frame(); 

if(pFrameRGB==NULL) 
    handle_error(); 
// Determine required buffer size and allocate buffer 
numBytes=avpicture_get_size(PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height); 
buffer=new uint8_t[numBytes]; 

　　Assign appropriate parts of buffer to image planes in:

 pFrameRGB avpicture_fill((AVPicture *)pFrameRGB, buffer, PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height); 

 

 

• Cleaning up 

　　OK, we've read and processed our video, now all that's left for us to do is clean up after ourselves:  

// Free the RGB image delete [] buffer; 
av_free(pFrameRGB); 
// Free the YUV frame av_free(pFrame); 
// Close the codec 
avcodec_close(pCodecCtx); 
// Close the video file 
av_close_input_file(pFormatCtx); 

　　Done!