FFPLAY的原理（五）
创建线程
Spawning Threads
Overview
Last time we added audio support by taking advantage of SDL's audio functions. SDL started a thread that made callbacks to a function we defined every time it needed audio. Now we're going to do the same sort of thing with the video display. This makes the code more modular and easier to work with - especially when we want to add syncing. So where do we start?
First we notice that our main function is handling an awful lot: it's running through the event loop, reading in packets, and decoding the video. So what we're going to do is split all those apart: we're going to have a thread that will be responsible for decoding the packets; these packets will then be added to the queue and read by the corresponding audio and video threads. The audio thread we have already set up the way we want it; the video thread will be a little more complicated since we have to display the video ourselves. We will add the actual display code to the main loop. But instead of just displaying video every time we loop, we will integrate the video display into the event loop. The idea is to decode the video, save the resulting frame in another queue, then create a custom event (FF_REFRESH_EVENT) that we add to the event system, then when our event loop sees this event, it will display the next frame in the queue. Here's a handy ASCII art illustration of what is going on:
________ audio _______ _____
| | pkts | | | | to spkr
| DECODE |----->| AUDIO |--->| SDL |-->
|________| |_______| |_____|
| video _______
| pkts | |
+---------->| VIDEO |
________ |_______| _______
| | | | |
| EVENT | +------>| VIDEO | to mon.
| LOOP |----------------->| DISP. |-->
|_______|<---FF_REFRESH----|_______|
The main purpose of moving controlling the video display via the event loop is that using an SDL_Delay thread, we can control exactly when the next video frame shows up on the screen. When we finally sync the video in the next tutorial, it will be a simple matter to add the code that will schedule the next video refresh so the right picture is being shown on the screen at the right time.
Simplifying Code
We're also going to clean up the code a bit. We have all this audio and video codec information, and we're going to be adding queues and buffers and who knows what else. All this stuff is for one logical unit, viz. the movie. So we're going to make a large struct that will hold all that information called the VideoState.
1 typedef struct VideoState { 2 AVFormatContext *pFormatCtx; 3 int videoStream, audioStream; 4 AVStream *audio_st; 5 PacketQueue audioq; 6 uint8_t audio_buf[(AVCODEC_MAX_AUDIO_FRAME_SIZE * 3) / 2]; 7 unsigned int audio_buf_size; 8 unsigned int audio_buf_index; 9 AVPacket audio_pkt; 10 uint8_t *audio_pkt_data; 11 int audio_pkt_size; 12 AVStream *video_st; 13 PacketQueue videoq; 14 VideoPicture pictq[VIDEO_PICTURE_QUEUE_SIZE]; 15 int pictq_size, pictq_rindex, pictq_windex; 16 SDL_mutex *pictq_mutex; 17 SDL_cond *pictq_cond; 18 SDL_Thread *parse_tid; 19 SDL_Thread *video_tid; 20 char filename[1024]; 21 int quit; 22 } VideoState;
Here we see a glimpse of what we're going to get to. First we see the basic information - the format context and the indices of the audio and video stream, and the corresponding AVStream objects. Then we can see that we've moved some of those audio buffers into this structure. These (audio_buf, audio_buf_size, etc.) were all for information about audio that was still lying around (or the lack thereof). We've added another queue for the video, and a buffer (which will be used as a queue; we don't need any fancy queueing stuff for this) for the decoded frames (saved as an overlay). The VideoPicture struct is of our own creations (we'll see what's in it when we come to it). We also notice that we've allocated pointers for the two extra threads we will create, and the quit flag and the filename of the movie.
So now we take it all the way back to the main function to see how this changes our program. Let's set up our VideoState struct:
1 int main(int argc, char *argv[]) { 2 SDL_Event event; 3 VideoState *is; 4 is = av_mallocz(sizeof(VideoState));
av_mallocz() is a nice function that will allocate memory for us and zero it out.
Then we'll initialize our locks for the display buffer (pictq), because since the event loop calls our display function - the display function, remember, will be pulling pre-decoded frames from pictq. At the same time, our video decoder will be putting information into it - we don't know who will get there first. Hopefully you recognize that this is a classic race condition. So we allocate it now before we start any threads. Let's also copy the filename of our movie into our VideoState.
pstrcpy(is->filename, sizeof(is->filename), argv[1]);
is->pictq_mutex = SDL_CreateMutex();
is->pictq_cond = SDL_CreateCond();
pstrcpy is a function from ffmpeg that does some extra bounds checking beyond strncpy.
Our First Thread
Now let's finally launch our threads and get the real work done:
schedule_refresh(is, 40);
is->parse_tid = SDL_CreateThread(decode_thread, is);
if(!is->parse_tid) {
av_free(is);
return -1;
}
schedule_refresh is a function we will define later. What it basically does is tell the system to push a FF_REFRESH_EVENT after the specified number of milliseconds. This will in turn call the video refresh function when we see it in the event queue. But for now, let's look at SDL_CreateThread().
SDL_CreateThread() does just that - it spawns a new thread that has complete access to all the memory of the original process, and starts the thread running on the function we give it. It will also pass that function user-defined data. In this case, we're calling decode_thread() and with our VideoState struct attached. The first half of the function has nothing new; it simply does the work of opening the file and finding the index of the audio and video streams. The only thing we do different is save the format context in our big struct. After we've found our stream indices, we call another function that we will define, stream_component_open(). This is a pretty natural way to split things up, and since we do a lot of similar things to set up the video and audio codec, we reuse some code by making this a function.
The stream_component_open() function is where we will find our codec decoder, set up our audio options, save important information to our big struct, and launch our audio and video threads. This is where we would also insert other options, such as forcing the codec instead of autodetecting it and so forth. Here it is:
1 int stream_component_open(VideoState *is, int stream_index) { 2 AVFormatContext *pFormatCtx = is->pFormatCtx; 3 AVCodecContext *codecCtx; 4 AVCodec *codec; 5 SDL_AudioSpec wanted_spec, spec; 6 if(stream_index < 0 || stream_index >= pFormatCtx->nb_streams) { 7 return -1; 8 } 9 // Get a pointer to the codec context for the video stream 10 codecCtx = pFormatCtx->streams[stream_index]->codec; 11 if(codecCtx->codec_type == CODEC_TYPE_AUDIO) { 12 // Set audio settings from codec info 13 wanted_spec.freq = codecCtx->sample_rate; 14 /* .... */ 15 wanted_spec.callback = audio_callback; 16 wanted_spec.userdata = is; 17 if(SDL_OpenAudio(&wanted_spec, &spec) < 0) { 18 fprintf(stderr, "SDL_OpenAudio: %s\n", SDL_GetError()); 19 return -1; 20 } 21 } 22 codec = avcodec_find_decoder(codecCtx->codec_id); 23 if(!codec || (avcodec_open(codecCtx, codec) < 0)) { 24 fprintf(stderr, "Unsupported codec!\n"); 25 return -1; 26 } 27 switch(codecCtx->codec_type) { 28 case CODEC_TYPE_AUDIO: 29 is->audioStream = stream_index; 30 is->audio_st = pFormatCtx->streams[stream_index]; 31 is->audio_buf_size = 0; 32 is->audio_buf_index = 0; 33 memset(&is->audio_pkt, 0, sizeof(is->audio_pkt)); 34 packet_queue_init(&is->audioq); 35 SDL_PauseAudio(0); 36 break; 37 case CODEC_TYPE_VIDEO: 38 is->videoStream = stream_index; 39 is->video_st = pFormatCtx->streams[stream_index]; 40 packet_queue_init(&is->videoq); 41 is->video_tid = SDL_CreateThread(video_thread, is); 42 break; 43 default: 44 break; 45 } 46 }This is pretty much the same as the code we had before, except now it's generalized for audio and video. Notice that instead of aCodecCtx, we've set up our big struct as the userdata for our audio callback. We've also saved the streams themselves as audio_st and video_st. We also have added our video queue and set it up in the same way we set up our audio queue. Most of the point is to launch the video and audio threads. These bits do it: 47 SDL_PauseAudio(0); 48 break; 49 /* ...... */ 50 is->video_tid = SDL_CreateThread(video_thread, is); 51 We remember SDL_PauseAudio() from last time, and SDL_CreateThread() is used as in the exact same way as before. We'll get back to our video_thread() function. 52 Before that, let's go back to the second half of our decode_thread() function. It's basically just a for loop that will read in a packet and put it on the right queue: 53 for(;;) { 54 if(is->quit) { 55 break; 56 } 57 // seek stuff goes here 58 if(is->audioq.size > MAX_AUDIOQ_SIZE || 59 is->videoq.size > MAX_VIDEOQ_SIZE) { 60 SDL_Delay(10); 61 continue; 62 } 63 if(av_read_frame(is->pFormatCtx, packet) < 0) { 64 if(url_ferror(&pFormatCtx->pb) == 0) { 65 SDL_Delay(100); /* no error; wait for user input */ 66 continue; 67 } else { 68 break; 69 } 70 } 71 // Is this a packet from the video stream? 72 if(packet->stream_index == is->videoStream) { 73 packet_queue_put(&is->videoq, packet); 74 } else if(packet->stream_index == is->audioStream) { 75 packet_queue_put(&is->audioq, packet); 76 } else { 77 av_free_packet(packet); 78 } 79 }
这里没有什么新东西，除了我们给音频和视频队列限定了一个最大值并且我们添加一个检测读错误的函数。格式上下文里面有一个叫做pb的 ByteIOContext类型结构体。这个结构体是用来保存一些低级的文件信息。函数url_ferror用来检测结构体并发现是否有些读取文件错误。
在循环以后，我们的代码是用等待其余的程序结束和提示我们已经结束的。这些代码是有益的，因为它指示出了如何驱动事件－－后面我们将显示影像。
1 while(!is->quit) { 2 SDL_Delay(100); 3 } 4 fail: 5 if(1){ 6 SDL_Event event; 7 event.type = FF_QUIT_EVENT; 8 event.user.data1 = is; 9 SDL_PushEvent(&event); 10 } 11 return 0;
我们使用SDL常量SDL_USEREVENT来从用户事件中得到值。第一个用户事件的值应当是SDL_USEREVENT，下一个是 SDL_USEREVENT＋1并且依此类推。在我们的程序中FF_QUIT_EVENT被定义成SDL_USEREVENT＋2。如果喜欢，我们也可以 传递用户数据，在这里我们传递的是大结构体的指针。最后我们调用SDL_PushEvent()函数。在我们的事件分支中，我们只是像以前放入 SDL_QUIT_EVENT部分一样。我们将在自己的事件队列中详细讨论，现在只是确保我们正确放入了FF_QUIT_EVENT事件，我们将在后面捕 捉到它并且设置我们的退出标志quit。
得到帧：video_thread
当我们准备好解码器后，我们开始视频线程。这个线程从视频队列中读取包，把它解码成视频帧，然后调用queue_picture函数把处理好的帧放入到图片队列中:
1 int video_thread(void *arg) { 2 VideoState *is = (VideoState *)arg; 3 AVPacket pkt1, *packet = &pkt1; 4 int len1, frameFinished; 5 AVFrame *pFrame; 6 pFrame = avcodec_alloc_frame(); 7 for(;;) { 8 if(packet_queue_get(&is->videoq, packet, 1) < 0) { 9 // means we quit getting packets 10 break; 11 } 12 // Decode video frame 13 len1 = avcodec_decode_video(is->video_st->codec, pFrame, &frameFinished, 14 packet->data, packet->size); 15 // Did we get a video frame? 16 if(frameFinished) { 17 if(queue_picture(is, pFrame) < 0) { 18 break; 19 } 20 } 21 av_free_packet(packet); 22 } 23 av_free(pFrame); 24 return 0; 25 }
在这里的很多函数应该很熟悉吧。我们把avcodec_decode_video函数移到了这里，替换了一些参数，例如：我们把AVStream保存在我 们自己的大结构体中，所以我们可以从那里得到编解码器的信息。我们仅仅是不断的从视频队列中取包一直到有人告诉我们要停止或者出错为止。
把帧队列化
让我们看一下保存解码后的帧pFrame到图像队列中去的函数。因为我们的图像队列是SDL的覆盖的集合（基本上不用让视频显示函数再做计算了），我们需要把帧转换成相应的格式。我们保存到图像队列中的数据是我们自己做的一个结构体。
1 typedef struct VideoPicture { 2 SDL_Overlay *bmp; 3 int width, height; 4 int allocated; 5 } VideoPicture;
我们的大结构体有一个可以保存这些缓冲区。然而，我们需要自己来申请SDL_Overlay（注意：allocated标志会指明我们是否已经做了这个申请的动作与否）。
为了使用这个队列，我们有两个指针－－写入指针和读取指针。我们也要保证一定数量的实际数据在缓冲中。要写入到队列中，我们先要等待缓冲清空以便于有位置 来保存我们的VideoPicture。然后我们检查看我们是否已经申请到了一个可以写入覆盖的索引号。如果没有，我们要申请一段空间。我们也要重新申请 缓冲如果窗口的大小已经改变。然而，为了避免被锁定，尽量避免在这里申请（我现在还不太清楚原因；我相信是为了避免在其它线程中调用SDL覆盖函数的原 因）。
1 int queue_picture(VideoState *is, AVFrame *pFrame) { 2 VideoPicture *vp; 3 int dst_pix_fmt; 4 AVPicture pict; 5 SDL_LockMutex(is->pictq_mutex); 6 while(is->pictq_size >= VIDEO_PICTURE_QUEUE_SIZE && 7 !is->quit) { 8 SDL_CondWait(is->pictq_cond, is->pictq_mutex); 9 } 10 SDL_UnlockMutex(is->pictq_mutex); 11 if(is->quit) 12 return -1; 13 // windex is set to 0 initially 14 vp = &is->pictq[is->pictq_windex]; 15 if(!vp->bmp || 16 vp->width != is->video_st->codec->width || 17 vp->height != is->video_st->codec->height) { 18 SDL_Event event; 19 vp->allocated = 0; 20 event.type = FF_ALLOC_EVENT; 21 event.user.data1 = is; 22 SDL_PushEvent(&event); 23 SDL_LockMutex(is->pictq_mutex); 24 while(!vp->allocated && !is->quit) { 25 SDL_CondWait(is->pictq_cond, is->pictq_mutex); 26 } 27 SDL_UnlockMutex(is->pictq_mutex); 28 if(is->quit) { 29 return -1; 30 } 31 }
这里的事件机制与前面我们想要退出的时候看到的一样。我们已经定义了事件FF_ALLOC_EVENT作为SDL_USEREVENT。我们把事件发到事件队列中然后等待申请内存的函数设置好条件变量。
让我们来看一看如何来修改事件循环：
1 for(;;) { 2 SDL_WaitEvent(&event); 3 switch(event.type) { 4 case FF_ALLOC_EVENT: 5 alloc_picture(event.user.data1); 6 break;
记住event.user.data1是我们的大结构体。就这么简单。让我们看一下alloc_picture()函数：
1 void alloc_picture(void *userdata) { 2 VideoState *is = (VideoState *)userdata; 3 VideoPicture *vp; 4 vp = &is->pictq[is->pictq_windex]; 5 if(vp->bmp) { 6 // we already have one make another, bigger/smaller 7 SDL_FreeYUVOverlay(vp->bmp); 8 } 9 // Allocate a place to put our YUV image on that screen 10 vp->bmp = SDL_CreateYUVOverlay(is->video_st->codec->width, 11 is->video_st->codec->height, 12 SDL_YV12_OVERLAY, 13 screen); 14 vp->width = is->video_st->codec->width; 15 vp->height = is->video_st->codec->height; 16 SDL_LockMutex(is->pictq_mutex); 17 vp->allocated = 1; 18 SDL_CondSignal(is->pictq_cond); 19 SDL_UnlockMutex(is->pictq_mutex); 20 }
你可以看到我们把SDL_CreateYUVOverlay函数从主循环中移到了这里。这段代码应该完全可以自我注释。记住我们把高度和宽度保存到VideoPicture结构体中因为我们需要保存我们的视频的大小没有因为某些原因而改变。
好，我们几乎已经全部解决并且可以申请到YUV覆盖和准备好接收图像。让我们回顾一下queue_picture并看一个拷贝帧到覆盖的代码。你应该能认出其中的一部分：
1 int queue_picture(VideoState *is, AVFrame *pFrame) { 2 if(vp->bmp) { 3 SDL_LockYUVOverlay(vp->bmp); 4 dst_pix_fmt = PIX_FMT_YUV420P; 5 pict.data[0] = vp->bmp->pixels[0]; 6 pict.data[1] = vp->bmp->pixels[2]; 7 pict.data[2] = vp->bmp->pixels[1]; 8 pict.linesize[0] = vp->bmp->pitches[0]; 9 pict.linesize[1] = vp->bmp->pitches[2]; 10 pict.linesize[2] = vp->bmp->pitches[1]; 11 // Convert the image into YUV format that SDL uses 12 img_convert(&pict, dst_pix_fmt, 13 (AVPicture *)pFrame, is->video_st->codec->pix_fmt, 14 is->video_st->codec->width, is->video_st->codec->height); 15 SDL_UnlockYUVOverlay(vp->bmp); 16 if(++is->pictq_windex == VIDEO_PICTURE_QUEUE_SIZE) { 17 is->pictq_windex = 0; 18 } 19 SDL_LockMutex(is->pictq_mutex); 20 is->pictq_size++; 21 SDL_UnlockMutex(is->pictq_mutex); 22 } 23 return 0; 24 }
这部分代码和前面用到的一样，主要是简单的用我们的帧来填充YUV覆盖。最后一点只是简单的给队列加1。这个队列在写的时候会一直写入到满为止，在读的时 候会一直读空为止。因此所有的都依赖于is->pictq_size值，这要求我们必需要锁定它。这里我们做的是增加写指针（在必要的时候采用轮转 的方式），然后锁定队列并且增加尺寸。现在我们的读者函数将会知道队列中有了更多的信息，当队列满的时候，我们的写入函数也会知道。
