ffmpeg api 使用scale_npp的问题总结
背景
使用ffmpeg cuda解码输出的像素格式是119，通过av_hwframe_transfer_data()函数可以设置传输到内存的格式为NV12。
而最终需要的像素格式是BGR24。ffmpeg的sws_scale()函数支持NV12 YUV420 到BGR24的转换，不支持119的转换。
目前测试数据显示，NV12和YUVJ420P转换bgr24的cpu占用分别是13.2% 3.5%，即NV12转换BGR24更慢。这也和NV12的数据组织方式有关。
查看sws_scale源码，处理NV12和YUVJ420P的区别如下：
1、NV12初始化时设置chrToYV12，而YUVJ420P不设置这个函数指针。
为尝试解决NV12转换BGR24的效率问题，尝试在GPU中将NV12转换为YUV420P，使用scale_npp的接口实现。对应的命令行如下，npp像素格式转换：
ffmpeg -vsync 0 -hwaccel_device 2 -hwaccel cuda -hwaccel_output_format cuda -i ~/vedio/drone1.flv -vf "scale_npp=format=yuv420p,hwdownload,format=yuv420p" ff22cuda2.yuv
同时查看ffmpeg源码，确认scale_npp支持NV12到YUV420P的转换：
vf_scale_npp.c
static const enum AVPixelFormat supported_formats[] = {
AV_PIX_FMT_YUV420P,
AV_PIX_FMT_NV12,
AV_PIX_FMT_YUV444P,
};
遇到的问题
问题一：Impossible to convert between the formats supported by the filter 'in' and the filter 'auto_scaler_0'
原因是 pix_fmt设置错误，需要设置为AV_PIX_FMT_CUDA
avfilter_graph_create_filter()函数作用是Create and add a filter instance into an existing graph. 参照doc/examples/filtering_video.c中的init_filters函数，调用avfilter_graph_create_filter时需设置args。
当前程序设置的是"video_size=1920x1080:pix_fmt=119:time_base=1/1000:pixel_aspect=1/1"
从AVCodecContext *dec_ctx获取的pix_fmt在解码前是0，需要修改为119才行。因为scale_npp的输入是显卡上的数据，cuda解码的输出格式就是119。
命令行方式使用scale_npp也必须设置-hwaccel_output_format cuda才行。
问题二：No hw context provided on input
原因是 input filter需要设置hw_frames_ctx；
1、 经查看报错代码在libavfilter/vf_scale_npp.c中的init_processing_chain()函数.
2、 查看ffmpeg命令行方式在调用scale_npp的区别，发现fftools/ffmpeg_filter.c中的configure_input_video_filter()函数，在创建filter之后设置了hw_frames_ctx;
if ((ret = avfilter_graph_create_filter(&ifilter->filter, buffer_filt, name, args.str, NULL, fg->graph)) < 0)
goto fail;
par->hw_frames_ctx = ifilter->hw_frames_ctx;
ret = av_buffersrc_parameters_set(ifilter->filter, par);
if (ret < 0)
goto fail;
av_freep(&par);
因此，在demo中通过AVBufferSrcParameters设置hw_frames_ctx即可解决此问题。
ffmeg filter源码相关
filter相关的实现代码都在libavfilter中，scale_npp相关的在libavfilter/vf_scale_npp.c中。
而ffmpeg命令行相关的功能代码均在fftools命令下，包括参数解析，编解码，缩放等。也是调用的libavcodec libavfilter等库。
其中filter相关都在fftools/fffmpeg_filter.c文件中处理。
重要的函数调用堆栈：
(gdb) bt
#0 filter_query_formats (ctx=0x3f65f40) at libavfilter/avfiltergraph.c:333
#1 0x00000000004dc154 in query_formats (graph=graph@entry=0x3f53c80, log_ctx=log_ctx@entry=0x0) at libavfilter/avfiltergraph.c:456
#2 0x00000000004dcfaf in graph_config_formats (log_ctx=<optimized out>, graph=<optimized out>) at libavfilter/avfiltergraph.c:1166
#3 avfilter_graph_config (graphctx=0x3f53c80, log_ctx=log_ctx@entry=0x0) at libavfilter/avfiltergraph.c:1277
#4 0x00000000004a3b0d in configure_filtergraph (fg=fg@entry=0x23f1940) at fftools/ffmpeg_filter.c:1107 //初始化filter_graph
#5 0x00000000004b432d in ifilter_send_frame (frame=0x336d880, ifilter=0x284aec0) at fftools/ffmpeg.c:2166
#6 send_frame_to_filters (ist=ist@entry=0x23f4e40, decoded_frame=decoded_frame@entry=0x336d880) at fftools/ffmpeg.c:2247
#7 0x00000000004b4b81 in decode_video (ist=ist@entry=0x23f4e40, pkt=pkt@entry=0x7fffffffda00, got_output=got_output@entry=0x7fffffffd710, duration_pts=duration_pts@entry=0x7fffffffd718,
eof=eof@entry=0, decode_failed=decode_failed@entry=0x7fffffffd714) at fftools/ffmpeg.c:2446
#8 0x00000000004b6cc2 in process_input_packet (no_eof=0, pkt=0x7fffffffd9a0, ist=0x23f4e40) at fftools/ffmpeg.c:2600
#9 process_input (file_index=<optimized out>) at fftools/ffmpeg.c:4491
#10 0x00000000004b9c53 in transcode_step () at fftools/ffmpeg.c:4611
#11 transcode () at fftools/ffmpeg.c:4665
(gdb) bt
#0 nppscale_deinterleave (ctx=ctx@entry=0x3f51a80, stage=stage@entry=0x3e836c8, out=0x3f64cc0, in=0x3f67c00) at libavfilter/vf_scale_npp.c:389
#1 0x00000000005c7840 in nppscale_scale (in=0x3f67c00, out=0x3f67e80, ctx=0x3f51a80) at libavfilter/vf_scale_npp.c:477 //执行nppscale转换
#2 nppscale_filter_frame (link=link@entry=0x3f66b40, in=0x3f67c00) at libavfilter/vf_scale_npp.c:526
#3 0x00000000004db273 in ff_filter_frame_framed (frame=0x3f67c00, link=0x3f66b40) at libavfilter/avfilter.c:1066
#4 ff_filter_frame_to_filter (link=0x3f66b40) at libavfilter/avfilter.c:1214
#5 ff_filter_activate_default (filter=<optimized out>) at libavfilter/avfilter.c:1263
#6 ff_filter_activate (filter=<optimized out>) at libavfilter/avfilter.c:1424
#7 0x00000000004de97c in ff_filter_graph_run_once (graph=graph@entry=0x3f53c80) at libavfilter/avfiltergraph.c:1456
#8 0x00000000004df918 in push_frame (graph=0x3f53c80) at libavfilter/buffersrc.c:184
#9 av_buffersrc_add_frame_internal (ctx=ctx@entry=0x3f65f40, frame=frame@entry=0x336d880, flags=flags@entry=4) at libavfilter/buffersrc.c:247
#10 0x00000000004dff5d in av_buffersrc_add_frame_flags (ctx=0x3f65f40, frame=frame@entry=0x336d880, flags=flags@entry=4) at libavfilter/buffersrc.c:167
#11 0x00000000004b4349 in ifilter_send_frame (frame=0x336d880, ifilter=0x284aec0) at fftools/ffmpeg.c:2173
#12 send_frame_to_filters (ist=ist@entry=0x23f4e40, decoded_frame=decoded_frame@entry=0x336d880) at fftools/ffmpeg.c:2247
#13 0x00000000004b4b81 in decode_video (ist=ist@entry=0x23f4e40, pkt=pkt@entry=0x7fffffffda00, got_output=got_output@entry=0x7fffffffd710, duration_pts=duration_pts@entry=0x7fffffffd718,
eof=eof@entry=0, decode_failed=decode_failed@entry=0x7fffffffd714) at fftools/ffmpeg.c:2446
#14 0x00000000004b6cc2 in process_input_packet (no_eof=0, pkt=0x7fffffffd9a0, ist=0x23f4e40) at fftools/ffmpeg.c:2600
#15 process_input (file_index=<optimized out>) at fftools/ffmpeg.c:4491
#16 0x00000000004b9c53 in transcode_step () at fftools/ffmpeg.c:4611
#17 transcode () at fftools/ffmpeg.c:4665
(gdb) bt
#0 hwdownload_filter_frame (link=link@entry=0x3f65700, input=0x3f67e80) at libavfilter/vf_hwdownload.c:152
#1 0x00000000004db273 in ff_filter_frame_framed (frame=0x3f67e80, link=0x3f65700) at libavfilter/avfilter.c:1066
#2 ff_filter_frame_to_filter (link=0x3f65700) at libavfilter/avfilter.c:1214
#3 ff_filter_activate_default (filter=<optimized out>) at libavfilter/avfilter.c:1263
#4 ff_filter_activate (filter=<optimized out>) at libavfilter/avfilter.c:1424
#5 0x00000000004de97c in ff_filter_graph_run_once (graph=graph@entry=0x3f53c80) at libavfilter/avfiltergraph.c:1456
#6 0x00000000004df918 in push_frame (graph=0x3f53c80) at libavfilter/buffersrc.c:184
#7 av_buffersrc_add_frame_internal (ctx=ctx@entry=0x3f65f40, frame=frame@entry=0x336d880, flags=flags@entry=4) at libavfilter/buffersrc.c:247
#8 0x00000000004dff5d in av_buffersrc_add_frame_flags (ctx=0x3f65f40, frame=frame@entry=0x336d880, flags=flags@entry=4) at libavfilter/buffersrc.c:167
#9 0x00000000004b4349 in ifilter_send_frame (frame=0x336d880, ifilter=0x284aec0) at fftools/ffmpeg.c:2173
#10 send_frame_to_filters (ist=ist@entry=0x23f4e40, decoded_frame=decoded_frame@entry=0x336d880) at fftools/ffmpeg.c:2247
#11 0x00000000004b4b81 in decode_video (ist=ist@entry=0x23f4e40, pkt=pkt@entry=0x7fffffffda00, got_output=got_output@entry=0x7fffffffd710, duration_pts=duration_pts@entry=0x7fffffffd718,
eof=eof@entry=0, decode_failed=decode_failed@entry=0x7fffffffd714) at fftools/ffmpeg.c:2446
#12 0x00000000004b6cc2 in process_input_packet (no_eof=0, pkt=0x7fffffffd9a0, ist=0x23f4e40) at fftools/ffmpeg.c:2600
#13 process_input (file_index=<optimized out>) at fftools/ffmpeg.c:4491
#14 0x00000000004b9c53 in transcode_step () at fftools/ffmpeg.c:4611
#15 transcode () at fftools/ffmpeg.c:4665
需要关注的函数：
av_buffersrc_add_frame_flags
av_buffersink_get_frame
avfilter_graph_create_filter
avfilter_graph_parse_ptr
avfilter_graph_config
sws_scale
fftools中的函数：
int configure_filtergraph(FilterGraph *fg)
static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, AVFilterInOut *in)
hw_device_setup_for_filter
static int hwaccel_retrieve_data(AVCodecContext *avctx, AVFrame *input)
static int transcode_step(void)
static int ifilter_send_frame(InputFilter *ifilter, AVFrame *frame)
需要关注的结构体定义：
AVFilterGraph
AVFilterContext
AVBufferSrcParameters
AVFilterInOut
AVFilter
AVCodecContext
AVFrame
参考信息
https://www.ffmpeg.org/doxygen/3.2/vf__scale__npp_8c_source.html
https://ffmpeg.org/doxygen/3.1/swscale_8c_source.html
https://github.com/FFmpeg/FFmpeg/tree/n4.3.2
https://stackoverflow.com/questions/47049312/how-can-i-convert-an-ffmpeg-avframe-with-pixel-format-av-pix-fmt-cuda-to-a-new-a
https://www.jianshu.com/p/ad05a94001b4
scale_npp: This is a scaling filter implemented in NVIDIA's Performance Primitives. It's primary dependency is the CUDA SDK, and it must be explicitly enabled by passing --enable-libnpp, --enable-cuda-nvcc and --enable-nonfree flags to ./configure at compile time when building FFmpeg from source. Use this filter in place of scale_cuda wherever possible.
像素格式说明：
AV_PIX_FMT_YUV420P = 0, ///< planar YUV 4:2:0, 12bpp, (1 Cr & Cb sample per 2x2 Y samples)
AV_PIX_FMT_RGB24 = 2, ///< packed RGB 8:8:8, 24bpp, RGBRGB...
AV_PIX_FMT_BGR24 = 3, ///< packed RGB 8:8:8, 24bpp, BGRBGR...
AV_PIX_FMT_YUVJ420P = 12, ///< planar YUV 4:2:0, 12bpp, full scale (JPEG), deprecated in favor of AV_PIX_FMT_YUV420P and setting color_range
AV_PIX_FMT_NV12 = 23, ///< planar YUV 4:2:0, 12bpp, 1 plane for Y and 1 plane for the UV components, which are interleaved (first byte U and the following byte V)
AV_PIX_FMT_CUDA = 119, ///< HW acceleration through CUDA. data[i] contain CUdeviceptr pointers exactly as for system memory frames.
ffmpeg编译时configure配置命令（前提需要准备cuda库，nv-codec-headers，libx264，fdk_aac等库）：
PKG_CONFIG_PATH="$HOME/local/lib/pkgconfig" ./configure --prefix="$HOME/local" --pkg-config-flags="--static" --extra-cflags="-I$HOME/local/include" --extra-ldflags="-L$HOME/local/lib" --extra-libs=-lpthread --extra-libs=-lm --bindir="$HOME/local/bin" --enable-gpl --enable-libfdk_aac --enable-libmp3lame --enable-libx264 --enable-nonfree --enable-gpl --enable-cuda --enable-cuvid --enable-nvdec --enable-nvenc --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64
编译ffmpeg因nv-codec-headers版本不对导致的报错：
bugfix：[h264_nvenc @ 0x32c2080] Driver does not support the required nvenc API version. Required: 10.0 Found: 9.0
https://blog.csdn.net/qq_23282479/article/details/107579032
https://forums.developer.nvidia.com/t/ffmpeg-nvenc-issue-driver-does-not-support-the-required-nvenc-api-version-required-9-1-found-9-0/109348
nv-codec-headers里的README记录了最低要求的驱动版本号（可以到github里面去看https://github.com/FFmpeg/nv-codec-headers）
如果cuda版本较低，可以在nv-codec-headers目录下执行git checkout sdk/9.0，切换回旧版本后，make clean之后重新编译ffmpeg即可。
