使用javacv录像，同时进行讯飞声纹认证
由于最近的demo中需要在活体检测的同时进行音视频录制 , 尝试使用MediaRecord和camera来录制视频 , 然而Camera.onPreviewFrame 不能与 MediaRecord同时调用。活体检测的原理其实是把camera的预览回调onPreviewFrame(byte[] data, Camera camera) 中的图片数据data作为参数传递到活体检测引擎中去拿返回的检测结果码,由于种种原因 , 不能使用Camera2实现 , 于是通过谷歌了解到javacv这个库可以录制视频 , 下了几个demo , 感觉不仅满足需求 , 录制的视频质量也还可以。使用javacv中的FrameRecorder进行录像，录像的时候，调用record方法写帧数据和音频数据，这时候我们有一个需求，录像的同时，要把声音实时拿过来进行声纹认证。由此产生了2个问题：
问题1：
语音识别用的是讯飞的SDK，要求声音采样率8k或16k。而设置FrameRecorder.setSampleRate(8000)后，再FrameRecorder.start()会报错，报错如下：
avcodec_encode_audio2() error 2: Could not encode audio packet.
问题2：
javacv官方录制demo中，从AudioRecord中read到的是ShortBuffer，而讯飞SDK方法要求传入byte，他的方法如下：
public void writeAudio(byte[] data, int start, int length)
百度谷歌无果，只好自己研究。
-
使用javacv进行录像
下面是使用javacv进行录像的示例代码：
1. 初始化 ffmpeg_recorder
public void initRecorder() { String ffmpeg_link = parentPath + "/" + "video.mp4"; Log.w(LOG_TAG, "init recorder"); if (yuvIplimage == null) { yuvIplimage = IplImage.create(cameraManager.getDefaultSize().width, cameraManager.getDefaultSize().height, IPL_DEPTH_8U, 2); Log.i(LOG_TAG, "create yuvIplimage"); } Log.i(LOG_TAG, "ffmpeg_url: " + ffmpeg_link); recorder = new FFmpegFrameRecorder(ffmpeg_link, cameraManager.getDefaultSize().width, cameraManager.getDefaultSize().height, 1); recorder.setFormat("mp4"); recorder.setSampleRate(sampleAudioRateInHz); // Set in the surface changed method recorder.setFrameRate(frameRate); Log.i(LOG_TAG, "recorder initialize success"); audioRecordRunnable = new AudioRecordRunnable(); audioThread = new Thread(audioRecordRunnable); try { recorder.start(); } catch (Exception e) { // TODO Auto-generated catch block e.printStackTrace(); } audioThread.start(); }
2. 捕捉摄像头视频数据：
public void onPreviewFrame(byte[] data, Camera camera) { int during = checkIfMax(new Date().getTime()); /* get video data */ if (yuvIplimage != null && isStart) { yuvIplimage.getByteBuffer().put(data); //yuvIplimage = rotateImage(yuvIplimage.asCvMat(), 90).asIplImage(); Log.v(LOG_TAG, "Writing Frame"); try { System.out.println(System.currentTimeMillis() - videoStartTime); if (during < 6000) { recorder.setTimestamp(1000 * during); recorder.record(yuvIplimage); } } catch (FFmpegFrameRecorder.Exception e) { Log.v(LOG_TAG, e.getMessage()); e.printStackTrace(); } } }
3. 捕捉声音数据：
class AudioRecordRunnable implements Runnable { @Override public void run() { android.os.Process .setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO); // Audio int bufferSize; short[] audioData; int bufferReadResult; bufferSize = AudioRecord.getMinBufferSize(sampleAudioRateInHz, AudioFormat.CHANNEL_CONFIGURATION_MONO, AudioFormat.ENCODING_PCM_16BIT); audioRecord = new AudioRecord(MediaRecorder.AudioSource.MIC, sampleAudioRateInHz, AudioFormat.CHANNEL_CONFIGURATION_MONO, AudioFormat.ENCODING_PCM_16BIT, bufferSize); audioData = new short[bufferSize]; Log.d(LOG_TAG, "audioRecord.startRecording()"); audioRecord.startRecording(); /* ffmpeg_audio encoding loop */ while (!isFinished) { // Log.v(LOG_TAG,"recording? " + recording); bufferReadResult = audioRecord.read(audioData, 0, audioData.length); if (bufferReadResult > 0) { // Log.v(LOG_TAG, "bufferReadResult: " + bufferReadResult); // If "recording" isn't true when start this thread, it // never get's set according to this if statement...!!! // Why? Good question... if (isStart) { try { Buffer[] barray = new Buffer[1]; barray[0] = ShortBuffer.wrap(audioData, 0, bufferReadResult); recorder.record(barray); // Log.v(LOG_TAG,"recording " + 1024*i + " to " + // 1024*i+1024); } catch (FFmpegFrameRecorder.Exception e) { Log.v(LOG_TAG, e.getMessage()); e.printStackTrace(); } } } } Log.v(LOG_TAG, "AudioThread Finished, release audioRecord"); /* encoding finish, release recorder */ if (audioRecord != null) { audioRecord.stop(); audioRecord.release(); audioRecord = null; Log.v(LOG_TAG, "audioRecord released"); } } }
解决问题1：
demo中默认设置FrameRecorder.setSampleRate(44100)没问题，想到一个办法，这个地方设置44100，在语音采集的地方设置8000，最后成功了。不过这个计算时间的方法要修改：
public static int getTimeStampInNsFromSampleCounted(int paramInt) { // return (int) (paramInt / 0.0441D); return (int) (paramInt / 0.0080D); }
解决问题2：
short数组转byte数组，注意数组长度变为原来的2倍
public static byte[] short2byte(short[] sData) { int shortArrsize = sData.length; byte[] bytes = new byte[shortArrsize * 2]; for (int i = 0; i < shortArrsize; i++) { bytes[i * 2] = (byte) (sData[i] & 0x00FF); bytes[(i * 2) + 1] = (byte) (sData[i] >> 8); sData[i] = 0; } return bytes; }
录制音频源码：
/** * 录制音频的线程 */ class AudioRecordRunnable implements Runnable { short[] audioData; private final AudioRecord audioRecord; private int mCount = 0; int sampleRate = Constants.AUDIO_SAMPLING_RATE; private AudioRecordRunnable() { int bufferSize = AudioRecord.getMinBufferSize(sampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT); audioRecord = new AudioRecord(MediaRecorder.AudioSource.MIC, sampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, bufferSize); audioData = new short[bufferSize]; } /** * 包含了音频的数据和起始位置 * * @param buffer */ private void record(Buffer buffer) { synchronized (mAudioRecordLock) { this.mCount += buffer.limit(); if (!mIsPause) { try { if (mRecorder != null) { mRecorder.record(sampleRate, new Buffer[]{buffer}); } } catch (FrameRecorder.Exception e) { e.printStackTrace(); } } } } /** * 更新音频的时间戳 */ private void updateTimestamp() { int i = Util.getTimeStampInNsFromSampleCounted(this.mCount); if (mAudioTimestamp != i) { mAudioTimestamp = i; mAudioTimeRecorded = System.nanoTime(); } } public void run() { android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO); if (audioRecord != null) { //判断音频录制是否被初始化 while (this.audioRecord.getState() == 0) { try { Thread.sleep(100L); } catch (InterruptedException localInterruptedException) { } } this.audioRecord.startRecording(); while ((runAudioThread)) { updateTimestamp(); int bufferReadResult = this.audioRecord.read(audioData, 0, audioData.length); if (bufferReadResult > 0) { if (recording || (mVideoTimestamp > mAudioTimestamp)) { record(ShortBuffer.wrap(audioData, 0, bufferReadResult)); } if (SpeechManager.getInstance().isListening()) { SpeechManager.getInstance().writeAudio(Util.short2byte(audioData), 0, bufferReadResult * 2); } } } SpeechManager.getInstance().stopListener(); this.audioRecord.stop(); this.audioRecord.release(); } } }
