rectify vuforia video background

Vuforia open web cam by itself while we have two ways(https://developer.vuforia.com/library/articles/Solution/How-To-Access-the-Camera-Image-in-Unity) to access the source video background and get it rectified!

 

1. result

     a stereo rectify result! (the test calibration reslut is not good)     

 

 

 

 

 

2. use the Image Class

          we could get the background image like:

          Image  image = CameraDevice.Instance.GetCameraImage(Image.PIXEL_FORMAT.RGBA8888);

          then we could copy it to a texture and get the data

          webcam_texture = new Texture2D (weight, height, TextureFormat.RGB24, false);

          webcam_data = new Color32[webcam_texture.width * webcam_texture.height];

          image.CopyToTexture(webcam_texture);

          webcam_data = webcam_texture.GetPixels32 ();

          this data is passed down to opencv dll and rectify with cv::remap() function

 

          however,  this way is far to slow!!! copying the image to texture takes lots of time and I ahieve only 15fps in my environment.

 

3. use and OpenGL texture

          Vuforia provide an example with shader program to modify the image in the background plane. we could encod the calibration map to a texture and work as coordinate lookup table, which works in GPU and would achieve high frame rate( in my environemnt, it's up to 60fps, up to the frame rate of my camera ).

          calibration maps is create like:

          C++: void initUndistortRectifyMap(InputArray cameraMatrix, InputArray distCoeffs, InputArray R, InputArray newCameraMatrix, Size size, int m1type, OutputArray map1, OutputArray map2)

 

          we usually use a CV_16SC2, mapx is a 2 channel 16-bit short, x and y coordinate. mapy is a 16-bit short, the fraction of interpolation. we change this maps to a 32 floating point each way cause GPU would interpolate itself with a floating point coordinate.

          C++: void convertMaps(InputArray map1, InputArray map2, OutputArray dstmap1, OutputArray dstmap2, int dstmap1type, bool nninterpolation=false )¶

 

          we create a two channel RG float texture as the lookup table

          texture = new Texture2D (width, height, TextureFormat.RGFloat, false);

          Material mat = GetComponent<Renderer>().material;

          mat.SetTexture ("_Texture2", texture);

 

          opencv dll open the calibration files, convert them to 32float, and passed them back to c sharp script as float array.

          map_x1 = new float[width * height];

          map_y1 = new float[width * height];

          IntPtr ptr =  get_map_x1 (opencv_wrapper_ptr);

          Marshal.Copy (ptr, map_x1, 0, width * height);

          ptr = get_map_y1 (opencv_wrapper_ptr);

          Marshal.Copy (ptr, map_y1, 0, width * height);

 

          then we feed the texture with normalized coordinate

          for (int i=0; i<width; ++i) {

            for(int j=0; j<height; ++j){

                Color color;

                color.r = map_x1[j*width+i]/(float)width;

                color.g = map_y1[j*width+i]/(float)height;

                color.b = 0.0f;

                color.a = 1.0f;

                texture.SetPixel(i,j,color);

                 }

             }

          

          In Shader file

          _Texture2 ("Texture 2 (RGFLOAT)", 2D) =  "white"{}

          uniform sampler2D _Texture2;

 

          in the fragment 

          //coordinate encoded in texture pixel

          float2 coord = tex2D (_Texture2, i.uv);

          half4 c = tex2D(_MainTex, coord);

     

          done!!!!

 

4. Future work

          I am newbee to unity and shader language. I am not sure whether fetching a positin far away from current vertex would be slow(as I remember, this is very slow in mobile GPU with GLES). if so, the coordinate access should be computed in the vertex part not the fragment. but what I have come gots good framerate, so enjoy!!