深度学习工程实践

Preknowledge

Visualization

1. Weight & Bias

Overfitting

Data Augmentation

1. Random crops and scales

   resize the short side of the image to [250, 400], and then crop [244, 244] (target size) from the image

Weight Regularization

L1/L2 Regularization

Stochastic Regularization

1. Dropout

   train时随机drop了一些输入会导致最后得到的输出期望降低了，因此在test时需要等比例增大输出

2. DropoutConnect : randomly dropout the connection between neural units

3. Cutout : randomly mask (set 0 or 255) the input picture

4. Mixup : (For classification) randomly mixup 2 input pictures and their labels

Training Trick

Hyperparameter optimization

Network Architectures

1. Squeeze-and-Excitation Network [SENet]

   

2. Neural architecture search [NAS] (auto machine learning)

Semantic Segmentation

1. Fully connected network

Upsampling

1. Unpooling

2. Max Unpooling

3. Transpose convolution (learnable upsampling)

Projects

Project 1

• Testing on Real Data

  • Take an HR image by your own phone;
  • Create LR burst images similar to the synthetic data shown in the paper;
  • Super-resolve the burst images;
  • (Change the Bayer pattern of LR images and super-resolve then again)
  • Compare the similarity between the predicted HR and GT HR images;
  • Capture and evaluate 20 images(repeat the above steps), including both indoor and outdoor scenes;compare and analyze the performance both quantitatively and qualitatively.

• Application to Downstream Tasks

  • Choose a downstream task (semantic segmentation or object detection):
  • apply your LR HR images to this downstream task;
  • Record and compare the performance difference.

• Collect training data by your own phones/cameras:

  • Fine-tune the trained model;
  • Record and compare the performance difference qualitatively and quantitatively.

• Others

Dataset	Model	PSNR↑	SSIM↓	LPIPS↑
SyntheticBurstVal	dbsr_synthetic_default.pth	39.167	0.946	0.081
	DBSRNet_syn_ep0100.pth	39.090	0.945	0.083
BurstSR	dbsr_burstsr_default.pth	47.701	0.984	0.029
	DBSRNet_burst_ep0040.pth	47.664	0.983	0.030
	DBSRNet_burst_bggr_ep0040.pth	43.867	0.964	0.063

解释：

DBSRNet_syn_ep0100.pth：使用提供的人工合成的数据集SyntheticBurstVal训练100 epoches后得到的模型

DBSRNet_burst_ep0040.pth：使用提供的真实拍摄的数据集BurstSR训练40 epoches后得到的模型

DBSRNet_burst_bggr_ep0040.pth：将数据集读取时使用的RGGB的Bayer样式转换为BGGR样式后重新训练的模型

Bayer Pattern	PSNR↑	SSIM↓	LPIPS↑
RGGB	28.249	0.813	0.156
BGGR	22.923	0.675	0.292
GRBG	22.543	0.659	0.326
GBRG	22.543	0.659	0.326

Project 2

• Take a few (at least 20,varying illumination,weather,etc.)image pairs with overlap in campus;

• Run OmniGlue (https://github.com/google-research/omniglue for correspondence estimation;

• Compute relative pose from estimated correspondences;

• Analyze failure cases

  当两幅图中能够匹配的地方全是重复的像素时会导致特征点匹配失败（无特征点）

  

• Repeat above experiments on drone-view image pairs (optional,data to be provided on piazza)

• Implement ACE0(https://github.com/nianticlabs/acezero)on your collected images, and analyze its performance;

• Others related to camera localization

Analysis

1. 图片翻转导致识别效果较差。模型在训练时没有训练过翻转图片

   数据增强、可变形卷积

2. 视角差异较大

   设计端到端网络

3. 图像模糊程度不同

4. 对于镜面物体效果较差

Project 3

• Take pictures(20 images)of your own or your friends'hands,with diverse perspective/viewpoint/background
• Evaluate the trained model of [Zhang et al.ICCV 2021]on your newly collected hand images;
• Visualize the estimated 2D 3D keypoints,and mesh models (overlayed with the original image);
• Analyze the performance,especially on failure scenarios.

Project 4

NeRF

• 神经网络模型对高频信息不敏感，可以对图片像素坐标(x, y)进行位置编码来提升

  

Project 5

Presentations

My Project

学习车道线检测相关论文及算法，基于开源数据集CULane改进目前最佳算法(CLRerNet)，提升检测精度

Reference

• https://paperswithcode.com/sota/lane-detection-on-culane

• https://github.com/hirotomusiker/clrernet