深度学习工程实践

Preknowledge

Visualization

1. Weight & Bias

Overfitting

Data Augmentation

1. Random crops and scales

   resize the short side of the image to [250, 400], and then crop [244, 244] (target size) from the image

Weight Regularization

L1/L2 Regularization

Stochastic Regularization

1. Dropout

   train时随机drop了一些输入会导致最后得到的输出期望降低了，因此在test时需要等比例增大输出

2. DropoutConnect : randomly dropout the connection between neural units

3. Cutout : randomly mask (set 0 or 255) the input picture

4. Mixup : (For classification) randomly mixup 2 input pictures and their labels

Training Trick

Hyperparameter optimization

Network Architectures

1. Squeeze-and-Excitation Network [SENet]

   

2. Neural architecture search [NAS] (auto machine learning)

Semantic Segmentation

1. Fully connected network

Upsampling

1. Unpooling

2. Max Unpooling

3. Transpose convolution (learnable upsampling)

Projects

Project 1

• Testing on Real Data

  • Take an HR image by your own phone;
  • Create LR burst images similar to the synthetic data shown in the paper;
  • Super-resolve the burst images;
  • (Change the Bayer pattern of LR images and super-resolve then again)
  • Compare the similarity between the predicted HR and GT HR images;
  • Capture and evaluate 20 images(repeat the above steps), including both indoor and outdoor scenes;compare and analyze the performance both quantitatively and qualitatively.

• Application to Downstream Tasks

  • Choose a downstream task (semantic segmentation or object detection):
  • apply your LR HR images to this downstream task;
  • Record and compare the performance difference.

• Collect training data by your own phones/cameras:

  • Fine-tune the trained model;
  • Record and compare the performance difference qualitatively and quantitatively.

• Others

Dataset	Model	PSNR↑	SSIM↓	LPIPS↑
SyntheticBurstVal	dbsr_synthetic_default.pth	39.167	0.946	0.081
	DBSRNet_syn_ep0100.pth	39.090	0.945	0.083
BurstSR	dbsr_burstsr_default.pth	47.701	0.984	0.029
	DBSRNet_burst_ep0040.pth	47.664	0.983	0.030
	DBSRNet_burst_bggr_ep0040.pth	43.867	0.964	0.063

解释：

DBSRNet_syn_ep0100.pth：使用提供的人工合成的数据集SyntheticBurstVal训练100 epoches后得到的模型

DBSRNet_burst_ep0040.pth：使用提供的真实拍摄的数据集BurstSR训练40 epoches后得到的模型

DBSRNet_burst_bggr_ep0040.pth：将数据集读取时使用的RGGB的Bayer样式转换为BGGR样式后重新训练的模型

Bayer Pattern	PSNR↑	SSIM↓	LPIPS↑
RGGB	28.249	0.813	0.156
BGGR	22.923	0.675	0.292
GRBG	22.543	0.659	0.326
GBRG	22.543	0.659	0.326

Project 2

• Take a few (at least 20,varying illumination,weather,etc.)image pairs with overlap in campus;

• Run OmniGlue (https://github.com/google-research/omniglue for correspondence estimation;

• Compute relative pose from estimated correspondences;

• Analyze failure cases

• Repeat above experiments on drone-view image pairs (optional,data to be provided on piazza)

• Implement ACE0(https://github.com/nianticlabs/acezero)on your collected images, and analyze its performance;

• Others related to camera localization

Analysis

1. 图片翻转导致识别效果较差。模型在训练时没有训练过翻转图片

   数据增强、可变形卷积

2. 视角差异较大

   设计端到端网络

3. 图像模糊程度不同

4. 对于镜面物体效果较差

Project 3

• Take pictures(20 images)of your own or your friends'hands,with diverse perspective/viewpoint/background
• Evaluate the trained model of [Zhang et al.ICCV 2021]on your newly collected hand images;
• Visualize the estimated 2D 3D keypoints,and mesh models (overlayed with the original image);
• Analyze the performance,especially on failure scenarios.

Project 4

• NeRF

• 神经网络模型对高频信息不敏感，可以对图片像素坐标(x, y)进行位置编码来提升

  

Project 5

Presentations

My Project

学习车道线检测相关论文及算法，基于开源数据集CULane改进目前最佳算法(CLRerNet)，提升检测精度

Reference

• https://paperswithcode.com/sota/lane-detection-on-culane

• https://github.com/hirotomusiker/clrernet