（原）人脸姿态识别Fine-Grained Head Pose EstimationWithout Keypoints

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

本文目录

• 1 简介
• 2 网络结构
• 3 代码
• 3.1 网络结构
• 3.2 训练代码
• 3.3 dataset代码
• 3.4 测试代码

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

转载请注明出处：

https://www.cnblogs.com/darkknightzh/p/12150096.html

论文：

Fine-Grained Head Pose EstimationWithout Keypoints

论文网址：

https://arxiv.org/abs/1710.00925v5

官方pytorch网址：

https://github.com/natanielruiz/deep-head-pose

说明：该代码是pytorch早期的版本。

参考网址：

https://www.cnblogs.com/aoru45/p/11186629.html

回到顶部(go to top)

1 简介

该论文通过训练多损失的CNN网络来预测人脸的pitch，yaw和roll三个角度值。两个损失分别是分类损失和回归损失。图像通过网络后，得到pitch，yaw和roll这三组特征。

训练阶段：分类损失将特征通过softmax后直接分类，回归损失则是计算归一化后特征的数学期望，并将期望和实际的角度计算MSE loss。

测试阶段：使用归一化后特征的数学期望得到预测的pitch，yaw和roll。

参考网址中指出：分类的loss占比会影响梯度方向，从而会起到一个导向作用，引导回归往一个合适的方向，这是梯度方向上的引导。

回到顶部(go to top)

2 网络结构

网络结构如下图所示：

输入图像通过Resnet50骨干网络得到特征，而后通过三个fc层，分别作为pitch，yaw和roll的分类输入。代码中CrossEntropyLoss包含了softmax，因而实际上和上图一致。感觉上图右侧按照下面的画法，更易理解吧（只画出了yaw，另外两个同理）。

分类时，起点-99度（包含），终点102度（不包含），间隔3，共67个值，里面共66个间隔，作为离散的类别，对这些使用CrossEntropyLoss计算损失。另一方面，将softmax后归一化的特征作为概率，和对应类别相乘并求和，作为期望，和实际角度计算MSELoss。

损失函数如下：

$L=H(y,\hat{y})+\alpha \centerdot MSE(y,\hat{y})$

其中$y$为预测值，$\hat{y}$为真实值。H代表交叉熵损失。

回到顶部(go to top)

3 代码

3.1 网络结构

hopenet网络结构代码如下：

class Hopenet(nn.Module):
    # Hopenet with 3 output layers for yaw, pitch and roll
    # Predicts Euler angles by binning and regression with the expected value
    def __init__(self, block, layers, num_bins):
        self.inplanes = 64
        super(Hopenet, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)
        self.bn1 = nn.BatchNorm2d(64)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.layer1 = self._make_layer(block, 64, layers[0])
        self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
        self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
        self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
        self.avgpool = nn.AvgPool2d(7)   # 至此为resnet的结构。得到特征
        self.fc_yaw = nn.Linear(512 * block.expansion, num_bins)   # 特征到分类的三个fc层。
        self.fc_pitch = nn.Linear(512 * block.expansion, num_bins) # 特征到分类的三个fc层。
        self.fc_roll = nn.Linear(512 * block.expansion, num_bins)  # 特征到分类的三个fc层。

        self.fc_finetune = nn.Linear(512 * block.expansion + 3, 3)  # Vestigial layer from previous experiments  未使用

        for m in self.modules():    # 初始化模型参数
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()

    def _make_layer(self, block, planes, blocks, stride=1):
        downsample = None
        if stride != 1 or self.inplanes != planes * block.expansion:
            downsample = nn.Sequential(
                nn.Conv2d(self.inplanes, planes * block.expansion, kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(planes * block.expansion),
            )

        layers = []
        layers.append(block(self.inplanes, planes, stride, downsample))
        self.inplanes = planes * block.expansion
        for i in range(1, blocks):
            layers.append(block(self.inplanes, planes))

        return nn.Sequential(*layers)

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        pre_yaw = self.fc_yaw(x)
        pre_pitch = self.fc_pitch(x)
        pre_roll = self.fc_roll(x)

        return pre_yaw, pre_pitch, pre_roll

说明：self.fc_finetune未使用

3.2 训练代码

def parse_args():
    """Parse input arguments."""
    parser = argparse.ArgumentParser(description='Head pose estimation using the Hopenet network.')
    parser.add_argument('--gpu', dest='gpu_id', help='GPU device id to use [0]', default=0, type=int)
    parser.add_argument('--num_epochs', dest='num_epochs', help='Maximum number of training epochs.', default=5, type=int)
    parser.add_argument('--batch_size', dest='batch_size', help='Batch size.', default=16, type=int)
    parser.add_argument('--lr', dest='lr', help='Base learning rate.', default=0.001, type=float)
    parser.add_argument('--dataset', dest='dataset', help='Dataset type.', default='Pose_300W_LP', type=str)
    parser.add_argument('--data_dir', dest='data_dir', help='Directory path for data.', default='', type=str)
    parser.add_argument('--filename_list', dest='filename_list', help='Path to text file containing relative paths for every example.', default='', type=str)
    parser.add_argument('--output_string', dest='output_string', help='String appended to output snapshots.', default = '', type=str)
    parser.add_argument('--alpha', dest='alpha', help='Regression loss coefficient.', default=0.001, type=float)
    parser.add_argument('--snapshot', dest='snapshot', help='Path of model snapshot.', default='', type=str)

    args = parser.parse_args()
    return args

def get_ignored_params(model):  # Generator function that yields ignored params.
    b = [model.conv1, model.bn1, model.fc_finetune]
    for i in range(len(b)):
        for module_name, module in b[i].named_modules():
            if 'bn' in module_name:
                module.eval()
            for name, param in module.named_parameters():
                yield param

def get_non_ignored_params(model):  # Generator function that yields params that will be optimized.
    b = [model.layer1, model.layer2, model.layer3, model.layer4]
    for i in range(len(b)):
        for module_name, module in b[i].named_modules():
            if 'bn' in module_name:
                module.eval()
            for name, param in module.named_parameters():
                yield param

def get_fc_params(model):   # Generator function that yields fc layer params.
    b = [model.fc_yaw, model.fc_pitch, model.fc_roll]
    for i in range(len(b)):
        for module_name, module in b[i].named_modules():
            for name, param in module.named_parameters():
                yield param

def load_filtered_state_dict(model, snapshot):
    # By user apaszke from discuss.pytorch.org
    model_dict = model.state_dict()
    snapshot = {k: v for k, v in snapshot.items() if k in model_dict}
    model_dict.update(snapshot)
    model.load_state_dict(model_dict)

if __name__ == '__main__':
    args = parse_args()

    cudnn.enabled = True
    num_epochs = args.num_epochs
    batch_size = args.batch_size
    gpu = args.gpu_id

    if not os.path.exists('output/snapshots'):
        os.makedirs('output/snapshots')

    # ResNet50 structure
    #  -99到99，以3为step，共67个bin，这些bin之间共66个位置。因而最后一个参数为66
    model = hopenet.Hopenet(torchvision.models.resnet.Bottleneck, [3, 4, 6, 3], 66)

    if args.snapshot == '':
        load_filtered_state_dict(model, model_zoo.load_url('https://download.pytorch.org/models/resnet50-19c8e357.pth'))
    else:
        saved_state_dict = torch.load(args.snapshot)
        model.load_state_dict(saved_state_dict)

    print('Loading data.')

    transformations = transforms.Compose([transforms.Scale(240),
            transforms.RandomCrop(224), transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])

    if args.dataset == 'Pose_300W_LP':   # 载入数据的dataset
        pose_dataset = datasets.Pose_300W_LP(args.data_dir, args.filename_list, transformations)
    elif args.dataset == 'Pose_300W_LP_random_ds':
        pose_dataset = datasets.Pose_300W_LP_random_ds(args.data_dir, args.filename_list, transformations)
    elif args.dataset == 'Synhead':
        pose_dataset = datasets.Synhead(args.data_dir, args.filename_list, transformations)
    elif args.dataset == 'AFLW2000':
        pose_dataset = datasets.AFLW2000(args.data_dir, args.filename_list, transformations)
    elif args.dataset == 'BIWI':
        pose_dataset = datasets.BIWI(args.data_dir, args.filename_list, transformations)
    elif args.dataset == 'AFLW':
        pose_dataset = datasets.AFLW(args.data_dir, args.filename_list, transformations)
    elif args.dataset == 'AFLW_aug':
        pose_dataset = datasets.AFLW_aug(args.data_dir, args.filename_list, transformations)
    elif args.dataset == 'AFW':
        pose_dataset = datasets.AFW(args.data_dir, args.filename_list, transformations)
    else:
        print('Error: not a valid dataset name')
        sys.exit()

    train_loader = torch.utils.data.DataLoader(dataset=pose_dataset, batch_size=batch_size, shuffle=True, num_workers=2)

    model.cuda(gpu)
    criterion = nn.CrossEntropyLoss().cuda(gpu)   # 自带softmax
    reg_criterion = nn.MSELoss().cuda(gpu)
    alpha = args.alpha   # Regression loss coefficient
    softmax = nn.Softmax().cuda(gpu)
    idx_tensor = [idx for idx in range(66)]
    idx_tensor = Variable(torch.FloatTensor(idx_tensor)).cuda(gpu)

    optimizer = torch.optim.Adam([{'params': get_ignored_params(model), 'lr': 0},
                                  {'params': get_non_ignored_params(model), 'lr': args.lr},
                                  {'params': get_fc_params(model), 'lr': args.lr * 5}],
                                   lr = args.lr)  # 不同层给与不同的学习率

    print('Ready to train network.')
    for epoch in range(num_epochs):
        for i, (images, labels, cont_labels, name) in enumerate(train_loader):  # labels为离散化的bin，cont_labels为实际值
            images = Variable(images).cuda(gpu)

            label_yaw = Variable(labels[:,0]).cuda(gpu)   # Binned labels
            label_pitch = Variable(labels[:,1]).cuda(gpu)
            label_roll = Variable(labels[:,2]).cuda(gpu)

            label_yaw_cont = Variable(cont_labels[:,0]).cuda(gpu)   # Continuous labels
            label_pitch_cont = Variable(cont_labels[:,1]).cuda(gpu)
            label_roll_cont = Variable(cont_labels[:,2]).cuda(gpu)

            yaw, pitch, roll = model(images)   # Forward pass

            loss_yaw = criterion(yaw, label_yaw)       # Cross entropy loss 此处为分类损失，对应类别0-65
            loss_pitch = criterion(pitch, label_pitch) # label_yaw等为离散化的bin，只有真实的bin为1，其他为0.
            loss_roll = criterion(roll, label_roll)    # pitch等为特征

            yaw_predicted = softmax(yaw)   # MSE loss  此处为回归损失
            pitch_predicted = softmax(pitch)  # yaw_predicted等为特征通过softmax后归一化的特征（即概率）
            roll_predicted = softmax(roll)

            # 离散型随机变量的一切可能的取值（idx_tensor）与对应的概率（yaw_predicted）乘积之和称为该离散型随机变量的数学期望
            yaw_predicted = torch.sum(yaw_predicted * idx_tensor, 1) * 3 - 99     # 将归一化的特征对应的实际预测值求和，最终为bz个。
            pitch_predicted = torch.sum(pitch_predicted * idx_tensor, 1) * 3 - 99 # 此处每个概率乘以对应类别，而后求和，得到期望。
            roll_predicted = torch.sum(roll_predicted * idx_tensor, 1) * 3 - 99 

            loss_reg_yaw = reg_criterion(yaw_predicted, label_yaw_cont)  # 预测值的总的期望和真实值计算回归损失
            loss_reg_pitch = reg_criterion(pitch_predicted, label_pitch_cont)
            loss_reg_roll = reg_criterion(roll_predicted, label_roll_cont)

            loss_yaw += alpha * loss_reg_yaw   # Total loss 总损失
            loss_pitch += alpha * loss_reg_pitch
            loss_roll += alpha * loss_reg_roll

            loss_seq = [loss_yaw, loss_pitch, loss_roll]
            grad_seq = [torch.ones(1).cuda(gpu) for _ in range(len(loss_seq))]
            optimizer.zero_grad()
            torch.autograd.backward(loss_seq, grad_seq)   # 此处可以认为是不同损失加权（权重均为1）。
            optimizer.step()

            if (i+1) % 100 == 0:
                print ('Epoch [%d/%d], Iter [%d/%d] Losses: Yaw %.4f, Pitch %.4f, Roll %.4f'
                       %(epoch+1, num_epochs, i+1, len(pose_dataset)//batch_size, loss_yaw.data[0], loss_pitch.data[0], loss_roll.data[0]))

        # Save models at numbered epochs.d
        if epoch % 1 == 0 and epoch < num_epochs:   # 保存模型
            print('Taking snapshot...')
            torch.save(model.state_dict(),
            'output/snapshots/' + args.output_string + '_epoch_'+ str(epoch+1) + '.pkl')

3.3 dataset代码

class Pose_300W_LP(Dataset):
    # Head pose from 300W-LP dataset
    def __init__(self, data_dir, filename_path, transform, img_ext='.jpg', annot_ext='.mat', image_mode='RGB'):
        self.data_dir = data_dir
        self.transform = transform
        self.img_ext = img_ext
        self.annot_ext = annot_ext

        filename_list = get_list_from_filenames(filename_path)

        self.X_train = filename_list
        self.y_train = filename_list
        self.image_mode = image_mode
        self.length = len(filename_list)

    def __getitem__(self, index):
        img = Image.open(os.path.join(self.data_dir, self.X_train[index] + self.img_ext))
        img = img.convert(self.image_mode)
        mat_path = os.path.join(self.data_dir, self.y_train[index] + self.annot_ext)

        # Crop the face loosely
        pt2d = utils.get_pt2d_from_mat(mat_path)   # 得到2D的landmarks
        x_min = min(pt2d[0,:])
        y_min = min(pt2d[1,:])
        x_max = max(pt2d[0,:])
        y_max = max(pt2d[1,:])

        k = np.random.random_sample() * 0.2 + 0.2    # k = 0.2 to 0.40
        x_min -= 0.6 * k * abs(x_max - x_min)
        y_min -= 2 * k * abs(y_max - y_min)          # why 2???
        x_max += 0.6 * k * abs(x_max - x_min)
        y_max += 0.6 * k * abs(y_max - y_min)
        img = img.crop((int(x_min), int(y_min), int(x_max), int(y_max)))   # 随机裁减

        pose = utils.get_ypr_from_mat(mat_path)  # We get the pose in radians，弧度
        pitch = pose[0] * 180 / np.pi   # 弧度转换为度
        yaw = pose[1] * 180 / np.pi
        roll = pose[2] * 180 / np.pi

        rnd = np.random.random_sample()  # 随机水平旋转
        if rnd < 0.5:
            yaw = -yaw    # 摇头需要取反
            roll = -roll  # 摆头需要取反
            img = img.transpose(Image.FLIP_LEFT_RIGHT)


        rnd = np.random.random_sample()  # Blur
        if rnd < 0.05:
            img = img.filter(ImageFilter.BLUR)

        # Bin values
        bins = np.array(range(-99, 102, 3))   # -99到99，以3为step，共67个bin，这些bin之间共66个位置。
        # 返回输入在bins中的位置。当输入<bin[0]，输出为0。由于此处输入不会<bin[0]，因而减1，使得输出范围为0-65
        binned_pose = np.digitize([yaw, pitch, roll], bins) - 1

        # Get target tensors
        labels = binned_pose
        cont_labels = torch.FloatTensor([yaw, pitch, roll])   # 实际的label

        if self.transform is not None:
            img = self.transform(img)

        return img, labels, cont_labels, self.X_train[index]

    def __len__(self):
        # 122,450
        return self.length

3.4 测试代码

说明：测试代码进行了修改，只读取一张图像，输入人脸框。

def parse_args():
    """Parse input arguments."""
    parser = argparse.ArgumentParser(description='Head pose estimation using the Hopenet network.')
    parser.add_argument('--gpu', dest='gpu_id', help='GPU device id to use [0]', default=-1, type=int)
    parser.add_argument('--snapshot', dest='snapshot', help='Path of model snapshot.', default='hopenet_robust_alpha1.pkl', type=str)
    args = parser.parse_args()
    return args

if __name__ == '__main__':
    args = parse_args()

    gpu = args.gpu_id

    if gpu >= 0:
        cudnn.enabled = True

    # ResNet50 structure
    model = hopenet.Hopenet(torchvision.models.resnet.Bottleneck, [3, 4, 6, 3], 66)  # 载入模型

    print('Loading snapshot.')
    if gpu >= 0:
        saved_state_dict = torch.load(args.snapshot)
    else:
        saved_state_dict = torch.load(args.snapshot, map_location=torch.device('cpu'))  # 无GPU的话，载入模型参数到CPU中
    model.load_state_dict(saved_state_dict)

    transformations = transforms.Compose([transforms.Scale(224),
    transforms.CenterCrop(224), transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])

    if gpu>=0:
        model.cuda(gpu)

    model.eval()  # Change model to 'eval' mode (BN uses moving mean/var).
    total = 0

    idx_tensor = [idx for idx in range(66)]
    idx_tensor = torch.FloatTensor(idx_tensor)
    if gpu>=0:
        idx_tensor = idx_tensor.cuda(gpu)

    img_path = '1.jpg'
    frame = cv2.imread(img_path, 1)
    cv2_frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

    x_min, y_min, x_max, y_max = 439,375,642,596   # 人脸框位置

    bbox_width = abs(x_max - x_min)   # 人脸框宽
    bbox_height = abs(y_max - y_min)  # 人脸框高
    # x_min -= 3 * bbox_width / 4
    # x_max += 3 * bbox_width / 4
    # y_min -= 3 * bbox_height / 4
    # y_max += bbox_height / 4
    x_min -= 50   # 增大人脸框
    x_max += 50
    y_min -= 50
    y_max += 30
    x_min = max(x_min, 0)
    y_min = max(y_min, 0)
    x_max = min(frame.shape[1], x_max)
    y_max = min(frame.shape[0], y_max)

    img = cv2_frame[y_min:y_max,x_min:x_max]    # Crop face loosely。裁减人脸
    img = Image.fromarray(img)

    img = transformations(img)   # Transform
    img_shape = img.size()
    img = img.view(1, img_shape[0], img_shape[1], img_shape[2])  # NCHW
    img = torch.tensor(img)
    if gpu >= 0:
        img = img.cuda(gpu)

    yaw, pitch, roll = model(img)  # 得到特征

    yaw_predicted = F.softmax(yaw)   # 特征归一化
    pitch_predicted = F.softmax(pitch)
    roll_predicted = F.softmax(roll)

    # 得到期望，将期望变换到连续值的度数
    yaw_predicted = torch.sum(yaw_predicted.data[0] * idx_tensor) * 3 - 99       # Get continuous predictions in degrees.
    pitch_predicted = torch.sum(pitch_predicted.data[0] * idx_tensor) * 3 - 99
    roll_predicted = torch.sum(roll_predicted.data[0] * idx_tensor) * 3 - 99

    frame = utils.draw_axis(frame, yaw_predicted, pitch_predicted, roll_predicted, tdx = (x_min + x_max) / 2, tdy= (y_min + y_max) / 2, size = bbox_height/2)
    print(yaw_predicted.item(), pitch_predicted.item(), roll_predicted.item())
    cv2.imwrite('{}.png'.format(os.path.splitext(img_path)[0]), frame)

其中draw_axis如下：

def draw_axis(img, yaw, pitch, roll, tdx=None, tdy=None, size = 100):
    pitch = pitch * np.pi / 180    # 变换到弧度
    yaw = -(yaw * np.pi / 180)
    roll = roll * np.pi / 180

    if tdx != None and tdy != None:
        tdx = tdx
        tdy = tdy
    else:
        height, width = img.shape[:2]
        tdx = width / 2
        tdy = height / 2

    # X-Axis pointing to right. drawn in red  # x轴长度，红色指向右侧
    x1 = size * (cos(yaw) * cos(roll)) + tdx
    y1 = size * (cos(pitch) * sin(roll) + cos(roll) * sin(pitch) * sin(yaw)) + tdy

    # Y-Axis | drawn in green    # y轴长度，绿色指向下方
    #        v
    x2 = size * (-cos(yaw) * sin(roll)) + tdx
    y2 = size * (cos(pitch) * cos(roll) - sin(pitch) * sin(yaw) * sin(roll)) + tdy

    # Z-Axis (out of the screen) drawn in blue   # z轴长度，蓝色，指向屏幕外面
    x3 = size * (sin(yaw)) + tdx
    y3 = size * (-cos(yaw) * sin(pitch)) + tdy

    cv2.line(img, (int(tdx), int(tdy)), (int(x1),int(y1)),(0,0,255),3)
    cv2.line(img, (int(tdx), int(tdy)), (int(x2),int(y2)),(0,255,0),3)
    cv2.line(img, (int(tdx), int(tdy)), (int(x3),int(y3)),(255,0,0),2)

    return img