PETR- Position Embedding Transformation for Multi-View 3D Object Detection

旷视 ECCV 2022

纯视觉BEV方案transformer网络3D检测

paper：[2203.05625] PETR: Position Embedding Transformation for Multi-View 3D Object Detection

code：GitHub - megvii-research/PETR: [ECCV2022] PETR: Position Embedding Transformation for Multi-View 3D Object Detection & [ICCV2023] PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images

目标：环视相机的2D特征，加3D位置编码，转成3D表征

1. 相机视椎空间离散化成栅格坐标点
2. 坐标点用相机参数转到自车空间3D坐标
3. 从相机抽特征，加上3D坐标，作为位置encoder输入，输出带3D位置的特征
4. 上面特征输入transformer decoder，和object query交互，输出检测结果

和DETR3D区别

DETR3D：

Detr3DHead__init__self.query_embedding = nn.Embedding(self.num_query, self.embed_dims * 2)forwardquery_embeds = self.query_embedding.weighths, init_reference, inter_references = self.transformer(mlvl_feats,query_embeds,reg_branches=self.reg_branches if self.with_box_refine else None,  # noqa:E501img_metas=img_metas,)Detr3DTransformer__init__self.embed_dims = self.decoder.embed_dimsself.reference_points = nn.Linear(self.embed_dims, 3)forward(self, mlvl_feats, query_embed, reg_branches=None, **kwargs):query_pos, query = torch.split(query_embed, self.embed_dims , dim=1)query_pos = query_pos.unsqueeze(0).expand(bs, -1, -1)reference_points = self.reference_points(query_pos).sigmoid()

PETR:

PETRHead__init__num_query=900self.embed_dims = 256self.reference_points = nn.Embedding(self.num_query, 3)nn.init.uniform_(self.reference_points.weight.data, 0, 1)self.query_embedding = nn.Sequential(nn.Linear(self.embed_dims*3//2, self.embed_dims),nn.ReLU(),nn.Linear(self.embed_dims, self.embed_dims),)forwardreference_points = self.reference_points.weightquery_embeds = self.query_embedding(pos2posemb3d(reference_points))reference_points = reference_points.unsqueeze(0).repeat(batch_size, 1, 1) #.sigmoid()outs_dec, _ = self.transformer(x, masks, query_embeds, pos_embed, self.reg_branches)PETRDNTransformer__init__forward(self, x, mask, query_embed, pos_embed, attn_masks=None, reg_branch=None)query_embed = query_embed.transpose(0, 1)  # [num_query, dim] -> [num_query, bs, dim]out_dec = self.decoder(query=target,key=memory,value=memory,key_pos=pos_embed,query_pos=query_embed,key_padding_mask=mask,attn_masks=[attn_masks, None],reg_branch=reg_branch,)

组成

1. 生成3D坐标
   1. 输入：栅格坐标点各相机外参
      1. 用CaDDN的LID采样空间
   2. 输出：3D坐标点
   3. 过程：
      1. 计算3D坐标点
      2. min-max正则化
2. 3D位置encoder
   1. 输入：2D特征，3D坐标点
   2. 输出：3D特征
   3. 过程：
      1. 3D坐标点输入MLP编码，输出3D位置embedding
      2. 2D特征输入1*1卷积，和3D位置embedding相加，得到3D特征
         1. 提特征：ResNet、SwinTR、VoVNetV2
      3. flatten 3D特征
3. object query
   1. 输入：参考点 nn.Embedding(self.num_query, 3) PETRHead.reference_points
   2. 输出：object query Q0  query_embeds
      1. 过程：正态分布初始化，余弦位置编码(pos2posemb3d)，MLP（PETRHead.query_embedding）
4. decoder
   1. 包含mha，ffn，训练中object query提取出障碍物高维特征
5. head，loss函数
   1. 分类头：出障碍物类别，focal loss
   2. 框头：输出相对于参考点的偏移量，L1 loss
   3. Hungarian匹配gt
   4. ，和DETR3D一样