开题相关

Photon-Effificient Computational 3-D and Reflflectivity Imaging With Single-Photon Detectors

Dongeek Shin, Student Member, IEEE, Ahmed Kirmani, Student Member, IEEE, Vivek K Goyal, Fellow, IEEE,and Jeffrey H. Shapiro, Life Fellow, IEEE

 

Abstract—Capturing depth and reflectivity images at low light levels from active illumination of a scene has wide-ranging applications. Conventionally, even with detectors sensitive to individual photons, hundreds of photon detections are needed at each pixel to mitigate Poisson noise. We develop a robust method for estimating depth and reflectivity using fixed dwell time per pixel and on the order of one detected photon per pixel averaged over the scene. Our computational image formation method combines physically accurate single-photon counting statistics with exploitation of the spatial correlations present in real-world reflectivity and 3-D structure. Experiments conducted in the presence of strong background light demonstrate that our method is able to accurately recover scene depth and reflectivity, while traditional imaging methods based on maximum likelihood (ML) estimation or approximations thereof lead to noisier images. For depth, performance compares favorably to signal-independent noise removal algorithms such as median filtering or block-matching and 3-D filtering (BM3D) applied to the pixelwise ML estimate; for reflectivity, performance is similar to signal-dependent noise removal algorithms such as Poisson nonlocal sparse PCA and BM3D with variance-stabilizing transformation. Our framework increases photon efficiency 100-fold over traditional processing and also improves, somewhat, upon first-photon imaging under a total acquisition time constraint in raster-scanned operation. Thus, our new imager will be useful for rapid, low-power, and noise-tolerant active optical imaging, and its fixed dwell time will facilitate parallelization through use of a detector array.

摘要-----从场景的主动照明中获取低照度下的深度和反射率图像具有广泛的应用。通常情况下，即使是对单个光子敏感的探测器，也需要在每个像素上进行数百个光子探测以减轻泊松噪声。我们开发了一种稳健的方法来估计深度和反射率，使用固定的每像素停留时间和在场景上平均每像素一个检测光子的顺序。我们的计算图像形成方法结合了物理上精确的单光子计数统计，利用了现实世界反射率和三维结构中的空间相关性。实验表明，在强背景光下，我们的方法能够准确地恢复场景深度和反射率，而传统的基于最大似然(ML)估计或近似的成像方法导致噪声图像。对于深度，性能优于信号无关的噪声去除算法，如中值滤波或块匹配和3-D滤波(BM3D)应用于像素级ML估计;对于反射率，性能类似于信号依赖的噪声去除算法，如Poisson非局部稀疏PCA和带方差稳定变换的BM3D。与传统处理相比，我们的框架将光子效率提高了100倍，并在一定程度上改进了光栅扫描操作中总采集时间限制下的首次光子成像。因此，我们的新成像仪将用于快速、低功耗和耐噪声的主动光学成像，其固定的停留时间将通过使用探测器阵列促进并行化。

 

 

Index Terms—3-D imaging, computational imaging, convex optimization, first-photon imaging, LIDAR, low-light imaging, Poisson noise, single-photon detection, time-of-flight imaging.

三维成像、计算成像、凸优化、第一光子成像、激光雷达、微光成像、泊松噪声、单光子探测、飞行时间成像。

 

 

ACTIVE optical imaging methods measure properties of a scene using illumination provided by the system itself. Light detection and ranging (LIDAR) [1], also known as laser radar or LADAR, is a well-known example using periodically pulsed laser light. For each illuminated patch or pixel in an imaged scene, a LIDAR system builds a histogram of photon detection times, relative to the most recent pulse emission, over some number of pulses in a period called the dwell time. The time delay of this histogram, relative to the transmitted pulse’s temporal profile, is related through the speed of light to the depth (equivalently, distance, range, or 3D structure) of the scene. The amplitude of this histogram is related to the reflectivity of the scene. For accurate depth and reflectivity estimation, the signal acquisition time must be long enough to collect the100 to 1000photons per pixel (ppp) needed to generate a finely binned histogram for each pixel.

主动光学成像方法利用系统本身提供的照明来测量场景的特性。光探测和测距(LIDAR)[1]，也称为激光雷达或LADAR，是一个使用周期性脉冲激光光的著名例子。对于成像场景中的每个被照亮的斑块或像素，LIDAR系统建立一个光子探测时间的直方图，相对于最近的脉冲发射，在一个称为驻留时间的周期内的一些脉冲。这个直方图的时延，相对于发射脉冲的时间剖面，通过光速与场景的深度(等价地，距离，范围，或3D结构)相关。该直方图的振幅与场景的反射率有关。为了准确地估计深度和反射率，信号采集时间必须足够长，以收集每像素100到1000个光子(ppp)，为每个像素生成精细的分箱直方图。

 

In this paper, we address the problem of achieving high photon efficiency in combined 3D and reflectivity imaging. We expound upon a framework, introduced in [2], that builds upon an approach initiated in [3], [4]. Like the first-photon imaging (FPI) method of [3], our computational imager distinguishes itself from other previous work by avoiding the formation of histograms and instead using probabilistic modeling at the level of individual detected photons. This physically accurate modeling of single-photon detection is combined with exploitation of the spatial correlations present in real-world scenes to achieve accurate 3D and reflectivity imaging from on the order of 1 detected ppp averaged over the scene, despite significant noise from background light and dark counts.

在本文中，我们讨论了在三维和反射率组合成像中实现高光子效率的问题。我们详细阐述了[2]中引入的一个框架，它建立在[3]和[4]中启动的方法之上。就像[3]的第一光子成像(FPI)方法一样，我们的计算成象器通过避免直方图的形成，而是在单个检测光子的水平上使用概率建模，从而有别于以往的其他工作。这种物理上精确的单光子探测建模与现实场景中出现的空间相关性相结合，实现了精确的3D和反射率成像，从场景中检测到的ppp平均为1阶，尽管背景光和暗计数有显著的噪声。

 

 

 

High photon efficiency is important when very little backreflected light reaches the detector, as will be the case with low optical power relative to the imaging range [5]. More generally, increasing photon efficiency improves the trade-offs among optical power,imaging range,detector size,and imaging speed. The method introduced here uses a deterministic dwell time, which is both more convenient for raster scanning and amenable to parallelization through the use of a detector array. This ease of applicability comes with somewhat improved performance over FPI when compared at equal total acquisition times in raster-scanned operation.

当很少的后反射光到达探测器时，高光子效率是很重要的，这将是相对于成像范围[5]的低光功率的情况。一般来说，提高光子效率可以改善光功率、成像范围、探测器大小和成像速度之间的权衡。这里介绍的方法使用确定的停留时间，这是更方便的光栅扫描和通过使用探测器阵列并行化。与光栅扫描操作中相同的总采集时间相比，这种易于应用的特性在一定程度上改善了FPI的性能。

 

A. Prior Work

1) Active Imaging Methods: Active 3D imaging systems differ in how they modulate their transmitted power, leading to a variety of trade-offs in accuracy, modulation frequency, optical power, and photon efficiency; see Fig. 1 for a qualitative summary. Temporal modulation enables absolute (unaliased) distance measurement by the time-of-flight (TOF) principle. Examples of TOF acquisition systems, ordered by increasing modulation bandwidth (decreasing pulse duration), include homodyne TOF cameras [6], pulsed TOF cameras [7], and picosecond laser radar systems [8]. Spatial modulation techniques include structured light [9] and active stereo imaging [10]. These spatial-modulation techniques have low photon efficiencies because they use an always-on optical source, whereas pulsed-TOF systems have higher photon efficiencies because they use sources that are on only for short intervals. Additionally, the systems using temporal modulation have better absolute-distance accuracy than those using spatial modulation. The advantage of spatial modulation tends to be cheaper sensing hardware, since high-speed sampling is not required.

之前的工作

1)主动成像方法:主动3D成像系统的传输功率调制方式不同，导致在精度、调制频率、光功率和光子效率方面的各种权衡;定性总结见图1。时序调制可以通过飞行时间(TOF)原理实现绝对(无混叠)距离测量。通过增加调制带宽(减少脉冲持续时间)来订购的TOF捕获系统的例子包括零差TOF摄像机[6]、脉冲TOF摄像机[7]和皮秒激光雷达系统[8]。空间调制技术包括结构光[9]和主动立体成像[10]。这些空间调制技术具有较低的光子效率，因为它们使用的是始终开着的光源，而脉冲tof系统具有较高的光子效率，因为它们使用的光源只开很短的时间间隔。此外，采用时间调制的系统比采用空间调制的系统具有更好的绝对距离精度。空间调制的优点是便宜的传感硬件，因为不需要高速采样。

 

 图1所示。最先进的主动光学三维传感技术的定性比较。光子效率定义为厘米精确深度成像所必需的光子每像素(ppp)。

 

 

The most photon-efficient TOF imagers—those requiring the fewest photons for accurate imaging—use single-photon avalanche diode (SPAD) detectors [11]. Earlier efforts in SPAD-based 3D imaging from on the order of 1 detected ppp are reported in [12]–[14]. The framework presented here improves upon these works in part due to the use of estimated reflectivity. This translates to SPAD-based imagers with lower optical power and lower system bandwidth without sacrificing image quality. There also has been significant recent interest in compressive methods for 3D imaging, with [15]–[17] and without [18] single-photon detection. While compressive methods may reduce some measures of acquisition cost, they do not generally improve photon efficiency

光子效率最高的TOF成像仪——那些需要最少光子才能精确成像的成像仪——使用单光子雪崩二极管(SPAD)探测器[11]。早期在基于spad的3D成像中，以1为顺序检测到的ppp被报道在[12]-[14]中。这里提出的框架改进了这些工作，部分原因是使用了估计的反射率。这意味着基于spad的成象器具有更低的光学功率和更低的系统带宽，而不牺牲图像质量。最近，人们对三维成像的压缩方法也有了很大的兴趣，其中包括[15]-[17]和没有[18]单光子检测的压缩方法。压缩方法虽然可以降低一些获取成本，但通常不能提高光子效率。

 

 

2) Optoelectronic Techniques for Low Light Levels: In lowlight scenarios, a variety of optoelectronic techniques are employed to improve robustness. Active imagers use lasers with narrow spectral bandwidths and spectral filters to suppress background light and minimize the Poisson noise it creates. However, optical filtering alone cannot completely eliminate background light, and it also causes signal attenuation. Range-gated imaging [19] is another common technique, but this method requires a priori knowledge of object location. Furthermore, a SPAD may be replaced with a superconducting nanowire single-photon detector (SNSPD) [20], which is much faster,haslowertimingjitter,andhaslowerdark-countratethan a SPAD. However, SNSPDs have much smaller active areas and hence have narrower fields of view than SPAD-based systems with the same optics.

2)微光光电子技术:在微光情况下，采用多种光电子技术来提高鲁棒性。主动成象器使用窄光谱带宽的激光器和光谱滤波器来抑制背景光并将其产生的泊松噪声降至最低。但是光滤波本身不能完全消除背景光，还会造成信号衰减。距离门控成像[19]是另一种常用的技术，但该方法需要物体位置的先验知识。此外，SPAD可以被超导纳米线单光子探测器(SNSPD)[20]取代，它比SPAD快得多，具有更低的时间抖动和更低的暗度。然而，与基于spad的相同光学系统相比，snspd的活动区域要小得多，因此视野更窄。

 

 

3) Image Denoising: For depth imaging using SPAD data, it is typical to first form an image from a pixel-by-pixel estimate of scene depth using a time-inhomogeneous Poisson process model for photon detection times and then to apply an image denoising method that exploits the scene’s spatial correlations. As discussed inSection IV-B,even for anindividual pixel, maximum likelihood (ML) estimation is considered prohibitively complex and thus is typically replaced by a log-matched filter, which is an approximation to it. In this two-step approach of pixelwise estimation and denoising, one commonly assumes a Gaussian noise model for the output of the first step. This is empirically justified for high light levels [21] and also justified by the asymptotic normality of ML estimates with large numbers of data samples. However, at low light levels with significant background light, performing denoising well is more challenging due to the resulting high-variance uniform noise on the observed photon arrival times. In Section VI, we compare our technique with a state-of-the-art denoising method that uses block matching. The superior performance of our method is due in part to classification of photon detection events as being due to signal (backscattered light) or noise (unwanted background light and dark counts) prior to any depth image formation; classical pixelwise image formation yields an extremely challenging denoising problem. Note that denoising of reflectivity images is a better-developed field than denoising of depth images, but existing methods are not designed for the very low light levels and regularly-textured natural scenes considered in this paper.

3)图像去噪:深度成像利用SPAD数据,这是典型的先形成一个图像的像素估计场景深度使用time-inhomogeneous光子检测次泊松过程模型,然后应用图像去噪方法,利用场景的空间相关性。正如第IV-B节所讨论的，即使对于单个像素，最大似性(ML)估计也被认为是非常复杂的，因此通常被对数匹配滤波器所取代，这是它的近似。在像素估计和去噪的两步方法中，人们通常假设第一步的输出是高斯噪声模型。这是经验证明的高光照水平[21]，也证明了ML估计的渐近正态性与大量的数据样本。然而，在有明显背景光的低光水平下，由于观测到的光子到达时间会产生高方差的均匀噪声，所以要想实现良好的去噪是比较困难的。在第六节中，我们将我们的技术与使用块匹配的最先进的去噪方法进行比较。我们方法的优越性能部分是由于光子检测事件的分类是由于信号(后向散射光)或噪声(不需要的背景光和暗计数)在任何深度图像形成之前;经典的像素图像形成产生了一个极具挑战性的去噪问题。值得注意的是，与深度图像去噪相比，反射率图像去噪是一个发展得更好的领域，但现有的方法不适用于本文考虑的非常低的光线水平和规则纹理的自然场景。

 

 

4) First-Photon Imaging: First-photon imaging (FPI) [3] is a method that forms 3D and reflectivity images using only the first detected photon at every pixel in a raster-scanned scene. FPI combines accurate first-photon detection statistics with the spatial correlations existing in natural scenes to achieve robust low-light imaging. The use of the first detection time in FPI, however, makes the dwell time at each pixel a random variable. Thus, FPI does not extend naturally from raster-scanned data collection to the use of SPAD arrays—since simultaneous measurement implies equal dwell times—thus precluding the dramatic speedup in image acquisition that such arrays enable。

4)第一光子成像:第一光子成像(FPI)[3]是一种仅使用光栅扫描场景中每个像素的第一检测光子形成三维和反射率图像的方法。FPI将精确的首次光子检测统计数据与自然场景中存在的空间相关性相结合，以实现鲁棒的弱光成像。然而，在FPI中使用第一次检测时间使每个像素的停留时间成为一个随机变量。因此，FPI不能自然地从光栅扫描数据采集扩展到SPAD阵列的使用，因为同时测量意味着相同的停留时间，因此排除了这种阵列在图像采集方面的显著加速。

 

In this paper, we develop models and methods analogous to FPI that apply when there is a fixed dwell time at each pixel. In the experimental configuration depicted in Fig. 2, we demonstrate that the performance of the new method is similar to or slightly better than FPI when compared for equal total acquisition time in raster-scanned operation. Furthermore, with an M-fold increase in laser power and an M-element SPAD array,our fixed dwell-time framework can provide this same robust imaging M times faster than a single-detector raster-scanned system.

在本文中，我们发展了类似于FPI的模型和方法，适用于每个像素处有固定驻留时间的情况。在图2所示的实验配置中，我们证明了在光栅扫描操作中，当总采集时间相等时，新方法的性能与FPI相似或略好于FPI。此外，由于激光功率增加了M倍，并采用了M单元SPAD阵列，我们的固定驻留时间框架可以提供同样稳定的成像，比单探测器光栅扫描系统快M倍。

 

图2所示。在[3]中使用随机停留时间和在这里使用恒定停留时间的实验成像设置。脉冲光源以光栅扫描模式照亮场景。后向散射光由时间分辨的单光子探测器收集。每个空间位置用精确的N个光脉冲(固定驻留时间)照明。白炽灯注入背景光，破坏承载信息的信号。在每个图像像素上记录光子检测次数和总光子计数。该数据集用于估算三维结构和反射率。该设置类似于具有一个泛光灯照明源和一个在固定驻留时间工作的单光子计数探测器阵列。

 

        B Main Contributions

1) Modeling: We introduce a physically accurate model for the signal produced by a SPAD under low-light conditions that incorporates an arbitrary illumination pulse shape, background (ambient) light contribution, dark counts, and the inhomogeneous Poisson process characteristics (shot noise from the quantum nature of light) given a fixed dwell time. The same model for a single illumination pulse was used in [3] (with limited explanation); while [3] used a random number of pulses, the analysis in this paper is for a fixed number of pulses.

        主要贡献

1)建模:我们引入了SPAD在弱光条件下产生的信号的物理精确模型，该模型包含了任意照明脉冲形状、背景(环境)光贡献、暗计数和给定固定停留时间的非均匀泊松过程特性(光量子特性的散粒噪声)。[3]中使用了单个照明脉冲的相同模型(解释有限);[3]使用的是随机脉冲数，而本文的分析是针对固定脉冲数。

 

 

2) Algorithmic: We provide a method for computational reconstruction of depth and reflectivity from noisy photondetection data. Our technique combines a shot-noise model for single-photon detection with simple means to exploit the high degree of spatial correlation present in real-world scenes. The modularity of the technique—combining spatial regularization for reflectivity, classification of detections as due to signal or noise, and spatial regularization for depth—makes it amenable to the generation of algorithmic variations to exploit more sophisticated spatial correlation models.

2)算法:我们提供了一种从噪声光子检测数据中计算重建深度和反射率的方法。我们的技术将单光子检测的散粒噪声模型与简单的方法相结合，以利用现实场景中存在的高度空间相关性。该技术的模块化——结合了用于反射率的空间正则化、由于信号或噪声引起的检测分类以及用于深度的空间正则化——使其易于生成算法变化以开发更复杂的空间相关模型。

 

 

 

3) Experimental: We experimentally demonstrate that our proposed 3D imager’s photon efficiency is more than 100 times higher than that of the conventional log-matched filter, which is awell-known proxy for pixelwise ML estimation.We also show that our 3D imager achieves sub-pulse-width depth resolution under short acquisition times, in which 54% of the pixels have missing data (no photon detections), and at high background levels, when any given photon detection has approximately probability 0.5 of originating from ambient light.

3)实验:我们通过实验证明，我们提出的三维成象器的光子效率比传统的对数匹配滤波器高100倍以上，传统的对数匹配滤波器是像素级ML估计的最佳替代。我们还表明，我们的3D成象器在较短的采集时间内实现了次脉冲宽度深度分辨率，其中54%的像素有丢失数据(没有光子检测)，并且在高背景水平下，当任何给定的光子检测大约有0.5来自环境光的概率。

 

 

The remainder of the paper is organized as follows. Section II introduces the LIDAR-like imaging configuration that we consider. The key probabilistic models for the measured data are derived in Section III. These models are related to conventional image formation in Section IV, and they are the basis for the novel image formation method in Section V. Section VI presents experimental results for the novel method, and Section VII provides additional discussion and conclusions. An appendix presents performance bounds for pixelwise estimators based on our modeling.

本文的其余部分组织如下。第二节介绍了我们所考虑的类似lidar的成像配置。第三节推导了测量数据的关键概率模型。这些模型都与第四节中的常规成像有关，是第五节中新的成像方法的基础。第六节给出了新方法的实验结果，第七节提供了额外的讨论和结论。附录给出了基于我们建模的像素级估计器的性能界限。

 

        C. Outline

The methods detailed in this paper were presented in preliminary, abbreviated form in [2]. The present manuscript provides additional context (Section I-A), details on derivations (Section III), performance bounds (Appendix), and experimental results that do not appear in [2] (Section VI). In particular, comparisons to Poisson non-local sparse PCA (NLSPCA) [22] and block-matching and 3D filtering (BM3D) [23]—with variance-stabilizing transform (VST) for reflectivity [24]— replace the use of less-sophisticated bilateral filtering [25] in the preliminary work.

　　C   大纲

本文详细介绍的方法在[2]中以初步的，缩写的形式给出。本手稿提供了附加的上下文(第I-A节)、推导的细节(第III节)、性能边界(附录)和[2]中没有出现的实验结果(第VI节)。特别是，与泊松非局部稀疏PCA (NLSPCA)[22]和块匹配3D滤波(BM3D)[23] -结合方差稳定变换(VST)对反作用[24]进行比较，在前期工作中替代了较为简单的双边滤波[25]。

 

 

 

 

 

    B. Detection

A SPAD detector provides time-resolved single-photon detections [11], called clicks. Its quantum efficiency η is the fraction of photons passing through the pre-detection optical filter that are detected. Each detected photon is time stamped within a time bin of duration measuring a few picoseconds. We will assume that this quantization of detection times is much shorter than Tp and thus negligible; i.e., we treat the detection times as continuous-valued variables.

　　B检测

SPAD探测器提供了时间分辨的单光子检测[11]，称为点击。它的量子效率η是通过探测前光学滤波器被探测到的光子的比例。每个被探测到的光子都在一个持续时间为几皮秒的时间箱内打上时间戳。我们将假设这种检测时间的量化比Tp短得多，因此可以忽略不计;也就是说，我们将检测时间视为连续值变量

 

 

 

A SPAD detector is not number-resolving, meaning that it reports at most one click in a short period of time. This is because SPAD detectors have a reset time or dead time after each click, during which there is no sensitivity to incident light. Here, we consider low-flux imaging where the probability of multiple clicks within one repetition period of duration Trwould be negligible even without reset time

SPAD检测器不进行数字解析，这意味着它在短时间内最多报告一次点击。这是因为SPAD探测器在每次点击后都有一个重置时间或死时间，在此期间对入射光不敏感。这里，我们考虑低通量成像，在一个持续时间为tri的重复周期内，即使没有重置时间，多次点击的概率也可以忽略不计

 

 

 C 数据采集 

每个像素(i, j)被N个激光脉冲序列照亮。我们的方法适用于N在10到100的数量级。因此，总停留时间为Ta= NTr。为了检测环境光和暗计数产生的显著噪声，我们还在工作光波长λ处照射光子通量bλ的背景光到检测器上。对于每个像素，我们记录光子探测总数ki,j，以及它们的探测次数{t(?) i,j}ki,j ?=1，其中后者是相对于前一个发射脉冲测量的。

 

 

 

               III. PROBABILISTIC MEASUREMENT MODEL

Illuminating pixel (i, j) with the pulse s(t) results in backreflected light with photon flux【 ri,j(t) = αi,js(t − 2zi,j/c) + bλ】 at the detector. The measurement of photon flux is through photon detections, and carefully modeling the relationships between the measured quantities and the reflectivity and depth variables is central to our imaging method.

　　　　III. 概率度量模型

用脉冲s(t)照亮像素(i, j)在探测器处产生光子通量为【ri,j(t) = αi,js(t−2zi,j/c) + bλ】的后反射光。光子通量的测量是通过光子探测来实现的，而仔细地模拟被测量与反射率和深度变量之间的关系是我们成像方法的核心。