Qianqian Fang

Qianqian Fang

Auto-generated profile, based on publicly available information

Qianqian Fang

Northeastern University, Department of Bioengineering, Boston, Massachusetts, United States

Research Focus & Metrics

Academic Career Length

21Years (2006-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
26 January 2018 in Journal of Biomedical Optics
Verified
Journal Article
Open access
PDF file
LYLeiming Yu
Leiming Yu
FNFanny Nina-Paravecino
Fanny Nina-Paravecino
David KaeliDavid Kaeli
David Kaeli
QFQianqian Fang
Qianqian Fang
We present a highly scalable Monte Carlo (MC) three-dimensional photon transport simulation platform designed for heterogeneous computing systems. Through the development of a massively parallel MC algorithm using the Open Computing Language framework, this research extends our existing graphics processing unit (GPU)-accelerated MC technique to a highly scalable vendor-independent heterogeneous computing environment, achieving significantly improved performance and software portability. A number of parallel computing techniques are investigated to achieve portable performance over a wide range of computing hardware. Furthermore, multiple thread-level and device-level load-balancing strategies are developed to obtain efficient simulations using multiple central processing units and GPUs. Yu, Nina-Paravecino, Kaeli, and Fang: Scalable and massively parallel Monte Carlo photon transport simulations for heterogeneous computing platforms The Monte Carlo (MC) method has been widely regarded as the gold standard for modeling light propagation inside complex random media, such as human tissues. MC, however, suffers from low computational efficiency because a large number of photons have to be simulated to achieve the desired solution quality. Sequential MC simulations require extensive computation and long runtimes, easily taking up to several hours.1,2 In recent years, studies on massively parallel MC algorithms have successfully reduced this computational cost down to seconds or minutes, due largely to the “embarrassingly parallelizable” nature of MC and the rapid adoption of low-cost many-core processors such as general-purpose graphics processing units (GPUs). Alerstam et al.3 first reported a proof-of-concept using GPUs to accelerate MC in a homogeneous domain. In 2009, Fang and Boas4 reported the first GPU-accelerated MC algorithm to model light transport inside a three-dimensional (3-D) heterogeneous domain, and released an open-source tool—Monte Carlo eXtreme (MCX). Nearly all GPU-based MC photon transport frameworks reported in the literature,3 4. 5. 6. – 7 including MCX, have been written exclusively using the CUDA programming model developed by NVIDIA.8 Because CUDA is specifically targeted for NVIDIA GPUs, most existing GPU MC codes cannot be executed on a central processing unit (CPU) or a high-performance GPU made by other manufacturers. In recent years, a generalized parallel computing solution—Open Computing Language (OpenCL)—has emerged.9 OpenCL was designed to target scalability and portability in high-performance computing. The OpenCL specification defines an open-standard parallel programming language for multicore CPUs, GPUs, and field-programmable gate arrays (FPGAs). It specifies a set of generalized programming interfaces to efficiently utilize the computing resources of dissimilar processors.9 A program written with the OpenCL computing model can be natively executed on cross-vendor processors, including not only NVIDIA GPUs but also Intel and AMD CPUs and GPUs. Furthermore, the OpenCL model employs a just-in-time (JIT) compilation model for parallel code execution.10 The OpenCL JIT compiler translates the source code, referred to as a “kernel,” to device assembly at runtime. This allows processor-specific optimizations to be applied to the code, achieving improved portability and efficiency. This work aims to improve and generalize our previously developed massively parallel photon transport simulation platform through the adoption of a heterogeneous computing framework using the OpenCL programming model. The generalized algorithm permits users to launch efficient photon transport simulations on not only NVIDIA GPUs but also CPUs, GPUs, and systems-on-a-chip processors, made by many vendors.11 The porting of MCX CUDA kernels to the OpenCL programming framework is relatively straightforward. A diagram of the generalized MC algorithm (MCX-CL) is shown in Fig. 1. The simulations start on the host (a CPU) by processing the user’s inputs. The host then decides on how to partition the total number of simulated photons modeled based on the targeted hardware characteristics to best leverage multiple computing devices (see below). The photon simulation kernel is then dynamically compiled by the OpenCL’s JIT compiler for each device. The simulation parameters, including domain settings, optical properties, and independent random number seeds for each thread, are allocated, initialized, and copied to each device. Once this preparation step is complete, the host instructs all activated devices to start photon transport simulations simultaneously. Each computing device launches a specific number of parallel computing threads, determined by the respective hardware settings (discussed below). Within each computing thread, a photon simulation loop, (Fig. 1 in Ref. 4) is carried out. The host waits for all devices to complete and then reads the solutions (3-D fluence and detected photon data) back to the host memory. Postprocessing is then performed to yield the final solution. <!-- named anchor --> Fig. 1 Generalized parallel Monte Carlo photon transport simulation workflow for heterogeneous systems. Several observations have been made during the implementation of MCX-CL. On a heterogeneous system, the JIT compiler and the execution library of different devices are independently implemented by their respective vendors. As a result, the same kernel may exhibit different execution behaviors on different OpenCL implementations. For example, using the AMD OpenCL implementation, multiple MCX kernels launched in the command queue are executed asynchronously (i.e., in parallel). With the NVIDIA OpenCL library, however, kernels in the same command queue are serialized. One has to launch multiple threads in order to use multiple NVIDIA devices in parallel. Moreover, the AMD OpenCL library supports both AMD GPUs and CPUs, but the NVIDIA OpenCL implementation only supports NVIDIA GPUs. Another caveat of OpenCL is the lack of intrinsic atomic operations for floating-point numbers. A workaround has been proposed12 when such operations are desired. Next, we characterize and optimize MCX-CL simulation performance across a range of devices, including CPUs and GPUs produced by Intel, NVIDIA, and AMD. We first run a profile-based analysis on a small set of selected devices, generalize the observations, and then develop a number of optimization strategies that can deliver portable performance to a wide range of devices. For profiling, we use AMD’s CodeXL toolset for AMD CPUs and GPUs and VTune Amplifier for Intel CPUs/GPUs. A number of observations can be made from our profiling results. First, the MCX-CL kernel is compute-intensive. On an AMD R9 Nano GPU, 91 million computing instructions are executed when running a benchmark problem; in comparison, only 0.5 million memory instructions were executed. Second, the number of parallel threads that MCX-CL can launch and execute is bounded by the available register space—the fastest memory in the device. For the AMD R9 Nano GPU, the available “vector register” space can only accommodate up to 768 threads (divided into 12 × 64-thread groups; a 64-thread group is referred to as a “wavefront” in AMD’s architecture) to run simultaneously inside a compute unit (CU, also called a “multiprocessor” in NVIDIA literature). The third observation is that the complex workflow of the MC simulation algorithm results in 62% “thread divergence,” which means that 62% of the time, only a subset of the threads inside a wavefront is executing instructions—caused by the presence of if/then/else branches. Because all 64 threads inside a wavefront are designed to execute instructions in lock-step fashion (single-instruction multiple threads), in the event that a subset of the threads need to take a different execution path, the wavefront has to be serialized, resulting in low execution efficiency. With these key characteristics in mind, we have implemented multiple optimization strategies to maximize MCX-CL’s simulation efficiency. First, to make the mathematical computation more efficient, we have utilized the “native” math functions—a set of functions with hardware-dependent accuracy provided by the OpenCL library (referred to Opt1). Second, to better utilize the available computing resources (in particular, register space), we have developed an automatic algorithm to calculate a “balanced” number of threads to ensure that all available registers are occupied (referred to Opt2). Generally speaking, a low thread number can result in low utilization of computing resources, whereas an excessively high number of threads can result in overhead due to frequent switches between queued thread blocks. An optimized thread number can balance resource utilization to address both inefficiencies. In this work, this thread number is estimated by multiplying the maximum concurrent threads per CU with the available CUs on the GPU. Additionally, we simplify the control flow of the kernel (referred to Opt3), aiming to reduce thread divergence. This is also expected to reduce the complexity of the kernel, providing the JIT compiler with a better chance to optimize the execution and allocate fewer registers. In Fig. 2, we report the MCX-CL simulation speed (in photons/ms) for three benchmarks (B1, B2, and B2a), before and after applying the aforementioned optimization strategies. Our baseline simulation is configured with a fixed thread number (N=214) and a workgroup size of 64. All three benchmarks simulate 108 photons inside a 60×60×60  mm3 domain, with an absorption coefficient μa=0.005  mm−1, a scattering coefficient μs=1.0  mm−1, an anisotropy g=0.01, and a refractive index n=1.37. The medium outside of the cube is assumed to be air. In B2 and B2a, a spherical inclusion (μa=0.002  mm−1, μs=5.0  mm−1, g=0.9, n=1.0) of radius 15 mm is placed at the center of the cube. In B1, a photon is terminated when it arrives at the cube’s boundary; in B2 and B2a, a reflection calculation is performed at the sphere and cube boundaries based on Snell’s law. The difference between B2 and B2a is that B2a applies atomic operations to avoid data races when accumulating the fluence rate in each voxel, whereas B2 uses nonatomic floating-point additions.4 In all three cases, a pencil beam along +z-axis enters the domain at (30, 30, 0) mm. Each speed value reported is obtained by running three simulations and selecting the highest speed. All tests were performed on Ubuntu 14.04, using the nvidia-375 driver for NVIDIA GPUs, amdgpu-pro 16.30.3 for AMD GPUs, and opencl-1.2-6.2 for Intel CPUs and GPUs. All simulations are verified to produce correct solutions. For comparison purposes, we also run the B2a benchmark using MCX (implemented in CUDA) on all NVIDIA GPUs. A speed comparison between MCX-CL and MCX is shown as an inset in Fig. 2. <!-- named anchor --> Fig. 2 The MCX-CL simulation speed (photons/ms) on different computing devices after applying three optimization schemes: Opt1: using hardware-native math library; Opt2: using optimized thread configuration; and Opt3: reducing thread divergence. The throughputs in the B1, B2, and B2a benchmarks are shown as a stacked-bar and the four bars for each hardware are baseline (•), Opt1 (+), Opt1+2 (×), Opt1+2+3 (#), displayed from left to right. The inset shows the speed comparison between the OpenCL and CUDA versions of the algorithm on NVIDIA GPUs. From Fig. 2, the first two optimization techniques consistently produced faster simulations, although the magnitude of the improvements vary from device to device. The acceleration due to hardware-optimized math functions (Opt1) yielded some decent speedup on AMD GPUs (7% to 12%), Intel GPUs and CPUs (11% to 17%), and a smaller improvement on NVIDIA GPUs (3% to 10%). Combining Opt1 with Opt2 (i.e., optimized thread/workgroup size), we have observed a significant improvement for the AMD GPUs (63% to 74%), along with a moderate improvement for Intel and AMD CPUs (12% to 21%); the speed of NVIDIA GPUs is also noticeably improved (6% to 12%). However, the results when applying simplification of control flow (i.e., Opt3) are mixed—for some NVIDIA GPUs (1080Ti, 980Ti, Titan X, 1050Ti), a noticeable speedup was observed; for two other NVIDIA GPUs (1080, 590) and all AMD and Intel CPUs/GPUs, we encountered a minor reduction in speed (1% to 7%). We want to note that the GTX 1050Ti experienced a 1.9× speedup with the help of Opt3. We believe the variation in speedup when applying control flow simplification is a result of the complex interplay between kernel complexity and compiler heuristics when optimizing the kernel. Nevertheless, the advantage of using a GPU over a CPU in photon transport simulation is clear. The AMD RX Vega 64 GPU performs 23× faster in B2/B2a tests compared to using the dual-Xeon E5-2658v3 CPUs (with all 48 CUs), 61× faster than the i7-7700k CPU (8 CUs) and 42× to 49× faster than Ryzen 1700X CPU (16 CUs). Moreover, comparing the runtimes between B2 and B2a on different devices, the average overhead due to atomic operations is only 5% for all NVIDIA GPUs newer than GTX 590, which experiences a 138% overhead; the average overhead is 31% and 46% for AMD and Intel GPUs, respectively. It is interesting to note that the B2a benchmark actually runs faster (∼20%) than the B2 test on the NVIDIA 980Ti and Titan X (both belong to the “Maxwell” architecture) when all three optimizations are used. We believe this is related to architecture-specific compiler optimizations. We also estimate the throughput per core (i.e., a stream processor in a GPU or a physical core in a CPU) and throughput per watt for all tested devices using the B1 benchmark with Opt1 and Opt2. Without surprise, CPUs report a significantly higher per-core performance than GPUs (256 and 143  photons/ms/core for i7-7700K and Ryzen 1700X, respectively, comparing to 115  photons/ms/core for Intel HD 520 GPU and on average 9 and 6  photons/ms/core for AMD and NVIDIA GPUs, respectively). This suggests that although CPUs have more powerful cores, GPUs excel in MC simulations with many less powerful cores. For the throughput per watt calculations, we divide the throughput by the thermal design power of each processor. The Intel HD 520 GPU reports the highest power efficiency at 184  photons/ms/W, followed by AMD (145  photons/ms/W) and NVIDIA (60  photons/ms/W) GPUs; that for the CPU is between 11 and 12  photons/ms/W. From the inset in Fig. 2, it appears that the CUDA-based MC is 2.1× to 5.4× faster than the OpenCL version on NVIDIA GPUs, except for the GTX 1050Ti. It is well-known that NVIDIA does not fully support OpenCL, as the current driver lacks the latest features supported by the hardware, such as...

30 August 2022 in Neurophotonics
Verified
Journal Article
Open access
PDF file
Hasan AyazHasan Ayaz
Hasan Ayaz
Wesley BakerWesley Baker
Wesley Baker
GBGiles Blaney
Giles Blaney
DBDavid A. Boas
David A. Boas
Heather BortfeldHeather Bortfeld
Heather Bortfeld
WZWenjun Zhou
Wenjun Zhou
We have come a long way since Jöbsis first demonstrated that noninvasive cerebral monitoring was possible in humans within the near-infrared light window (approximately 700 to 1000 nm) in the late 1970s.1 What started as a tool to quantify blood oxygenation in the brain has exploded into a whole field of study that aims to exploit this window to reveal a battery of information related to human brain physiology, including brain metabolism, blood flow, autoregulation, perfusion pressure, water concentration, light scattering, exogenous contrast agents, and more. In this text, we use the term “diffuse optics” to broadly describe any technique that employs a near-infrared light source and a photodetector to capture multiply scattered photons in the diffusion limit (i.e., in the limit wherein the absorption coefficient, μa, is much less than the reduced scattering coefficient, μs′, and the source-detector separation is much greater than the photon mean free path). The term “diffuse” has historical connotations in the field and reflects how near-infrared photons propagate through biological tissue. However, it should be noted that this term is in no way associated with how the field/community is organized or conducts its science. On the contrary, the breadth and depth of the progress made in the field over the past decade could only be accomplished by a cohesive, focused, and tight-knit community of researchers!

22 February 2020 in Neurophotonics
Verified
Journal Article
Open access
PDF file
ATAnh Phong Tran
Anh Phong Tran
SYShijie Yan
Shijie Yan
QFQianqian Fang
Qianqian Fang
Functional near-infrared spectroscopy (fNIRS) has played an increasingly important role in functional neuroimaging.1 Using light in the red and near-infrared range, the hemodynamic response of the brain is probed through careful placement of sources and detectors on the scalp surface at multiple wavelengths. Relative changes in oxygenated (HbO2) and deoxygenated hemoglobin (HbR) concentrations, as a result of neural activities, lead to variations in light intensities at the detectors that are used to infer the locations of these activities. The accuracy of this inference depends greatly not only upon an accurate representation of the complex human brain anatomy but also on the surrounding tissues that affect the migrations of photons from the sources to the detectors. Using the anatomical scans of the patient’s head or a resembling atlas, Monte Carlo (MC) simulations are often used in conjunction with tissue optical properties to approximate the photon pathlengths that are used to reconstruct the changes in HbR and HbO2. While much simplified brain models, such as planar2,3 or spherical layers,4 as well as approximated photon propagation models, such as the diffusion approximation (DA),5 have been widely utilized by the research community, their limitations are recognized by a number of studies.3,6 In addition, modeling brain anatomy accurately also plays important roles in other quantitative neuroimaging modalities, such as electroencephalography (EEG)7 and magnetoencephalography.8

4 September 2018 in Biomedical Optics Express
Verified
Journal Article
Open access
PDF file
RYRuoyang Yao
Ruoyang Yao
XIXavier Intes
Xavier Intes
QFQianqian Fang
Qianqian Fang
Perturbation Monte Carlo (pMC) has been previously proposed to rapidly recompute optical measurements when small perturbations of optical properties are considered, but it was largely restricted to changes associated with prior tissue segments or regions-of-interest. In this work, we expand pMC to compute spatially and temporally resolved sensitivity profiles, i.e. the Jacobians, for diffuse optical tomography (DOT) applications. By recording the pseudo random number generator (PRNG) seeds of each detected photon, we are able to “replay” all detected photons to directly create the 3D sensitivity profiles for both absorption and scattering coefficients. We validate the replay-based Jacobians against the traditional adjoint Monte Carlo (aMC) method, and demonstrate the feasibility of using this approach for efficient 3D image reconstructions using in vitro hyperspectral wide-field DOT measurements. The strengths and limitations of the replay approach regarding its computational efficiency and accuracy are discussed, in comparison with aMC, for point-detector systems as well as wide-field pattern-based and hyperspectral imaging systems. The replay approach has been implemented in both of our open-source MC simulators - MCX and MMC (http://mcx.space)

7 February 2019 in Neurophotonics
Verified
Journal Article
Open access
PDF file
PCPaolo Cassano
Paolo Cassano
ATAnh Phong Tran
Anh Phong Tran
HKHusam Katnani
Husam Katnani
BBBenjamin S. Bleier
Benjamin S. Bleier
Michael R. HamblinMichael R. Hamblin
Michael R. Hamblin
QFQianqian Fang
Qianqian Fang
Brain photobiomodulation (PBM) with near-infrared (NIR) and red light is a growing field of research.1,2 Over-the-counter light-emitting diodes (LEDs), as well as in-office laser devices, are already used off-label for transcranial photobiomodulation (t-PBM) in patients with neuropsychiatric disorders.3 Given the off-label use of t-PBM, only patients who did not respond to, or did not tolerate FDA-approved (or evidence-based) treatments are typically referred to t-PBM. The use of t-PBM devices is expected to increase as several studies are being conducted to test the efficacy and tolerability of t-PBM for neuropsychiatric disorders; specific FDA indications for the use of t-PBM might also arise. A wide range of on-going or concluded t-PBM clinical trials, including those targeting major depressive disorder (MDD) (NCT02898233, NCT02959307), generalized anxiety disorder (NCT03420456), dementia (NCT03160027), and traumatic brain injury (TBI) with posttraumatic stress disorder (NCT02356861), have been reported in public databases.4

8 October 2020 in Biomedical Optics Express
Verified
Journal Article
Closed access
PDF file
SYShijie Yan
Shijie Yan
QFQianqian Fang
Qianqian Fang
Over the past decade, an increasing body of evidence has suggested that three-dimensional (3-D) Monte Carlo (MC) light transport simulations are affected by the inherent limitations and errors of voxel-based domain boundaries. In this work, we specifically address this challenge using a hybrid MC algorithm, namely split-voxel MC or SVMC, that combines both mesh and voxel domain information to greatly improve MC simulation accuracy while remaining highly flexible and efficient in parallel hardware, such as graphics processing units (GPU). We achieve this by applying a marching-cubes algorithm to a pre-segmented domain to extract and encode sub-voxel information of curved surfaces, which is then used to inform ray-tracing computation within boundary voxels. This preservation of curved boundaries in a voxel data structure demonstrates significantly improved accuracy in several benchmarks, including a human brain atlas. The accuracy of the SVMC algorithm is comparable to that of mesh-based MC (MMC), but runs 2x-6x faster and requires only a lightweight preprocessing step. The proposed algorithm has been implemented in our open-source software and is freely available at http://mcx.space.