Proj CDeepFuzz Paper Reading: PyTorch: an imperative style, high-performance deep learning library

Abstract

本文： PyTorch
Task: detail the implementation and architecture of PyTorch
Github: https://github.com/pytorch/pytorch
特点：

1. PyTorch同时关注可用性和速度
2. PyTorch provides an imperative and Pythonic programming style

1. Intro

2. Background

3. Design Principles

Be Pythonic, Put researchers first, Provide pragmatic performance, Worse is better

4. Usability centric design

4.1 Deep learning models are just Python programs

4.2 Interoperability and extensibility

4.3 Automatic differentiation

5 Performance focused implementation

5.1 An efficient C++ core

5.2 Separate control and data flow

PyTorch is designed to execute operators asynchronously on GPU by leveraging the CUDA stream mechanism [38] to queue CUDA kernel invocations to the GPUs hardware FIFO. This allows the system to overlap the execution of Python code on CPU with tensor operators on GPU

5.3 Custom caching tensor allocator

5.4 Multiprocessing

5.5 Reference counting

6. Evaluation

6.1 Asynchronous dataflow

6.2 Memory management

6.3 Benchmarks

6.4 Adoption

7. Conclusion and future work