Proj CDeepFuzz Paper Reading: DeepStellar: Model-based quantitative analysis of stateful deep learning systems

Abstract

背景：关于test RNN-based stateful system的研究较少

本文：DeepStellar
Method:

1. model RNN as an abstract state transition system
2. design 2 trace similarity metrics and 5 coverage criteria
3. STSim: state-based similarity
4. TTSim: transition-based similarity
5. BSCov: basic state coverage
6. n-SBCov: n-Step Basic Boundary Coverage
7. WSCov: Weighted State Coverage
8. BTCov: Basic Transition Coverage
9. WTCov: Weighted Transition Coverage
10. adversarial detection classifier(linear regression classifier), coverage guided test
11. represent RNN as Discrete-Time Markov Chain to capture its statistical behaviors

实验
Competitors: FGSM, BIM, DeepFool
datasets: DeepSpeech 0.1.1, DeepSpeech 0.3.0, MNIST-LSTM, MNIST-GRU
结果：

1. the similarity metrics could effectively capture the differences between samples with very small perturbations(Q: why neeed to capture the difference in perturbations?)
2. the coverage criteria are useful in revealing erroneous behaviours.