Acceleration
- Python list $\rightarrow$ Numpy array
- CPU
- bandwidth (register size, how much data over time)
- latency (time it takes)
- throughput (amount of data)
- Complexity
- Compression (space complexity)
- Acceleration (time complexity)
- Deep learning compiler
Pruning
- 가지치기 (중요하지 않은 노드, 파라미터 제거)
- Iterative prunning
- Remove the least important neuron
- process
- init model
- apply mask to nodes/filters
- train network until early stopping
- apply prunning metric to choose a fraction p of nodes/filters
- Lottery ticket hypothesis (ICLR 2019)
- initialization lottery: their connections have initial weights that make training particularly effective.
- Search for ticket
- Iterative magnitude pruning
- Iterative magnitude pruning with rewinding
