When weight pruning helps TinyML
How weight pruning shapes TinyML models, and where hardware constraints limit the gains.
Published:
TinyML runs machine learning on devices with limited memory and power [1]. Weight pruning is one way to shrink a model, alongside quantization and choosing a smaller architecture [2]. Whether it helps depends on the model and the hardware that will run it.
Weight pruning and winning tickets
Magnitude pruning removes connections with small weights, usually followed by retraining [2]. The Lottery Ticket Hypothesis offers a related result. In the experiments reported by Frankle and Carbin, sparse subnetworks trained from their original initial weights could match the full network's test accuracy [3]. Initialization matters. The result does not mean that large weights always contain the model's "true signal."
Deploying on constrained hardware
On microcontrollers, model storage, working memory, latency, and energy use are separate constraints. A smaller weight file does not guarantee lower peak SRAM use or faster inference. Deployment work such as DTMM [4] and pruning for charging-infrastructure anomaly detection [5] considers pruning in the context of a particular device and task.
Measure the deployed model on the target hardware, including preprocessing and any sparse-weight decoding. Compare accuracy, flash storage, peak working memory, latency, and energy with the original model.
Where pruning reaches its limits
Pruning can reduce accuracy, especially when a model already has little spare capacity. Unstructured sparsity also needs a storage format and inference implementation that can use it. Otherwise, a dense kernel may still process the zero weights. Structured pruning removes whole filters or channels and can make smaller dense operations possible. Compression and hardware support need to be evaluated together [6].
References
[1] P. Warden and D. Situnayake, TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers. O'Reilly Media, 2019.
[2] S. Han, H. Mao, and W. J. Dally, Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, 2016.
[3] J. Frankle and M. Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, 2019.
[4] DTMM: Deploying TinyML Models on Extremely Weak IoT Devices with Pruning, IEEE INFOCOM 2024.
[6] M. Deutel, P. Woller, C. Mutschler, and J. Teich, Energy-efficient Deployment of Deep Learning Applications on Cortex-M based Microcontrollers using Deep Compression, 2023.