DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method

Key points

• Modern neural networks are heavily over-parameterized.1
• This redundancy incurs substantial compute and memory overhead during training and inference.1
• During training, DADP measures connection importance via the accumulated product of pre-synaptic activations and post-synaptic error gradients.1
• Using a single global threshold instead of fixed layer budgets, DADP dynamically allocates sparsity across network depth while naturally inducing neuron- and channel-level pruning.1

Sources (1)

• [1]DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 12:38 PM

  We propose Dynamic Activity-Dependent Pruning (DADP), a biologically inspired structural plasticity mechanism.

  Modern neural networks are heavily over-parameterized.

Extractive summary: sentences quoted from the sources.