Interaction Effects in Hybrid Compression of Small Language Models
Interaction Effects in Hybrid Compression of Small Language Models
Iheb Bouriel, Qassim Nasir, Manar Abu Talib
Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence
Main Track. Pages 5666-5674.
https://doi.org/10.24963/ijcai.2026/631
We investigate the interaction between quantization and pruning in the compression of open-weight small language models (SLMs), as these techniques are frequently combined in practice without a clear understanding of whether their effects are antagonistic, synergistic or additive. Prior work has reported ordering effects and studied pruning or quantization in isolation, but rarely quantifies their interaction across models and evaluation metrics. We introduce an interaction coefficient that isolates non-additive effects and apply it to Falcon3-1B-Base and LLaMA-3.2-1B, with a limited study on Qwen2.5-1.5B. Our experiments cover four hybrid application orders, multiple pruning sparsities and 4/8-bit quantization. We find non-monotonic interactions: memory savings are dominated by 4-bit quantization, while pruning alone provides limited benefits. Hybrids exhibit the greatest effects at global unstructured pruning sparsity s ∈ {0.2, 0.3} while getting close to additivity at s=0.5. At low sparsity, truthfulness improves, while reasoning losses diminish at higher sparsity. Future work will extend interaction analysis to hardware-aware sparse kernels and larger models.
Keywords:
Natural Language Processing: Language models
Machine Learning: Optimization
Machine Learning: Benchmarks
Natural Language Processing: Interpretability and analysis of models for NLP
