Whole-slide images (WSIs) provide gigapixel-scale visual evidence for cancer diagnosis, yet diagnostically relevant regions are typically sparse and embedded within large amounts of weakly informative tissue. Existing multiple instance learning methods often aggregate all patches by using attention mechanisms or sequence modeling, leading to redundant computation and limited adaptability to slides with highly variable sizes and lesion burdens. We propose Proportional Memory-guided Differential-attention Multiple Instance Learning (PMDMIL) network, a scalable MIL framework that follows a “less is more” principle for WSI classification. PMDMIL retains a fixed ratio of patches, pre-filters noise with a class-wise memory bank, and fuses the sparse survivors through differential attention to amplify subtle yet decisive morphological differences. Experiments on five public WSI benchmarks demonstrate that PMDMIL consistently outperforms representative MIL and sequence-based baselines across multiple evaluation metrics. Notably, on the DHMC dataset, our model can achieve competitive performance, while processing only 10% of patches, indicating that effective instance selection is more important than exhaustive aggregation for WSI classification.