Predicting responses to genetic perturbation is pivotal for elucidating gene regulatory machinery. However, existing methods often rely on statistical perspectives to model differential expression, overlooking the constraints of the underlying molecular interactome, which renders predictions susceptible to spurious correlations. Based on the fact that a genetic perturbation is a molecular stimulus acting through functional connectivity to reconfigure cellular expression profiles, we propose PertDCR, a framework for context-aware Perturbation response prediction via Distance-Constrained Refinement. Specifically, PertDCR addresses the heterogeneity of perturbation effects by contextualizing the perturbation source with relevant gene programs inferred from the protein-protein interaction network as well as cell representation. To activate biologically valid responses, PertDCR predicts the distance between the perturbation source and highly responsive genes within the PPI network, and dynamically modulates the response intensity based on distance, thereby facilitating accurate perturbation prediction. Extensive experiments demonstrate that PertDCR achieves state-of-the-art performance across diverse unseen genetic perturbations, and the predictions are consistent with underlying biological mechanisms.