Weakly supervised Referring Expression Comprehension (WREC) aims to localize referred objects using only image-text pairs without box-level annotations. Existing one-stage methods predominantly rely on anchor-level alignment, which suffers from two fundamental limitations: (1) anchors represent local visual patches rather than holistic objects, and (2) they lack the capability to model inter-object relations. To address these issues, we propose OCAAD, an Object-Centric Alignment and Anchor Distillation framework. Our key insight is that different self-attention heads in DINOv2 naturally attend to distinct semantic regions, effectively capturing object-level information. Building on this, OCAAD introduces two complementary modules: (1) Anchor-Object Distillation Module (AODM), which bridges the semantic gap between anchors and objects by transferring object-level knowledge from attention heads to anchors via overlap-aware contrastive learning; and (2) Intra-Modal Relation Consistency (IRC), which explicitly models inter-object relations by enforcing the relational structure among linguistic entities to match that of their visual counterparts. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate that OCAAD achieves new state-of-the-art performance, validating the effectiveness of objectcentric alignment for WREC. Code is available at https://github.com/VILAN-Lab/OCAAD.