The Metaverse is envisioned to support immersive, large-scale social interactions via virtual reality (VR) displays. However, scalability remains a major bottleneck: as the number of concurrent users grows, tracking and updating display content incurs quadratic overhead, often limiting a shared virtual space to only a few dozen participants. Our key observation is that user attention in social VR is highly selective, with users primarily focusing on socially relevant peers rather than all visible users. Motivated by this observation, we propose SAGE, a social-aware graph-based VR display framework that enables personalized displays based on inferred social relevance. SAGE introduces a dual-graph learning architecture to jointly model long-term social structures and short-term spatiotemporal co-presence patterns, generating complementary interest scores for display prioritization. Based on these scores, we formulate scalable VR display support as a multi-dimensional resource allocation problem and design a lightweight coordination mechanism with provable guarantees, including incentive compatibility and individual rationality. Experiments on Metaverse datasets show that SAGE improves interaction-relevance prediction by 11.64% and increases social welfare by up to 2.4× compared to state-of-the-art schemes. It scales to support up to 1,000 concurrent users and remains robust against strategic manipulation.