Inferring latent social ties from large-scale spatiotemporal mobility traces is a foundational AI task with broad applicability. Existing hypergraph-based methods often model higher-order relations by treating hyperedges as static snapshots, thus failing to capture the temporal dynamics and co-evolution of user interactions. Meanwhile, many approaches still struggle to distinguish stable social gatherings from transient noisy co-occurrences. To address these challenges, we propose a novel Dual-Channel Hybrid Graph Neural Network (HyGNN) that jointly models temporal dynamics and high-order structural dependencies. The framework consists of two complementary components: (1) a temporal meeting graph channel that decomposes multi-user interactions into ordered snapshots and aligns trajectory evolution through message propagation, and (2) a gathering hypergraph channel that uses a structure-aware encoder with homogeneity-guided aggregation to filter noise in high-order co-occurrences. A mutual-information-aware fusion module integrates both views while preserving their distinct semantics. Extensive experiments across three real-world datasets show that HyGNN consistently outperforms state-of-the-art baselines in PRAUC and ROCAUC metrics, and robustly captures the nonlinear interplay between mobility and sociality. The code is available at https://github.com/KunLiangChen/HyGNN.