Despite being a cost-effective modality for volumetric imaging, freehand three-dimensional (3D) ultrasound produces inherently sparse data due to the significant elevational gaps left by tracked 2D sweeps. This sparsity poses a unique challenge for Implicit Neural Representations (INRs). While successful in other domains, INRs applied here tend to fail as the learned signed distance fields (SDF) collapse toward zero inside the object, leading to the loss of concavities and the incorrect closure of anatomical gaps. Our analysis identifies the root cause as a statistical bias in gradient-based sampling objectives, showing that symmetric volumetric sampling mathematically drives the expected SDF value to zero. To rectify this, we present a geometry-aware framework that explicitly anchors the non-negative half-space. Our method utilizes a boundary-directed exterior sampling strategy to ensure non-negative constraints in empty areas, complemented by an ellipsoid-based adversarial mechanism to regularize the global field distribution. Experiments on multiple anatomical datasets demonstrate that our approach mitigates field collapse and improves geometric fidelity and topological consistency on most metrics. Code is available at https://github.com/jiuanchen/Pulling-Fails-SDF.