Time series forecasting aims to predict future values using historical observations, which is crucial for many practical applications with complex temporal dynamics. Recent frequency-domain forecasting methods have utilized spectral representations to model periodicity, but they usually rely on an implicit assumption: periodic components and aperiodic residuals are orthogonal in the frequency domain. However, in practice, this assumption often does not hold due to issues such as spectral leakage, superposition of multiple periods, and the gradual periodic changes. This paper re-examines frequency-domain modeling from the perspective of spectral non-identifiability and proposes a Spectral Memory Learner with Dual Retrieval (SMLDR) for time series forecasting. Instead of modeling periodicity in the frequency dimension, SMLDR represents periodic components as reusable spectral prototypes stored in learnable memory. The dual retrieval mechanism extracts historical periodic components and predicts them in the frequency domain, while a lightweight time-domain branch is used to model aperiodic dynamics to predict future residuals.Extensive experiments on multiple time series forecasting benchmarks show that SMLDR consistently achieves excellent performance, and the learnable memory can capture complex periodicity.