To address the challenge of extracting directional information in laser self-mixing interference (SMI) systems under the weak optical feedback regime, this paper proposes a novel method that integrates a random forest (RF) model with time-frequency synergistic processing for direction extraction and displacement reconstruction. First, the SMI signal generated by the Lang-Kobayashi model is denoised using singular value decomposition (SVD). Subsequently, fringe valley detection and identification are performed, and the signal is segmented into individual fringes for resampling. The full set of sampling points of each resampled fringe is constructed as a feature set, and an RF model is trained to learn the mapping between fringe morphology and motion direction, thereby robustly acquiring directional information. The trained RF model achieves classification accuracies of 97.99% and 98.99% on the validation and test sets, respectively, demonstrating strong generalization and robustness. Subsequently, the short-time Fourier transform (STFT) is applied to obtain the time-frequency spectrum of the SMI signal. Combined with the directional information provided by the RF model, a direction-encoded time-frequency ridge is extracted. On this basis, the target displacement is precisely reconstructed via instantaneous velocity integration. Numerical simulations verify the reconstruction performance under noisy conditions, extremely weak optical feedback, and varying optical feedback conditions. Finally, an experimental system is established, and the measurement results show that a displacement reconstruction error of 28.6 nm is achieved for a non-cooperative vibrating target. The proposed method effectively resolves the direction ambiguity challenge under weak feedback that limits traditional algorithms, while also circumventing the need for precise estimation of system parameters. It is particularly suitable for micro-displacement measurement under weak and time-varying optical feedback conditions, providing an effective technical pathway for advancing nanoscale non-contact precision measurement using SMI technology.