As a key transmission component in mechanical systems, gearboxes often work under harsh environmental and heavy-load conditions, making them prone to damage. In recent years, built-in encoder signals have emerged as an effective method for health monitoring of mechanical equipment due to their low cost and ease of acquisition. However, the fault-related features within encoder signals are typically weak and often masked by various interference components, which poses a significant challenge for accurate identification and extraction of fault-induced impulses. To address this issue, this article proposes a fault feature extraction and diagnosis method for encoder signals based on a weighted bi-domain sparse decomposition model. In the proposed model, the morphological differences between fault impulses and interference components are analyzed in both the time and frequency domains. By introducing weighted coefficients, periodic binary vectors, and non-convex penalty functions, the model constructs two dedicated non-convex regularization terms that enforce periodic group sparsity in the time domain for fault impulses and spectral sparsity in the frequency domain for harmonic interferences, thereby achieving accurate separation and representation of target features. Furthermore, an iterative solving algorithm for the weighted bi-domain sparse decomposition model is developed by integrating the alternating direction method of multipliers and the majorization-minimization strategy to enhance computational efficiency and convergence stability. Finally, the effectiveness of the proposed weighted bi-domain sparse decomposition model is validated using both simulated data and experimental data from a planetary gearbox test bench. The results indicate that, compared with two existing methods, namely the tunable Q wavelet transform-based decomposition method and the maximum correlated kurtosis deconvolution method, the proposed method shows a significant advantage in fault impulse feature extraction accuracy, demonstrating strong potential for practical engineering applications.