To improve the continuous detection and robustness of facial feature recognition in fatigue driving detection processes and reduce the configuration requirements of in-vehicle terminals, a lightweight face detection and tracking method for driver fatigue detection was designed. First, MobileNetv3-Small was used as the lightweight network model for facial feature extraction, and parameter-free attention module (SimAM) and depthwise over-parameterized convolution (DOConv) were introduced to construct feature mapping and lightweight feature enhancement modules for deep optimization and focusing on facial region information. Next, DeepSort was integrated for continuous classification tracking, optimizing the impact of facial occlusion on detection performance. Then, facial feature detection and keypoints were combined to determine fatigue driving based on percentage of eyelid closure over the pupil over time (PERCLOS) fatigue threshold, continuous eye closure frames, and the total number of yawns. Experimental results showed that the average precision and recall rates reached 98.9% and 97.2%, improving by 1.8% and 6.3%, respectively. Meanwhile, the floating-point computation was only 28.3% of the baseline model, and the model size was only 7MB. The final fatigue recognition rate was 95.6%, verifying that the detection and tracking method can effectively extract facial features in fatigue driving detection, achieving high detection accuracy, robustness, and meeting the lightweight deployment requirements of in-vehicle terminals.