In order to detect transformer oil leakage in time and ensure the normal operation of power system, a transformer oil leakage detection method based on improved YOLOv5 is proposed in this paper. By adding 4 times sampling layers to the FPN structure of YOLOv5 model, the feature information is fused across layers to increase the detection accuracy of the model. The Wise-IoU boundary frame loss function with dynamic non-monotone focusing mechanism is introduced to accelerate the training and reasoning of the network, and the overall performance of the model is further improved by balancing the learning of low-quality samples and high-quality samples. Finally, inspired by the Transformer model, the EfficientViT model is used as the backbone network, which significantly reduces the number of parameters in the model, sacrificing a small amount of detection performance, but still maintaining better performance than the original model. Using the self-built outdoor transformer oil leakage data set for training and testing, the results show that compared with the original model, the precision is increased by 8.6%, the recall is increased by 8.5%, the mAP@0.5 is increased by 7.8%, and the number of parameters is decreased by 42.3%, which is beneficial to engineering deployment.