To address the problem of multi-object tracking under the unknown motion of the intelligent vehicle, a visual multi-object tracking method is proposed, which is based on the coherent point drift. First, the unknown motion model of the intelligent vehicle is formulated by the coherent point drift algorithm. The local object state transformation relationship is achieved. Secondly, an adaptive feature fusion function is constructed, which is based on appearance similarity and motion consistency. Therefore, the Hungarian algorithm is utilized to solve the correspondence between the track and the detection. Finally, the robust data association for the intelligent vehicle is realized. Compared with the current five mainstream multi-object tracking methods, results show that the proposed algorithm has better results in multiple indicators. Compared with the SCEA algorithm, the multi-object tracking accuracy of the proposed method is increased by 6. 3% in the large motion scene of the KITTI dataset. Under the real-shot experimental data, the multi-object tracking accuracy of the proposed method is increased by 7. 3% , which can effectively perform multi-object tracking under the unknown motion of the intelligent vehicle.