When a user holds a rigid tool to slide on the material surface, the texture features of the material surface through vibration of the tool is felt. These vibration acceleration data contain rich texture category information, which provides a basis for texture classification. Texture classification based on tactile sense is of great significance for applications such as haptic human-computer interaction and fine manipulation of robots. At present, the methods of manually designing features related to texture and simple feature extraction using convolutional neural network have been applied to tactile texture classification. However, these methods fail to pay attention to the selection of time scale and the time dependence between tactile serial data, and there are still problems such as insufficient feature extraction of tactile data and poor classification accuracy. To solve the above problems, this article proposes a fusion model which combines multi-scale convolutional network and bidirectional long short memory network to capture multi-scale geometric local spatial features and time dependent features of tactile signals at the same time. The proposed model learns the tactile features of material surface texture from an open tactile data set, and trains them on the open texture vibration acceleration database. The experimental results show that the proposed model achieves the highest texture classification accuracy of 92. 1% robustly and efficiently.