Due to the effects of coating, material birefringence， and internal stress, polarization aberration is ubiquitous in high numerical aperture (NA) optical systems. In this case, the imaging quality is highly dependent on the polarization of incident light. Conventional methods to characterize the polarization aberration suffer from complex systems and measurement inconvenience. To address this problem, a back-and-forth optical setup is proposed to measure polarization-resolved wavefront and reconstruct the Jones matrix of the objective in this work, which helps reduce the system complexity. Firstly, a numerical model based on the ray tracing method and scalar diffraction theory is established to simulate intensity distributions of focused light fields at different axial positions. Secondly, the problem of wavefront retrieval is transformed into an optimization problem, and an inverse model is formulated for retrieving the Jones matrix utilizing hybrid particle swarm optimization and gradient descent algorithms, where the deviation between modelled and target intensity distributions is minimized by optimizing the coefficients of Zernike polynomials accounting for the polarization aberration. Thirdly, numerical simulation is implemented to examine the model accuracy under predefined polarization aberrations, showing a good agreement between reconstructed and target Jones matrix elements with retrieval errors less than 10-3. Finally, experimental measurements of polarization aberration are performed on a commercial high-NA microscope objective to retrieve the coefficients of Zernike polynomials, where modelled intensity distributions agree well with the target. Theoretical and experimental results demonstrate that the proposed algorithm can effectively retrieve the objective′s polarization aberration with the advantages of a simple configuration and user-friendly operation. It is anticipated that this work provides a novel technical approach for manufacturing and characterizing high-NA optical systems.