The persistent attrition of deaf and hard-of-hearing students from Science, Technology, Engineering, and Mathematics (STEM) pathways in Nigerian special education institutions remains a concern that has attracted very little practical intervention. This study examined how an AI-based mathematics tutoring system incorporating Nigerian Sign Language (NSL) influences academic performance and persistence among deaf students at the Federal College of Education (Special), Oyo. A pre-test-post-test quasi-experimental design was adopted, involving 80 purposively selected students (40 experimental, 40 control) drawn from a population of 134 registered deaf students. The AI-NSL tutoring system was developed using adaptive learning algorithms and embedded NSL video models covering core mathematics topics. Data were collected using the Mathematics Achievement Test (MAT) and the STEM Persistence Scale (SPS). Mean, standard deviation, and independent samples t-test were used to analyse the data at a 0.05 level of significance. Findings revealed that students exposed to the AI-NSL tutoring system recorded significantly higher post-test scores (M = 68.42, SD = 9.31) compared to those in the control group (M = 51.17, SD = 11.64), with t(78) = 7.42, p < .001. Similarly, STEM persistence scores were significantly higher in the experimental group. The study concludes that integrating Nigerian Sign Language into AI-driven mathematics instruction holds real promise for reducing STEM dropout among deaf learners. Recommendations include institutional adoption of the AI-NSL system and development of NSL-compliant digital content across other STEM disciplines.