Conventional coupled-channels analyses, that take account of only the collective excitations of the colliding nuclei, have failed to reproduce the different behavior of the experimental quasi-elastic barrier distributions for the $^{20}$Ne + $^{90,92}$Zr systems. To clarify the origins of this difference, we investigate the effect of non-collective excitations of the Zr isotopes. Describing these excitations in a random-matrix model, we explicitly take them into account in our coupled-channels calculations. The non-collective excitations are capable of reproducing the observed smearing of the peak structure in the barrier distribution for $^{20}$Ne + $^{92}$Zr, while not significantly altering the structure observed in the $^{20}$Ne + $^{90}$Zr system. The difference is essentially related to the closed neutron shell in $^{90}$Zr.