The hole-state random phase approximation (hRPA) and the particle-state random phase approximation (pRPA) for systems like odd $A$ nuclei are discussed. These hRPA and pRPA are formulated based on the Hartree-Fock ground state. An extension of hRPA and pRPA based on a correlated ground state is given using time-dependent density-matrix theory. Applications to the single-particle states around $^{16}$O are presented. It is shown that inclusion of ground-state correlation affects appreciably the results of hRPA and pRPA. The question of the coupling of the center of mass motion of the core to the particle (hole) is also discussed.