Using the harmonic superspace approach, we construct the three-dimensional N=4 supersymmetric quantum mechanics of the supermultiplet (3,4,1) coupled to an external SU(2) gauge field. The off-shell N=4 supersymmetry requires the gauge field to be a static form of the 't Hooft ansatz for the 4D self-dual SU(2) gauge fields, that is a particular solution of Bogomolny equations for BPS monopoles. We present the explicit form of the corresponding superfield and component actions, as well as of the quantum Hamiltonian and N=4 supercharges. The latter can be used to describe a more general N=4 mechanics system, with an arbitrary BPS monopole background and on-shell N=4 supersymmetry. The essential feature of our construction is the use of semi-dynamical spin (4,4,0) multiplet with the Wess-Zumino type action.