Searches for the permanent electric dipole moments (EDMs) of molecules, atoms, nucleons and nuclei provide powerful probes of CP violation both within and beyond the Standard Model (BSM). The interpretation of experimental EDM limits requires careful delineation of physics at a wide range of distance scales, from the long-range atomic and molecular scales to the short-distance dynamics of physics at or beyond the Fermi scale. In this review, we provide a framework for disentangling contributions from physics at these disparate scales, building out from the set of dimension four and six effective operators that embody CP violation at the Fermi scale. We survey existing computations of hadronic and nuclear matrix elements associated with Fermi-scale CP violation in systems of experimental interest, and quantify the present level of theoretical uncertainty in these calculations. Using representative BSM scenarios of current interest, we illustrate how the interplay of physics at various scales generates EDMs at a potentially observable level.