The masses of the low-lying strange and charm baryons are evaluated using two degenerate flavors of twisted mass sea quarks for pion masses in the range of about 260 MeV to 450 MeV. The strange and charm valence quark masses are tuned to reproduce the mass of the kaon and D-meson at the physical point. The tree-level Symanzik improved gauge action is employed. We use three values of the lattice spacing, corresponding to $\beta=3.9$, $\beta=4.05$ and $\beta=4.2$ with $r_0/a=5.22(2)$, $r_0/a=6.61(3)$ and $r_0/a=8.31(5)$ respectively. %spacings $a=0.0855(5)$ and $a=0.0667(3)$ determined from the pion decay constant. We examine the dependence of the strange and charm baryons on the lattice spacing and strange and charm quark masses. The pion mass dependence is studied and physical results are obtained using heavy baryon chiral perturbation theory to extrapolate to the physical point.