We present results on the mass of the nucleon and the $\Delta$ using two dynamical degenerate twisted mass quarks. The evaluation is performed at four quark masses corresponding to a pion mass in the range of 690-300 MeV on lattices of size 2.1 fm and 2.7 fm. We check for cutoff effects by evaluating these baryon masses on lattices of spatial size 2.1 fm with lattice spacings $\alpha(\beta$ = 3.9) = 0.0855(6) fm and $\alpha(\beta$ = 4.05) = 0.0666(6) fm, determined from the pion sector and find them to be within our statistical errors. Lattice results are extrapolated to the physical limit using continuum chiral perturbation theory. The nucleon mass at the physical point provides a determination of the lattice spacing. Using heavy baryon chiral perturbation theory at O(p^3) we find $\alpha(\beta$ =3.9) =0.0879(12) fm, with a systematic error due to the chiral extrapolation estimated to be about the same as the statistical error. This value of the lattice spacing is in good agreement with the value determined from the pion sector. We check for isospin breaking in the $\Delta$-system. We find that $\Delta^{++,−}$ and $\Delta^{+,0}$ are almost degenerate pointing to small flavor violating effects.