There is growing evidence that the estimations of the beaming Doppler factor in TeV BL Lac object based on the Self Synchrotron Compton (SSC) models are in strong disagreement with those deduced from the unification models between blazars and radio galaxies. When corrected from extragalactic absorption by the diffuse infrared background (DIrB), the SSC one-zone models require very high Lorentz factor (around 50) to avoid strong gamma-gamma absorption. However, the statistics on beamed vs. unbeamed objects, as well as the luminosity contrast, favors much lower Lorentz factor of the order of 3. In this paper, we show that for the special case of Markarian 501, the need for very high Lorentz factor is unavoidable for all one-zone models where all photons are assumed to be produced at the same location at the same time. Models assuming a double structure with two different beaming patterns can partially solve the problem of luminosity contrast, but we point out that they are inconsistent with the statistics on the number of detected TeV sources. The only way to solve the issue is to consider inhomogeneous models, where low energy and high energy photons are not produced at the same place, allowing for much smaller Lorentz factors. It implies that the jet is stratified, but also that the particle energy distribution is close to a monoenergetic one, and that pair production is likely to be significant. The implications on relativistic jet physics and particle acceleration mechanism are discussed.