Solar chromospheric fibrils, as observed in the core of strong chromospheric spectral lines, extend from photospheric field concentrations suggesting that they trace magnetic field lines. These images have been historically used as proxies of magnetic fields for many purposes. We use a Bayesian hierarchical model to analyze several tens of thousands of pixels in spectro-polarimetric chromospheric images of penumbrae and chromospheric fibrils. We compare the alignment between the field azimuth inferred from the linear polarization signals through the transverse Zeeman effect and the direction of the fibrils in the image. We conclude that, in the analyzed fields of view, fibrils are often well aligned with the magnetic field azimuth. Despite this alignment, the analysis also shows that there is a non-negligible dispersion. In penumbral filaments, we find a dispersion with a standard deviation of ~16 degrees, while this dispersion goes up to ~34 degrees in less magnetized regions.