It is well-known that stars with giant planets are on average more metal-rich than stars without giant planets, whereas stars with detected low-mass planets do not need to be metal-rich. With the aim of studying the weak boundary that separates giant planets and brown dwarfs (BDs) and their formation mechanism, we analyze the spectra of a sample of stars with already confirmed BD companions both by radial velocity and astrometry. We employ standard and automatic tools to perform an EW-based analysis and to derive chemical abundances from CORALIE spectra of stars with BD companions. We compare these abundances with those of stars without detected planets and with low-mass and giant-mass planets. We find that stars with BDs do not have metallicities and chemical abundances similar to those of giant-planet hosts but they resemble the composition of stars with low-mass planets. The distribution of mean abundances of $alpha$-elements and iron peak elements of stars with BDs exhibit a peak at about solar abundance whereas for stars with low-mass and high-mass planets the [X$_alpha$/H] and [X$_{rm Fe}$/H] peak abundances remain at $sim -0.1$~dex and $sim +0.15$~dex, respectively. We display these element abundances for stars with low-mass and high-mass planets, and BDs versus the minimum mass, $m_C sin i$, of the most-massive substellar companion in each system, and we find a maximum in $alpha$-element as well as Fe-peak abundances at $m_C sin i sim 1.35pm 0.20$ jupiter masses. We discuss the implication of these results in the context of the formation scenario of BDs in comparison with that of giant planets.