To fully answer your question, it would be helpful to know if they are almost all off by some constant factor, or if they are scattered, or something else.
A known issue that should only affect a small percentage of the data is that the LSS catalogs use the tile-based redshifts and fastspecfit uses the healpix-based ones. But, that would result in exact agreement for almost all of the data (because most are single tile spectra), some small disagreement for a small fraction (because we don't expect BGS redshifts to change much from increased S/N), and large disagreement for a very small fraction (~< 1% of the multi-tile spectra).
(From your description, I doubt that is your issue, but might face it at some point.)