Microorganisms are the primary agents responsible for the modification, degradation, and/or detoxification of pollutants, and thus, they play a major role in their natural attenuation; yet, little is known about the structure and diversity of the subsurface community and relationships between microbial community and groundwater hydrochemistry. In this study, denaturing gradient gel electrophoresis (DGGE) and terminal restriction fragment length polymorphism (T-RFLP) allowed a comparative microbial community analysis of sulfate-contaminated groundwater samples from nine different wells in the region of Baogang rare earth tailings. Using real-time PCR, the abundance of total bacteria and the sulfate-reducing genes of aprA and dsrB were quantified. Statistical analyses showed a clear distinction of the microbial community diversity between the contaminated and uncontaminated samples, with Proteobacteria being the most dominant members of the microbial community. SO42− concentrations exerted a significant effect on the variation of the bacterial community (P < 0.05), with higher concentrations of sulfate reducing the microbial diversity (H′ index), indicating that human activity (e.g., mining industries) was a possible factor disturbing the structure of the bacterial community. Quantitative analysis of the functional genes showed that the proportions of dsrB to total bacteria were 0.002–2.85 %, and the sulfate-reducing bacteria (SRB) were predominant within the prokaryotic community in the groundwater. The uncontaminated groundwater with low sulfate concentration harbored higher abundance of SRB than that in the polluted samples, while no significant correlation was observed between sulfate concentrations and SRB abundances in this study, suggesting other environmental factors possibly contributed to different distributions and abundances of SRB in the different sites. The results should facilitate expanded studies to identify robust microbe-environment interactions and provide a strong foundation for qualitative exploration of the bacterial diversity in rare earth tailings groundwater that might ultimately be incorporated into the remediation of environmental contamination.