Use of cover crops in an integrated agricultural system can reduce demand of inorganic phosphorus (P) fertilizers, where the subsequent crops can take up P accumulated in cover crops biomass after the decomposition. In this research we hypothesized that some cover crops can take up P from less labile fractions and recycle it back to the soil through plant residues resulting in better P use efficiency of the system; cover crops are capable of P uptake from subsurface layers which leads to the accumulation of this P on the surface after the decomposition of their residues; and cover crop species respond differently to distinct inorganic P sources. To examine these hypotheses, a field experiment was conducted over nine successive years in South Brazil. The experimental treatments were established as a split-plot randomized block design, in a 3 × 6 factorial scheme, considering three P treatments [no-P, single superphosphate (SSP), and rock phosphate (RP)] as main plot and six cover crop treatments (common vetch, white lupin, fodder radish, ryegrass, black oat, and a fallow) as subplots. Soil samples were collected from the depths of 0–5, 5–10, and 10–15 cm and analyzed by Hedley P fractionation. In P-unfertilized cropping system, the amounts of labile and mod-labile P fractions were not modified by cover crops related to fallow. When inorganic P fertilizers were applied, the amount of labile and mod-labile P pools under fallow were higher than under cover crops in 5–10 cm depth. Black oat and common vetch cycled more labile P under SSP and RP, respectively. Common vetch and ryegrass resulted in the highest accumulation of organic P on the surface under SSP and RP, respectively. Black oat was capable to change P extracted by 1.0 M HCl to more labile forms. Fodder radish showed the highest P uptake in comparison with other cover crops. The higher P balance efficiency of the system was achieved under SSP in comparison with RP application but it seems that cover crops are more effective at improving the efficiency under RP compared to SSP.