To address the continuous growth in high-speed ubiquitous access required by residential users and enterprises, Telecommunication operators must upgrade their networks to higher data rates. For optical fiber access networks that directly connect end users to metro/regional network, capacity upgrade must be done in a cost- and energy-efficient manner. 40 Gb/s is the possible lane rate for the next generation passive optical networks (NG-PONs). Ideally, existing 10 G PON components could be reused to support 40 Gb/s lane-rate NG-PON transceiver, which requires efficient modulation format and digital signal processing (DSP) to alleviate the bandwidth limitation and fiber dispersion. The major contribution of this work is to offer insight performance comparisons of 40 Gb/s lane rate electrical three level Duobinary, optical Duobinary, and four-level pulse amplitude modulation (PAM-4) for incorporating low complex DSPs, including linear and nonlinear Volterra equalization, as well as maximum likelihood sequence estimation. Detailed analysis and comparison of the complexity of various DSP algorithms are performed. Transceiver bandwidth optimization is also undertaken. The results show that the choices of proper modulation format and DSP configuration depend on the transmission distances of interest