Driven by a large number of emerging diversified services, in the 5G and beyond era, concurrent direct inter-ONU and upstream communications inside a PON-based mobile access network are highly desirable to provide dynamic, ultra-dense, and fast ONU-to-ONU (without involving an OLT) and ONU-to-OLT connections. To cost-effectively deliver highly dynamic and low latency direct inter-ONU communications, this paper proposes and experimentally demonstrates novel concurrent direct inter-ONU and upstream communications in an upstream 27 km, >62.47 Gbit/s IMDD PON. For supporting inter-ONU communications between a large number of ONUs, an advanced passive remote node is also proposed. Based on different passive optical components, this remote node can be implemented using two approaches, which can, respectively, reduce the inter-ONU signal power losses by >12.2 dB and >16.6 dB (for 128 ONUs) in comparison with existing inter-ONU communication techniques’ remote nodes. In each ONU and OLT, a single pair of cascaded IFFT/FFT-based point-to-multipoint (P2MP) flexible optical transceivers are employed to simultaneously and dynamically establish multiple ONU-to-ONU and ONU-to-OLT communications according to actual users’ requirements. Experimental results show that the proposed network has excellent robustness against various transmission system impairments, including chromatic dispersion, the Rayleigh and Brillouin backscattering effects, and the channel interference effects. For each ONU, dynamic channel allocation can be made without compromising its overall performance.