A novel digital signal processing (DSP)-based scheme for physical layer security in coherent optical communication systems is proposed and numerically investigated. The optical layer signal encryption is accomplished by two dispersive elements and one phase modulator (PM) driven by a DSP-generated encryption key, whilst signal decryption uses similar components but with inverted dispersion values and security keys. A critical aspect of the DSP-based physical layer security is that the security keys, driving the PMs to hide/recover the data signals, must be highly unpredictable and noise-like, thus orthogonal frequency division multiplexing (OFDM) signals are employed as they possess these characteristics, they can also be easily generated and cover a suitably wide range of unique keys. Numerical simulations are conducted to determine optimum system parameters for achieving a high level of security, the key parameters requiring optimization are the dispersion of the dispersive elements and the bandwidth of the security keys. Using these determined optimum parameters, in-depth investigations are undertaken of encryption/decryption induced transmission performance penalties, sensitives to various parameter offsets and operation over various transmission distances. To observe any data signal dependencies, various performance metrics are investigated for different combinations of modulation formats (DQPSK and 16QAM) and baud rates (40Gbaud and 100Gbaud) for the transmitted data signals. The proposed DSP-based physical layer security scheme is shown to have the potential to achieve, in a low-cost and highly effective manner, a high level of physical layer security with acceptable performance penalties for existing coherent optical communication systems.