We propose and demonstrate a novel physical, secure high-speed optical communication scheme based on synchronous chaotic spectral phase encryption (CSPE) and decryption (CSPD). The CSPE is performed by a module composed of two dispersion components and one phase modulator (PM) between them, and the CSPD is carried out by a twin module with reverse dispersions and inverse PM driving signals. The PM driving signals of the CSPE and CSPD modules are privately synchronized chaotic signals that are independently generated by local external-cavity semiconductor lasers subject to common injection. The numerical results indicate that with the CSPE, the original message can be encrypted as a noise-like signal, and the timing clock of the original message is efficiently hidden in the encrypted signal. Based on the private synchronization of the chaotic PM driving signals, only the legal receiver can decrypt the message correctly, while the eavesdropper is not able to intercept a useful message. Moreover, the proposed scheme can also support secure symmetric bidirectional high-speed WDM transmissions. This work shows a prospective way to implement high-speed secure optical communications at the physical layer.