This paper proposes the first attempt to utilize multi-modal learning method for the representation learning of the solar radio spectrums. The solar radio signals sensed from different frequency channels, which present different characteristics, are regarded as different modalities. We employ a multimodal neural network to learn the representations of the solar radio spectrum, which can distinguish the differences and learn the interactions between different modalities. The original solar radio spectrums are firstly pre-processed, including normalization, denoising, channel competition and etc., before being fed into the multimodal learning network. Experimental results have demonstrated that the proposed multimodal learning network can learn the representation of the solar radio spectrum more effectively, and improve the classification accuracy.