Women's health and mortality are significantly threatened by breast cancer, underscoring the importance of timely detection and treatment. Mammograms are an extremely useful and trustworthy diagnostic tool for early detection and screening of breast cancer. Mammograms based CADe systems have helped doctors in predicting BI-RADS categories and make better decisions and have somewhat reduced diagnostic errors. As deep learning algorithms advance, deep learning-based CADe systems become a practical means of resolving these problems and greatly improving the accuracy. The purpose of this review is to discuss the current techniques that have been developed for BI-RADS category classification in the fields of deep learning and convolutional neural networks. Additionally, the paper demonstrates the progression of models introduced in the past ten years. It also discusses the shortcomings of models proposed in the literature for the prediction of BI-RADS categories from mammography radiology reports and mammography images, in addition to summarizing the current challenges. Lastly, it proposes a novel multi-modal approach to predict the BI-RADS categories from radiology reports and mammography images.