The IEC 61850 protocol uses GOOSE messages for fast, reliable power system communication, but is vulnerable to DoS attacks that can disrupt protection and control communications. To address this vulnerability, an LSTM model can learn sequence-based communication patterns, but its performance depends on hyperparameter configuration. Therefore, an approach is needed to optimize the LSTM configuration. The present study proposes an optimized LSTM system for detecting DoS attacks in GOOSE communications. The proposed approach integrates GOOSE-specific feature extraction, feature selection, class imbalance handling, and hyperparameter optimization to enhance the model's detection capabilities. The novelty of this research lies in the use of GOOSE communication attribute characteristics, such as stNum, sqNum, and timeAllowedToLive, within the Optimized LSTM approach to improve the reliability of DoS attack detection in GOOSE. The evaluation results demonstrate that the Optimized LSTM model exhibits a high degree of accuracy, with a mean accuracy of 99.76%, a precision of 99.84%, a recall of 99.69%, and an F1-score of 99.76%. These findings are corroborated by statistical analysis, which reveals significant disparities between models in terms of accuracy, F1-score, and ROC-AUC (p-value < 0.0001). Moreover, the Optimized LSTM model has been demonstrated to surpass competing models, including the Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), and Transformer models (adjusted p-value 0.0059–0.0176). These results support the conclusion that optimizing the LSTM model's hyperparameters can enhance DoS detection performance and advance deep learning-based detection in GOOSE communications.