Abstract: The precise identification of the number of spiral arms in spiral galaxies using deep learning methods plays an important role in tracing the evolution trajectory of galaxies and understanding the evolution laws of spiral structures in different cosmic periods. However, the features of spiral galaxy images, particularly the distribution characteristics of spiral center and arm brightness, have not been properly taken into account by existing deep learning models. Therefore, this article proposes a brightness-based bidirectional fusion Manhattan attention mechanism, and combines this attention mechanism with the EfficentNet model for identifying the number of spiral arms in spiral galaxies. Experimental study was carried out using the spiral galaxy images chosen from the Galaxy Zoo 2 dataset as an example. The results show that the overall accuracy of the EfficientNet model on the test dataset can reach 95.52% by integrating an attention mechanism, especially good for the recognition of 3, 4, and more spiral arms. Research has shown that well-designed attention mechanisms can reveal the brightness distribution characteristics of spiral galaxies and improve the effectiveness of identifying the number of spiral arms.