Vegetation growing beyond safe limits has the potential to pose a threat to safety and the reliability of overhead powerlines, as well as cause financial losses for infrastructure providers. Identifying potential obstructions to overhead powerlines is crucial for addressing these issues. This study proposes the EFF-UNET semantic segmentation technique on the VEPL dataset to identify areas of overlap between vegetation and overhead powerlines by overlaying the two models. Visually, overhead powerlines have a thin pixel structure and are difficult to distinguish from the background or vegetation, whereas the feature extraction process in the U-Net encoder can degrade small objects due to progressive resolution loss. Modifications to the encoder in the baseline U-Net architecture utilize the EfficientNet family by comparing variants B0 through B7 to produce the best model. EfficientNet specifically employs compound scaling to optimize the network’s resolution, depth, and width during feature extraction, thereby preserving information integrity during downsampling. Experimental results demonstrate the superiority of EfficientNetB7 through a measured trade-off compared to other models, where for vegetation segmentation, this model achieves an IoU of 0.9824, Accuracy of 0.9905, Dice of 0.9911, and Loss of 0.0089. Meanwhile, for powerline segmentation, the results show an IoU of 0.9153, Accuracy of 0.9978, Dice of 0.9558, and Loss of 0.0442. Based on these findings, EFF-UNET model successfully addresses the shortcomings of conventional models in preserving feature representation. This model is capable of improving the performance of vegetation and overhead powerlines segmentation to produce precise encroachment areas, thereby enabling accurate on-site infrastructure inspections.