Optimization of XOR Cryptographic Keys using a Hybrid Genetic Algorithm and Simulated Annealing - Naufal Muzakki; Nur Rochmah Dyah Puji Astuti (SISTEMASI, 2026)
The security level of basic encryption algorithms, such as XOR, is highly dependent on the randomness and bit distribution pattern of the applied key. The use of stochastic optimization approaches, such as Genetic Algorithm (GA), in key generation often faces challenges due to premature convergence, a condition in which the search halts at a local optimum before achieving maximal entropy. This study proposes a sequential hybrid algorithm strategy that combines GA with Simulated Annealing (SA) to address fitness stagnation in PDF document encryption. The strategy is implemented through a two-phase mechanism: GA performs global exploration to identify potential solution regions, followed by SA performing local exploitation with a perturbation mechanism guided by the Metropolis probability. The algorithm’s performance is evaluated through a comparative study between conventional GA and the hybrid GA-SA. Experimental results indicate that the hybrid strategy successfully increases the average fitness value by 4.86%, achieves a Shannon entropy of 7.8952, and attains an NIST test P-value of 0.5299. These improvements demonstrate that the integration of SA effectively enhances the final solution quality of GA, producing cryptographic keys with more uniform bit distribution, passing statistical randomness tests, and exhibiting robustness against pattern analysis.