This research focuses on the development of a web-based Computer Based Test (CBT) Application for Madrasah Tsanawiyah (MTs) Faqih Hasyim in Sidoarjo, East Java. This application aims to overcome conventional evaluation problems with paper and pen, such as cheating, scoring errors, and data management difficulties. Using web technologies such as HTML, CSS, PHP, and JavaScript, the application integrates the Fisher-Yates Shuffle algorithm to randomize the order of questions, reducing the possibility of cheating during the exam. Equipped with session tokens, exam timing, and result reports, the app improves the overall efficiency of the evaluation process. This study refers to previous research and highlights the importance of adopting technology to simplify assessment and facilitate online data management. The Fisher-Yates Shuffle algorithm, used to generate a unique sequence of questions for each student during the exam, minimizes the risk of cheating. The app implements a staged exam, ensuring students who meet the minimum criteria proceed to the next stage. In addition, the app achieves a degree of randomization of the exam content, eliminating patterns that may affect the exam results. The feasibility testing results show that the app is highly feasible, with a feasibility percentage of 92.4%. This conclusion signifies the app's success in achieving its development goals and making a positive contribution in easing the exam grading process. It is recommended to simplify the interface for senior users, improve the responsiveness of the website display for mobile access, and add a time limitation method in future research. Keywords: Fisher-Yates Shuffle Algorithm, Computer Base Test, Question Randomization.