Transformasi digital di sektor publik masih menghadapi fenomena black-box, yaitu keputusan administratif berbasis kecerdasan buatan (AI) yang sulit dijelaskan dan diaudit sehingga berpotensi menimbulkan bias serta ketidakadilan, termasuk dalam penyaluran bantuan sosial (DTKS). Penelitian ini bertujuan merekayasa model sistem pendukung keputusan publik menggunakan pendekatan Explainable AI (XAI) melalui metode LIME dan SHAP. Dengan metodologi Design Science Research (DSR), penelitian menghasilkan artefak yang mampu mengubah data teknis menjadi justifikasi yang transparan dan mudah dipahami. Kebaruan penelitian terletak pada integrasi lapisan interpretabilitas berbasis LIME dan SHAP sebagai instrumen meaningful human control untuk mewujudkan akuntabilitas parametrik yang dapat diaudit secara real-time. Hasil penelitian menunjukkan bahwa implementasi XAI mampu meminimalkan ruang gelap diskresi birokrasi, mengurangi potensi manipulasi data, serta memperkuat akuntabilitas hukum. Selain meningkatkan efisiensi birokrasi, model ini mendukung penerapan prinsip Digital Good Governance. Penelitian menyimpulkan bahwa transparansi algoritmik merupakan pilar penting keadilan birokrasi dan merekomendasikan audit algoritma berbasis XAI pada sistem pelayanan publik. Digital transformation in the public sector is often constrained by the "black-box" phenomenon, where artificial intelligence (AI) makes administrative decisions without transparent explanations, creating risks of bias and social inequality, particularly in the distribution of social assistance (DTKS). This study aims to develop a public decision support system using Explainable Artificial Intelligence (XAI) through the Local Interpretable Model-agnostic Explanations (LIME) and Shapley Additive Explanations (SHAP) methods. Employing the Design Science Research (DSR) methodology, the study produces a technological artifact that transforms complex technical data into transparent and understandable justifications. The novelty of this research lies in integrating an explainability layer based on LIME and SHAP as an instrument of meaningful human control to enable real-time, auditable parametric accountability. The findings demonstrate that XAI effectively reduces opaque bureaucratic discretion, minimizes opportunities for data manipulation, and strengthens legal accountability. Beyond improving bureaucratic efficiency, the proposed model supports the implementation of Digital Good Governance. The study concludes that algorithmic transparency is a fundamental pillar of bureaucratic justice and recommends mandatory XAI-based algorithm audits for public service systems.