<![CDATA[The increasing amount of online news related to the fertilizer industry requires an automated data processing mechanism to produce structured and reliable information. Manual collection of news data from multiple news portals is inefficient, time-consuming, and prone to errors due to differences in website structures and continuously updated content. This study aims to implement an Extract, Transform, and Load (ETL) process using the Luigi workflow orchestration framework to automate the processing of news data related to fertilizer, subsidized fertilizer, and fertilizer scarcity. The ETL pipeline was developed using Python by collecting news data through web scraping from three national news portals, namely ANTARA, Detik, and Republika. The extracted data were merged, article contents were retrieved, cleaned, normalized, filtered based on publication years 2025–2026, and finally stored in a MySQL database. Luigi was utilized to manage task dependencies and execute the workflow automatically through a structured pipeline. The implementation results show that the developed ETL pipeline successfully automated the entire data processing workflow, improved processing efficiency, reduced manual intervention, and produced structured data ready for further utilization. The application of Luigi also simplified workflow monitoring and maintenance through its scheduler, making the ETL process more reliable and easier to manage for future data engineering developments.]]>
Copyrights © 2026
