<![CDATA[The increasing use of social media as a platform for discussing mental health issues presents significant opportunities for psychological and social media research. However, the lack of automated, fail-resilient, and clean data provisioning remains a major infrastructure challenge. This study developed an automated ETL (Extract, Transform, Load) pipeline using Apache Airflow and Docker Compose to collect and process bilingual social media data from the X platform (Twitter). The system integrates a container-based architecture with persistent volumes to ensure infrastructure resilience and data continuity. A bilingual text preprocessing pipeline was implemented to support automated cleaning and normalization of English and Indonesian social media content. Experimental results show that the system successfully collected 1,768 unique tweets with a preprocessing throughput of 75 tweets per second and achieved a 39% reduction in average text length as part of preprocessing efficiency measurement. These measurements are intended to evaluate ETL preprocessing efficiency and throughput rather than analytical or linguistic quality. Resilience testing confirmed 100% data integrity with zero data loss across multiple failure scenarios, including container restarts and process suspensions. In addition, the implementation of ShortCircuitOperator successfully mitigated API credit exhaustion (HTTP 402) through a graceful degradation mechanism. This study contributes a reproducible and reliable data engineering framework for automated social media data collection and preprocessing, resulting in structured bilingual datasets ready for downstream analytical tasks. However, mental health in this study is positioned as a domain-specific case study, and the resulting dataset is not intended to serve as a clinical diagnostic tool.]]>
Copyrights © 2026
