Article Per Year (5 Year)
p-Index From 2021 - 2026
0.444
P-Index
This Author published in this journals
All Journal Journal of Technology Informatics and Engineering
Hengning Rao
Electrical and Computer Engineering, UIUC, IL, USA
Computer Science & IT

Published : 2 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 2 Documents
Search

 1 
Trajectory Reliability Prediction for Generalist AI Agents: Tool-Use Failure Analysis and Success Forecasting on ZClawBench Ziliang Samuel Zhong; Chenyu Li; Hengning Rao
Journal of Technology Informatics and Engineering Vol. 5 No. 1 (2026): APRIL | JTIE : Journal of Technology Informatics and Engineering
Publisher : University of Science and Computer Technology

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.51903/jtie.v5i1.539

Abstract

Generalist AI agents increasingly perform complex tasks through planning, tool execution, action revision, and artifact generation rather than isolated response generation. This study empirically investigates trajectory reliability prediction on ZClawBench, a public OpenClaw-style agent benchmark containing 696 model-task trajectories across 116 tasks, six model families, and six scenario categories. The study evaluates whether operational trajectory signals—including tool-call volume, replanning behavior, tool errors, invalid-action ratio, recovery patterns, looping indicators, trajectory length, and task category—can predict task success before manual evaluation. Since the dataset provides trajectories and identifiers but lacks explicit per-instance success labels, the binary target is reconstructed from official model-by-category score distributions and treated as a modeling assumption. Five-fold cross-validation with task-level splitting was applied to prevent task leakage. Logistic Regression, Random Forest, XGBoost, sequence TF-IDF classification, and a rubric-based trajectory judge were compared. Logistic Regression achieved the strongest calibrated performance, obtaining ROC-AUC of 0.970, F1-score of 0.913, Brier score of 0.066, and expected calibration error of 0.018. Feature analysis indicated that task difficulty, no-progress behavior, response size, tool errors, and invalid actions contributed most to reliability prediction. The findings suggest that lightweight trajectory diagnostics can support agent monitoring, failure triage, and routing decisions, while further validation with direct case-level evaluation labels is required for deployment claims.
Few-Shot Cold-Start Workload Forecasting for New AI Inference Tenants with Time-Series Foundation Models Shilu He; Chengliang Li; Hengning Rao
Journal of Technology Informatics and Engineering Vol. 4 No. 1 (2025): APRIL | JTIE : Journal of Technology Informatics and Engineering
Publisher : University of Science and Computer Technology

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.51903/jtie.v4i1.546

Abstract

This paper presents a reproducible empirical study of few-shot cold-start workload forecasting for new AI inference tenants using the Alibaba GPU-disaggregated DLRM serving trace. Instance lifecycles are transformed into hourly active-demand series, and resource reservations are normalized into capacity units to evaluate 24-hour forecasting under zero-shot, 5-shot, 10-shot, and full-history settings. Seven forecasting methods are compared: archetype mean prior, persistence, moving average, linear trend, seasonal naive, global residual ridge, and CT-TSFM, a compact cross-tenant time-series foundation model. The cold-start evaluation uses 46 held-out tenants, with 110 source tenants for pretraining and calibration. Results show that hourly demand is strongly persistence-dominated. Zero-shot forecasting yields a mean absolute error (MAE) of 326.26 normalized capacity units, whereas only five observations reduce MAE to 4.00 for persistence, global residual ridge, and CT-TSFM. Validation consistently selects a residual gate of 0.0 for CT-TSFM, indicating that retaining the persistence prior and rejecting cross-tenant residual transfer is the most reliable strategy. Calibration intervals achieve approximately 85–87% coverage against a 90% target. The findings demonstrate that a few recent observations substantially improve cold-start forecasting, while source-tenant metadata alone provides limited zero-shot planning capability.
Co-Authors Chengliang Li Chenyu Li Shilu He Ziliang Samuel Zhong