<![CDATA[Generalist AI agents increasingly perform complex tasks through planning, tool execution, action revision, and artifact generation rather than isolated response generation. This study empirically investigates trajectory reliability prediction on ZClawBench, a public OpenClaw-style agent benchmark containing 696 model-task trajectories across 116 tasks, six model families, and six scenario categories. The study evaluates whether operational trajectory signals—including tool-call volume, replanning behavior, tool errors, invalid-action ratio, recovery patterns, looping indicators, trajectory length, and task category—can predict task success before manual evaluation. Since the dataset provides trajectories and identifiers but lacks explicit per-instance success labels, the binary target is reconstructed from official model-by-category score distributions and treated as a modeling assumption. Five-fold cross-validation with task-level splitting was applied to prevent task leakage. Logistic Regression, Random Forest, XGBoost, sequence TF-IDF classification, and a rubric-based trajectory judge were compared. Logistic Regression achieved the strongest calibrated performance, obtaining ROC-AUC of 0.970, F1-score of 0.913, Brier score of 0.066, and expected calibration error of 0.018. Feature analysis indicated that task difficulty, no-progress behavior, response size, tool errors, and invalid actions contributed most to reliability prediction. The findings suggest that lightweight trajectory diagnostics can support agent monitoring, failure triage, and routing decisions, while further validation with direct case-level evaluation labels is required for deployment claims.]]>
Copyrights © 2026
