Questions on dataset contents and agent interaction trajectories

[zchuz]

Title: Questions on Dataset Structure, Trace Discrepancy, and Request for Agent Interaction Traces

Hello, thank you for creating and sharing the TaskCraft dataset. It's a fantastic resource for the community. I have studied it carefully and have a few questions about this datasets.

Question1:

Relationship between the three core files: Could you please clarify the intended relationship between pure_qa.json, multihop_subtask_trace.json, and atomic_trace.json?
I noticed that the number of multi-hop questions with decomposition in multihop_subtask_trace.json is less than the number of multi-hop questions traces in pure_qa.json. Could you explain the reason behind this? Why are decomposition traces not provided for some questions that are present in the trace file?

Question2:

Request for Agent-Environment Interaction Traces
Is it possible to provide the trajectories of the Agent's real interactions with the environment on the TaskCraft dataset? Similar to the T-A-O (Thought-Action-Observation) format of interaction trajectories used in ReACT. (Mentioned in Section 4.4)

Thank you for your time and consideration.

[CC1287h]

Hello, thank you for your attention and questions!

Question 1:

(1) Relationship between the three core files: Could you please clarify the intended relationship between pure_qa.json, multihop_subtask_trace.json, and atomic_trace.json?

The tasks contained in each file are explained as follows:

TaskCraft_Dataset/

├── pure_qa.jsonl # All tasks with pure QA pairs

├── atomic_trace.jsonl # Atomic tasks with tool calling traces

└── multihop_subtask_trace.jsonl # Multi-step decomposition for the multihop task.

(2)I noticed that the number of multi-hop questions with decomposition in multihop_subtask_trace.json is less than the number of multi-hop questions traces in pure_qa.json. Could you explain the reason behind this?

The multihop_subtask_trace.json file contains a list. The multiple elements in the list are the multiple decomposed subtasks of the multi-hop task. After taking out all the tasks (each hop number), the complete task set is formed: pure_qa.json.

(3) Why are decomposition traces not provided for some questions that are present in the trace file?

Because the trajectory file contains atomic tasks, they do not need to be decomposed into multiple subtasks. Multi-hop tasks are extended from atomic tasks.

For detailed extension steps, please refer to Figure 3 and the appendix of the paper https://arxiv.org/abs/2506.10055.
For a more detailed introduction to the dataset, please refer to https://huggingface.co/datasets/PersonalAILab/TaskCraft

Question 2:
Request for Agent-Environment Interaction Traces
Is it possible to provide the trajectories of the Agent's real interactions with the environment on the TaskCraft dataset? Similar to the T-A-O (Thought-Action-Observation) format of interaction trajectories used in ReACT. (Mentioned in Section 4.4)

atomic_trace.json contains the agent's execution trace for solving atomic tasks. If you need pre-processed trace data for SFT or RL, please refer to https://huggingface.co/datasets/PersonalAILab/AFM-WebAgent-SFT-Dataset.
This includes traces generated from Taskcraft tasks.