In an agent-driven application, suppose a utility generates structured interim data, such as JSON outputs from an entity extraction process, that needs to be accessible across various phases of a single, prolonged operation. This data's existence is tied strictly to the duration of that specific operation, lasting up to 30 minutes, and it must endure across multiple tool retries; however, persistence beyond the current session is not required. Which storage approach offers the most effective solution for this situation?
Answer & explanation
Correct answer: D
WHY D is correct: A Delta table keyed by task_id is the appropriate Databricks-native solution for within-task intermediate state that must persist across tool retries (retry = same task_id, re-read the last committed state). Delta's ACID properties ensure consistency during retries, and the data can be cleaned up after the task completes. This matches the 'configure a persistent datastore for intermediate structured information' objective.
WHY NOT B: A module-level dictionary lives in the memory of one serving process. Keying it by task_id separates the tasks, but the state is lost if the container restarts or scales, and a retry that lands on another replica cannot see it, so it does not reliably survive tool retries. Entries are also never cleaned up unless the code does it.
WHY NOT C: Passing all intermediate JSON artifacts as part of the conversation messages inflates the context window with potentially large structured objects, consuming tokens unnecessarily. Long context windows increase latency and cost and may exceed the model's context limit for lengthy tasks.
WHY NOT A: The MLflow artifact store is designed for logging training/evaluation artifacts (model files, eval datasets), not for real-time read/write of within-task intermediate state during agent execution. It has high overhead per write and is not optimized for rapid sequential reads within a running inference task.