​

ADR-0003: Refactor Simulation Workflow to Smol Agents

​

Status

​

Date

​

Context

1. Complexity: The current implementation is approximately 1,500 lines in a single file (langgraph_flow.py), making it difficult to maintain and extend.
2. Limited Visibility: Users have limited visibility into the actual testing that occurs in the sandbox environment, making it difficult to understand and trust the simulation results.
3. Debugging Challenges: The graph-based nature of LangGraph makes debugging complex workflows challenging, as state transitions are not always intuitive.
4. Sandboxing Limitations: The current implementation lacks robust sandboxing, which is important for securely executing code during simulations.
5. Explanation Quality: The explanations generated by the current system lack detailed reasoning and transparency about data sources and assumptions.

​

Decision

1. Modularizing the Code: Breaking down the monolithic langgraph_flow.py into smaller, focused modules with clear responsibilities.
2. Implementing Agent-Based Architecture: Creating specialized agents for different aspects of the simulation (diff analysis, sandbox testing, metrics analysis, explanation generation).
3. Enhancing Sandboxing: Leveraging Smol Agents’ built-in sandboxing options (local, E2B, Docker) for more secure code execution.
4. Improving Visibility: Providing users with more visibility into the testing process and results.
5. Enhancing Explanations: Structuring prompts to explicitly request detailed reasoning, data sources, and assumptions.

​

Consequences

​

Positive

1. Improved Maintainability: Smaller, focused modules will be easier to maintain, test, and extend.
2. Better Security: Native integration with E2B and other sandboxing options will improve the security of code execution during simulations.
3. Enhanced User Experience: Users will have more visibility into the simulation process and results, increasing trust and understanding.
4. More Flexible Workflow: The agent-based approach will allow for more flexible and adaptable workflows that can handle a wider range of scenarios.
5. Simplified Debugging: Python-native control flow and error handling will make debugging easier compared to graph-based state transitions.
6. Reduced Code Size: The modular approach is expected to result in less code overall, with better separation of concerns.

​

Negative

1. Migration Effort: Significant effort will be required to refactor the existing code, potentially delaying other feature development.
2. Learning Curve: Team members will need to learn the Smol Agents framework, although it is generally considered simpler than LangGraph.
3. Potential Regressions: The refactoring could introduce regressions or new bugs that will need to be identified and fixed.
4. Documentation Updates: Existing documentation and examples will need to be updated to reflect the new architecture.

​

Alternatives Considered

​

1. Refactor the Existing LangGraph Implementation

• Breaking down the monolithic file into smaller modules
• Improving the graph structure for better debugging
• Enhancing the prompts for better explanations
• Adding custom sandboxing

• Less disruptive change
• Leverages existing knowledge and code

• Would still be constrained by LangGraph’s graph-based architecture
• Custom sandboxing would require significant effort
• Debugging would still be challenging

​

2. Use a Different Agent Framework (e.g., LangChain)

• More mature and widely used
• Extensive documentation and examples
• Rich ecosystem of tools and integrations

• More complex and heavyweight than Smol Agents
• Less focused on code execution
• Would still require significant refactoring effort

​

3. Custom Implementation Without a Framework

• Complete control over the implementation
• No external dependencies
• Could be tailored exactly to our needs

• Would require significantly more development effort
• Would need to reinvent solutions to problems already solved by existing frameworks
• Would require more maintenance over time

​

Implementation Plan

​

Phase 1: Core Module Creation

1. Create the basic module structure
2. Implement utility functions and shared code
3. Extract core functionality from langgraph_flow.py into appropriate modules

​

Phase 2: Agent Implementation

1. Create agent definitions using Smol Agents
2. Implement the sandbox integration with E2B
3. Develop the workflow orchestration

​

Phase 3: CLI Integration

1. Update the CLI to use the new workflow
2. Implement progress reporting and output formatting
3. Add support for the --memory flag and history subcommand

​

Phase 4: Testing and Documentation

1. Write comprehensive tests for each module
2. Update documentation to reflect the new architecture
3. Create examples demonstrating the new workflow

​

References

• Smol Agents Documentation
• E2B Sandboxing
• Simulation Improvements Plan
• Issue #234: Simulation History Display Problems
• Issue #245: Improve Explanation Quality
• PR #256: Initial Prototype of Smol Agents Integration