|
1 | | -# Natural Language to Robot Framework |
| 1 | +# Mark 1 - Natural Language to Robot Framework |
2 | 2 |
|
3 | | -This project is a sophisticated, AI-driven framework that translates natural language test cases into executable Robot Framework code. It leverages a multi-agent system to interpret, plan, validate, and self-correct test automation scripts, providing a seamless experience for testers and developers. |
| 3 | + |
| 4 | + |
| 5 | + |
| 6 | + |
4 | 7 |
|
5 | | ---- |
| 8 | +Transform natural language test descriptions into executable Robot Framework code using advanced AI agents. Mark 1 is an intelligent test automation platform that bridges the gap between human language and automated testing. |
6 | 9 |
|
7 | | -## Key Features |
| 10 | +## ✨ Key Features |
8 | 11 |
|
9 | | -- **Natural Language Conversion:** Simply describe a test case in plain English, and Mark 1 will generate the corresponding Robot Framework script. |
10 | | -- **Multi-Agent System:** Utilizes a robust pipeline of specialized AI agents (Planner, Identifier, Validator) to ensure high-quality code generation. |
11 | | -- **Self-Correction Loop:** If the generated code is syntactically incorrect, the system automatically attempts to fix its own mistakes, significantly improving reliability. |
12 | | -- **Containerized Execution:** Tests are executed in a clean, isolated Docker container, guaranteeing consistency and eliminating the "it works on my machine" problem. |
13 | | -- **Flexible Model Support:** Seamlessly switch between local LLMs (via Ollama) and powerful online models like Google's Gemini for maximum flexibility. |
14 | | -- **Detailed Debugging:** Automatically saves detailed HTML logs and reports from every test run, allowing for easy inspection and debugging of test failures. |
| 12 | +- 🤖 **Multi-Agent AI System**: Sophisticated pipeline with specialized agents for planning, identification, validation, and self-correction |
| 13 | +- 🔄 **Self-Healing Tests**: Automatic error detection and correction through intelligent validation loops |
| 14 | +- 🐳 **Containerized Execution**: Clean, isolated test runs in Docker containers for consistent results |
| 15 | +- 🌐 **Flexible Model Support**: Works with both local LLMs (via Ollama) and cloud models (Google Gemini) |
| 16 | +- 📊 **Detailed Reporting**: Comprehensive HTML logs and reports for easy debugging |
| 17 | +- ⚡ **Real-time Streaming**: Live progress updates and instant feedback |
| 18 | +- 🎯 **Smart Element Detection**: AI-powered web element locator generation |
15 | 19 |
|
16 | | ---- |
| 20 | +## 🏗️ Architecture |
17 | 21 |
|
18 | | -## Architecture Overview |
| 22 | +Mark 1 employs a sophisticated multi-agent workflow: |
19 | 23 |
|
20 | | -Mark 1 employs a sophisticated multi-agent workflow to ensure the generated code is both accurate and robust. |
| 24 | +1. **Step Planner Agent** - Decomposes natural language into structured test plans |
| 25 | +2. **Element Identifier Agent** - Generates optimal web element locators |
| 26 | +3. **Code Assembler Agent** - Creates syntactically correct Robot Framework code |
| 27 | +4. **Validator Agent** - Ensures code quality and correctness |
| 28 | +5. **Self-Correction Orchestrator** - Automatically fixes validation errors |
21 | 29 |
|
22 | | -1. **Step Planner Agent:** The initial agent receives the natural language query. Its job is to decompose the query into a high-level, structured plan, identifying each distinct action required for the test case. |
23 | | -2. **Element Identifier Agent:** This agent takes the plan and, for each step involving a UI element, uses an AI model to determine the best and most stable web locator (e.g., CSS selector, XPath). |
24 | | -3. **Code Assembler Agent:** A deterministic agent that assembles the planned and located steps into a syntactically-structured `.robot` file. |
25 | | -4. **Validator Agent:** Before execution, this crucial agent inspects the generated code against a set of rules (e.g., ensuring keywords have the correct arguments). |
26 | | -5. **Self-Correction Orchestrator:** If the Validator Agent finds a flaw, the orchestrator feeds the error back to the Step Planner Agent, which then attempts to generate a corrected plan. This loop can run multiple times to resolve errors automatically. |
| 30 | +## 🚀 Quick Start |
27 | 31 |
|
28 | | ---- |
| 32 | +### Prerequisites |
29 | 33 |
|
30 | | -## Getting Started |
| 34 | +- **Python 3.9+** - [Download Python](https://python.org/downloads/) |
| 35 | +- **Docker** - [Install Docker](https://docs.docker.com/get-docker/) |
| 36 | +- **Git** - [Install Git](https://git-scm.com/downloads) |
| 37 | +- **(Optional) Ollama** - For local AI models [Install Ollama](https://ollama.com/) |
| 38 | + |
| 39 | +### Installation |
| 40 | + |
| 41 | +1. **Clone the repository** |
| 42 | + ```bash |
| 43 | + git clone <repository-url> |
| 44 | + cd natural-language-robot-framework |
| 45 | + ``` |
| 46 | + |
| 47 | +2. **Configure your AI model** |
| 48 | + ```bash |
| 49 | + cp backend/.env.example backend/.env |
| 50 | + ``` |
| 51 | + |
| 52 | + **For Google Gemini (Recommended)**: |
| 53 | + ```env |
| 54 | + MODEL_PROVIDER=online |
| 55 | + GEMINI_API_KEY="your-gemini-api-key-here" |
| 56 | + ONLINE_MODEL=gemini-1.5-pro-latest |
| 57 | + ``` |
31 | 58 |
|
32 | | -### Prerequisites |
| 59 | + **For Local Models (Ollama)**: |
| 60 | + ```env |
| 61 | + MODEL_PROVIDER=local |
| 62 | + LOCAL_MODEL=llama3 |
| 63 | + ``` |
33 | 64 |
|
34 | | -- **Docker:** Required for running the containerized test execution environment. [Install Docker](https://docs.docker.com/get-docker/). |
35 | | -- **Python 3.9+** |
36 | | -- **(Optional) Ollama:** If you wish to use a local LLM, you must have Ollama installed and running. [Install Ollama](https://ollama.com/). |
37 | | - |
38 | | -### Installation & Setup |
39 | | - |
40 | | -1. **Clone the Repository** |
41 | | - ```bash |
42 | | - git clone <repository-url> |
43 | | - cd <repository-directory> |
44 | | - ``` |
45 | | - |
46 | | -2. **Configure Environment Variables** |
47 | | - Copy the example `.env` file: |
48 | | - ```bash |
49 | | - cp backend/.env.example backend/.env |
50 | | - ``` |
51 | | - Now, edit `backend/.env` to configure your desired model provider. |
52 | | - |
53 | | - **For Local Models (via Ollama):** |
54 | | - ```dotenv |
55 | | - MODEL_PROVIDER=local |
56 | | - LOCAL_MODEL=llama3 # The name of the model you have pulled with Ollama |
57 | | - ``` |
58 | | - |
59 | | - **For Online Models (Google Gemini):** |
60 | | - ```dotenv |
61 | | - MODEL_PROVIDER=online |
62 | | - GEMINI_API_KEY="YOUR_GEMINI_API_KEY" |
63 | | - # ONLINE_MODEL=gemini-1.5-pro-latest # (Optional) Specify a different Gemini model |
64 | | - ``` |
65 | | - |
66 | | -3. **Make Scripts Executable** (for Linux/macOS) |
67 | | - ```bash |
68 | | - chmod +x run.sh test.sh |
69 | | - ``` |
70 | | - |
71 | | -### Running the Application |
72 | | - |
73 | | -1. **Start the Server** |
74 | | - ```bash |
75 | | - ./run.sh |
76 | | - ``` |
77 | | - This script automatically creates a Python virtual environment, installs all dependencies, and starts the FastAPI server on `http://localhost:5000`. |
78 | | - |
79 | | -2. **Access the Web Interface** |
80 | | - Open your browser and navigate to `http://localhost:5000`. |
| 65 | +3. **Start the application** |
| 66 | + ```bash |
| 67 | + # Linux/macOS |
| 68 | + chmod +x run.sh |
| 69 | + ./run.sh |
81 | 70 |
|
82 | | ---- |
| 71 | + # Windows (Git Bash) |
| 72 | + bash run.sh |
| 73 | + ``` |
83 | 74 |
|
84 | | -## Usage |
| 75 | +4. **Access the web interface** |
| 76 | + Open your browser to `http://localhost:5000` |
85 | 77 |
|
86 | | -1. Enter your test case in plain English in the text area (e.g., "search for the latest news on google and verify the title"). |
87 | | -2. Click "Generate and Run". |
88 | | -3. The backend will process the query, generate the Robot Framework code, and execute it in a Docker container. |
89 | | -4. The results, including the generated code and execution logs, will be displayed on the page. |
| 78 | +## 💡 Usage Examples |
90 | 79 |
|
91 | | -### Debugging Failed Tests |
| 80 | +Simply describe your test in natural language: |
92 | 81 |
|
93 | | -If a test fails, you can find detailed information in the `robot_tests` directory. The framework automatically saves the following files from the container to your local machine: |
94 | | -- `log.html`: A detailed, step-by-step log of the test execution with expandable views. |
95 | | -- `report.html`: A high-level report of the test run. |
96 | | -- `output.xml`: The machine-readable XML output from Robot Framework. |
| 82 | +### Basic Examples |
| 83 | +- *"Open Google, search for 'Robot Framework tutorials', and click the first result"* |
| 84 | +- *"Navigate to GitHub, search for 'selenium automation', and star the top repository"* |
| 85 | +- *"Go to YouTube, search for 'Python automation', and play the first video"* |
97 | 86 |
|
98 | | -Simply open `log.html` in your browser to get a complete picture of the failure. |
| 87 | +### Advanced Examples |
| 88 | +- *"Visit Amazon, search for 'wireless headphones', filter by rating above 4 stars, and add the top result to cart"* |
| 89 | +- *"Open LinkedIn, search for 'QA Engineer' jobs in San Francisco, and apply filters for remote work"* |
| 90 | +- *"Navigate to Stack Overflow, search for 'pytest fixtures', and upvote the most helpful answer"* |
99 | 91 |
|
100 | | ---- |
| 92 | +## 🛠️ Configuration |
| 93 | + |
| 94 | +### Environment Variables |
| 95 | + |
| 96 | +| Variable | Description | Default | |
| 97 | +|----------|-------------|---------| |
| 98 | +| `MODEL_PROVIDER` | AI model provider (`online` or `local`) | `online` | |
| 99 | +| `GEMINI_API_KEY` | Google Gemini API key | Required for online | |
| 100 | +| `ONLINE_MODEL` | Gemini model name | `gemini-1.5-pro-latest` | |
| 101 | +| `LOCAL_MODEL` | Ollama model name | `llama3` | |
| 102 | +| `SECONDS_BETWEEN_API_CALLS` | Rate limiting delay | `0` | |
| 103 | + |
| 104 | +### Getting API Keys |
| 105 | + |
| 106 | +**Google Gemini API Key**: |
| 107 | +1. Visit [Google AI Studio](https://aistudio.google.com/app/apikey) |
| 108 | +2. Create a new API key |
| 109 | +3. Copy the key to your `.env` file |
| 110 | + |
| 111 | +## 🧪 Testing |
| 112 | + |
| 113 | +Test the API endpoint directly: |
| 114 | + |
| 115 | +```bash |
| 116 | +chmod +x test.sh |
| 117 | +./test.sh |
| 118 | +``` |
| 119 | + |
| 120 | +Or use curl: |
| 121 | + |
| 122 | +```bash |
| 123 | +curl -X POST \ |
| 124 | + -H "Content-Type: application/json" \ |
| 125 | + -d '{"query": "go to google.com and search for python tutorials", "model": "gemini-1.5-pro-latest"}' \ |
| 126 | + http://localhost:5000/generate-and-run |
| 127 | +``` |
| 128 | + |
| 129 | +## 📁 Project Structure |
101 | 130 |
|
102 | | -## License |
| 131 | +``` |
| 132 | +├── backend/ |
| 133 | +│ ├── main.py # FastAPI application |
| 134 | +│ ├── robot_generator.py # Multi-agent AI system |
| 135 | +│ ├── requirements.txt # Python dependencies |
| 136 | +│ └── .env # Configuration |
| 137 | +├── frontend/ |
| 138 | +│ ├── index.html # Web interface |
| 139 | +│ ├── script.js # Frontend logic |
| 140 | +│ └── style.css # Styling |
| 141 | +├── robot_tests/ # Generated test files and reports |
| 142 | +├── run.sh # Startup script |
| 143 | +└── test.sh # Testing script |
| 144 | +``` |
103 | 145 |
|
104 | | -This project is licensed under the Apache License, Version 2.0. See the [LICENSE](LICENSE) file for details. |
| 146 | +## 🐛 Debugging |
105 | 147 |
|
106 | | -## Contributing |
| 148 | +When tests fail, detailed logs are automatically saved: |
| 149 | + |
| 150 | +- `robot_tests/{run_id}/log.html` - Step-by-step execution log |
| 151 | +- `robot_tests/{run_id}/report.html` - High-level test report |
| 152 | +- `robot_tests/{run_id}/output.xml` - Machine-readable results |
| 153 | + |
| 154 | +Open `log.html` in your browser for comprehensive failure analysis. |
| 155 | + |
| 156 | +## 🤝 Contributing |
| 157 | + |
| 158 | +We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details. |
| 159 | + |
| 160 | +### Development Setup |
| 161 | + |
| 162 | +1. Fork the repository |
| 163 | +2. Create a feature branch: `git checkout -b feature/amazing-feature` |
| 164 | +3. Make your changes |
| 165 | +4. Run tests: `./test.sh` |
| 166 | +5. Commit: `git commit -m 'Add amazing feature'` |
| 167 | +6. Push: `git push origin feature/amazing-feature` |
| 168 | +7. Open a Pull Request |
| 169 | + |
| 170 | +Before contributing, please read our [Contributor License Agreement](CLA.md). |
| 171 | + |
| 172 | +## 📄 License |
| 173 | + |
| 174 | +This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details. |
| 175 | + |
| 176 | +## 🆘 Support |
| 177 | + |
| 178 | +- 📚 **Documentation**: Check this README and code comments |
| 179 | +- 🐛 **Issues**: Report bugs via GitHub Issues |
| 180 | +- 💬 **Discussions**: Join our GitHub Discussions |
| 181 | +- 📧 **Email**: Contact the maintainers |
| 182 | + |
| 183 | +## 🎯 Roadmap |
| 184 | + |
| 185 | +- [ ] Support for mobile app testing |
| 186 | +- [ ] Integration with CI/CD pipelines |
| 187 | +- [ ] Visual test result dashboards |
| 188 | +- [ ] Multi-language test generation |
| 189 | +- [ ] Advanced AI model fine-tuning |
| 190 | +- [ ] Cloud deployment options |
| 191 | + |
| 192 | +## ⭐ Show Your Support |
| 193 | + |
| 194 | +If Mark 1 helps streamline your testing workflow, please consider: |
| 195 | +- ⭐ Starring this repository |
| 196 | +- 🐛 Reporting issues |
| 197 | +- 💡 Suggesting new features |
| 198 | +- 🤝 Contributing code |
| 199 | + |
| 200 | +--- |
107 | 201 |
|
108 | | -We welcome contributions to this project! Please see the [CONTRIBUTING.md](CONTRIBUTING.md) file for details on how to get started and to review our contributor license agreement. |
| 202 | +**Built with ❤️ for the test automation community** |
0 commit comments