-
Notifications
You must be signed in to change notification settings - Fork 15
Add Karate integration example with Testomat.io support #104
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| { | ||
| "permissions": { | ||
| "allow": [ | ||
| "mcp__context7__resolve-library-id", | ||
| "mcp__context7__get-library-docs", | ||
| "Bash(mkdir:*)" | ||
| ], | ||
| "deny": [], | ||
| "ask": [] | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,238 @@ | ||
| # Karate Testomat.io Integration Example | ||
|
|
||
| This example demonstrates how to properly integrate Karate tests with Testomat.io using unique test IDs for Scenario Outline examples. | ||
|
|
||
| ## Overview | ||
|
|
||
| This project contains a simple Karate test that demonstrates: | ||
| - Testing the Swagger Petstore API (https://petstore.swagger.io/v2/) | ||
| - Proper use of unique `@T...` test IDs in Scenario Outline examples | ||
| - Correct integration with Testomat.io for individual test tracking | ||
|
|
||
| ## Features | ||
|
|
||
| ### Test Scenarios | ||
| 1. **Get pet by ID** - Tests retrieving pets with different IDs (existing and non-existing) | ||
| 2. **Add new pet** - Tests creating new pets with different data | ||
|
|
||
| ### Unique Test IDs | ||
| Each Scenario Outline example has a unique test ID in the format `@T[8 alphanumeric characters]`: | ||
| - `T1a2b3c4d`, `T5e6f7g8h`, `T9i0j1k2l` (GET scenarios) | ||
| - `Tm3n4o5p6`, `Tq7r8s9t0`, `Tu1v2w3x4` (POST scenarios) | ||
|
|
||
| ## 🚀 Setup Framework From Scratch (In a Nutshell) | ||
|
|
||
| ### Quick Setup (5 minutes) | ||
|
|
||
| ```bash | ||
| # 1. Create project directory | ||
| mkdir my-karate-project | ||
| cd my-karate-project | ||
|
|
||
| # 2. Create Maven directory structure | ||
| mkdir -p src/test/java/com/example/karate | ||
| mkdir -p src/test/resources | ||
|
|
||
| # 3. Create pom.xml with Karate dependencies | ||
| cat > pom.xml << 'EOF' | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why do we need to do things like that? |
||
| <project xmlns="http://maven.apache.org/POM/4.0.0" | ||
| xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 | ||
| http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
| <groupId>com.example</groupId> | ||
| <artifactId>karate-project</artifactId> | ||
| <version>1.0.0</version> | ||
| <properties> | ||
| <maven.compiler.source>11</maven.compiler.source> | ||
| <maven.compiler.target>11</maven.compiler.target> | ||
| <karate.version>1.4.1</karate.version> | ||
| </properties> | ||
| <dependencies> | ||
| <dependency> | ||
| <groupId>com.intuit.karate</groupId> | ||
| <artifactId>karate-junit5</artifactId> | ||
| <version>${karate.version}</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| </dependencies> | ||
| <build> | ||
| <testResources> | ||
| <testResource> | ||
| <directory>src/test/java</directory> | ||
| <excludes><exclude>**/*.java</exclude></excludes> | ||
| </testResource> | ||
| </testResources> | ||
| </build> | ||
| </project> | ||
| EOF | ||
|
|
||
| # 4. Create test runner | ||
| cat > src/test/java/com/example/karate/TestRunner.java << 'EOF' | ||
| package com.example.karate; | ||
| import com.intuit.karate.Results; | ||
| import com.intuit.karate.Runner; | ||
| import org.junit.jupiter.api.Test; | ||
| import static org.junit.jupiter.api.Assertions.assertTrue; | ||
|
|
||
| public class TestRunner { | ||
| @Test | ||
| void testAll() { | ||
| Results results = Runner.path("classpath:") | ||
| .outputJunitXml(true) | ||
| .parallel(1); | ||
| assertTrue(results.getFailCount() == 0, results.getErrorMessages()); | ||
| } | ||
| } | ||
| EOF | ||
|
|
||
| # 5. Create your first .feature file | ||
| cat > src/test/resources/sample-test.feature << 'EOF' | ||
| Feature: Sample API Test | ||
|
|
||
| Background: | ||
| * url 'https://petstore.swagger.io/v2' | ||
|
|
||
| Scenario Outline: Get pet - <testCase> @<testId> | ||
| Given path 'pet', <petId> | ||
| When method get | ||
| Then status <expectedStatus> | ||
|
|
||
| Examples: | ||
| | testCase | petId | expectedStatus | testId | | ||
| | existing | 1 | 200 | T1a2b3c4d | | ||
| | nonexistent | 999 | 404 | T5e6f7g8h | | ||
| EOF | ||
|
|
||
| # 6. Run your first test | ||
| mvn test | ||
| ``` | ||
|
Comment on lines
+23
to
+110
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As it's a demo project that will not be or will be rarely updated, I would rather do this using Maven Archetype or, simpler, tell the user to clone the project with a ready-to-use structure. Both ways are more user-friendly and faster to use. |
||
|
|
||
| ### That's it! You now have: | ||
| ✅ Working Karate framework | ||
| ✅ Sample API test with unique test IDs | ||
| ✅ JUnit XML reports in `target/karate-reports/` | ||
| ✅ Ready for Testomat.io integration | ||
|
|
||
| ### Next Steps | ||
| ```bash | ||
| # Install report-xml tool (if not installed) | ||
| npm install -g @testomatio/reporter | ||
|
|
||
| # Import to Testomat.io | ||
| TESTOMATIO=your_key TESTOMATIO_TITLE_IDS=1 npx report-xml "target/karate-reports/**/*.xml" | ||
| ``` | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Java 11+ | ||
| - Maven 3.6+ | ||
| - Internet connection (to access Petstore API) | ||
| - Node.js (for Testomat.io import) | ||
|
|
||
| ## Running Tests | ||
|
|
||
| ### Local Execution | ||
|
|
||
| ```bash | ||
| # Run all tests | ||
| mvn test | ||
|
|
||
| # Run specific test class | ||
| mvn test -Dtest=KarateTest | ||
|
|
||
| # Run with specific Karate options | ||
| mvn test -Dkarate.options="--tags @T1a2b3c4d" | ||
| ``` | ||
|
|
||
| ### Alternative Feature File Execution | ||
|
|
||
| ```bash | ||
| # Run specific feature file directly | ||
| mvn test -Dkarate.options="classpath:simple-api-test.feature" | ||
| ``` | ||
|
|
||
| ## Testomat.io Integration | ||
|
|
||
| ### 1. Generate JUnit XML Reports | ||
|
|
||
| Tests automatically generate JUnit XML reports in `target/karate-reports/` when executed. | ||
|
|
||
| ### 2. Import to Testomat.io | ||
|
|
||
| ```bash | ||
| # Import test results using unique test IDs | ||
| TESTOMATIO=your_project_key TESTOMATIO_TITLE_IDS=1 npx report-xml "target/karate-reports/**/*.xml" | ||
| ``` | ||
|
|
||
| ### Key Integration Points | ||
|
|
||
| - **TESTOMATIO_TITLE_IDS=1**: Ensures Testomat.io uses the `@T...` IDs from scenario titles | ||
| - **Unique Test IDs**: Each example in Scenario Outline has a unique `@T...` ID | ||
| - **Proper Naming**: Scenario titles include `@<testId>` placeholder | ||
|
|
||
| ## Project Structure | ||
|
|
||
| ``` | ||
| karate-example/ | ||
| ├── pom.xml | ||
| ├── README.md | ||
| └── src/ | ||
| └── test/ | ||
| ├── java/ | ||
| │ └── com/ | ||
| │ └── testomat/ | ||
| │ └── karate/ | ||
| │ └── KarateTest.java | ||
| └── resources/ | ||
| └── simple-api-test.feature | ||
| ``` | ||
|
|
||
|
Comment on lines
+176
to
+191
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If this structure refers to the project the user creates in the first part of the README, then it's wrong. If it does not then as a user I can't see where I get this from. |
||
| ## API Details | ||
|
|
||
| ### Swagger Petstore API | ||
| - **Base URL**: https://petstore.swagger.io/v2 | ||
| - **Endpoints Used**: | ||
| - `GET /pet/{petId}` - Retrieve pet by ID | ||
| - `POST /pet` - Add new pet | ||
| - **Expected Behavior**: | ||
| - Pet IDs 1-10 typically exist (status 200) | ||
| - Pet ID 999+ typically don't exist (status 404) | ||
| - POST requests create new pets (status 200) | ||
|
|
||
| ## Test ID Format | ||
|
|
||
| Test IDs follow the pattern: `T` + 8 alphanumeric characters | ||
| - Must be unique within the project | ||
| - Used in scenario titles via `@<testId>` placeholder | ||
| - Enables proper test tracking in Testomat.io | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Common Issues | ||
|
|
||
| 1. **Tests fail due to network**: Ensure internet access to petstore.swagger.io | ||
| 2. **XML reports not generated**: Check `target/karate-reports/` directory exists | ||
| 3. **Testomat.io import issues**: Verify `TESTOMATIO_TITLE_IDS=1` is set | ||
|
|
||
| ### Verification | ||
|
|
||
| 1. **Run tests successfully**: `mvn test` should show 6 scenarios passed | ||
| 2. **Check XML reports**: Verify unique test case names contain `@T...` IDs | ||
| 3. **Testomat.io import**: Each scenario should appear as separate test | ||
|
|
||
| ## Example Output | ||
|
|
||
| When tests run successfully, you should see: | ||
| ``` | ||
| 6 scenarios (6 passed) | ||
| ``` | ||
|
|
||
| JUnit XML will contain entries like: | ||
| ```xml | ||
| <testcase name="Get pet by ID - Pet 1 @T1a2b3c4d" .../> | ||
| <testcase name="Get pet by ID - Pet 2 @T5e6f7g8h" .../> | ||
| ``` | ||
|
|
||
| This ensures each test appears separately in Testomat.io rather than as retries of the same test. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" | ||
| xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 | ||
| http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
|
|
||
| <modelVersion>4.0.0</modelVersion> | ||
| <groupId>com.testomat</groupId> | ||
| <artifactId>karate-example</artifactId> | ||
| <version>1.0.0</version> | ||
| <packaging>jar</packaging> | ||
|
|
||
| <properties> | ||
| <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> | ||
| <java.version>11</java.version> | ||
| <maven.compiler.version>3.8.1</maven.compiler.version> | ||
| <maven.surefire.version>3.0.0-M7</maven.surefire.version> | ||
| <karate.version>1.4.1</karate.version> | ||
| <junit.version>5.8.2</junit.version> | ||
| </properties> | ||
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>com.intuit.karate</groupId> | ||
| <artifactId>karate-junit5</artifactId> | ||
| <version>${karate.version}</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.junit.jupiter</groupId> | ||
| <artifactId>junit-jupiter</artifactId> | ||
| <version>${junit.version}</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| </dependencies> | ||
|
|
||
| <build> | ||
| <testResources> | ||
| <testResource> | ||
| <directory>src/test/java</directory> | ||
| <excludes> | ||
| <exclude>**/*.java</exclude> | ||
| </excludes> | ||
| </testResource> | ||
| <testResource> | ||
| <directory>src/test/resources</directory> | ||
| </testResource> | ||
| </testResources> | ||
|
|
||
| <plugins> | ||
| <plugin> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-compiler-plugin</artifactId> | ||
| <version>${maven.compiler.version}</version> | ||
| <configuration> | ||
| <encoding>UTF-8</encoding> | ||
| <source>${java.version}</source> | ||
| <target>${java.version}</target> | ||
| <compilerArgument>-Werror</compilerArgument> | ||
| </configuration> | ||
| </plugin> | ||
|
|
||
| <plugin> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-surefire-plugin</artifactId> | ||
| <version>${maven.surefire.version}</version> | ||
| <configuration> | ||
| <argLine>-Dfile.encoding=UTF-8</argLine> | ||
| </configuration> | ||
| </plugin> | ||
| </plugins> | ||
| </build> | ||
|
|
||
| </project> |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| package com.testomat.karate; | ||
|
|
||
| import com.intuit.karate.Results; | ||
| import com.intuit.karate.Runner; | ||
| import org.junit.jupiter.api.Test; | ||
| import static org.junit.jupiter.api.Assertions.assertTrue; | ||
|
|
||
| public class KarateTest { | ||
|
|
||
| @Test | ||
| void testAll() { | ||
| Results results = Runner.path("classpath:simple-api-test.feature") | ||
| .outputJunitXml(true) | ||
| .parallel(1); | ||
| assertTrue(results.getFailCount() == 0, results.getErrorMessages()); | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| Feature: Pet Store API Testing | ||
|
|
||
| # This feature demonstrates proper Karate-Testomat.io integration | ||
| # Each Scenario Outline example has a unique @T... ID for proper test separation | ||
|
|
||
| Background: | ||
| * url 'https://petstore.swagger.io/v2' | ||
|
|
||
| Scenario Outline: Get pet by ID - Pet <petId> @<testId> | ||
| # Test retrieving pets with different IDs and expected status codes | ||
| Given path 'pet', <petId> | ||
| When method get | ||
| Then status <expectedStatus> | ||
| # Log successful retrievals for debugging | ||
| * if (<expectedStatus> == 200) karate.log('Pet found:', response.name) | ||
| # Log not found cases for debugging | ||
| * if (<expectedStatus> == 404) karate.log('Pet not found for ID:', <petId>) | ||
|
|
||
| Examples: | ||
| | petId | expectedStatus | testId | | ||
| | 1 | 200 | T1a2b3c4d | | ||
| | 2 | 200 | T5e6f7g8h | | ||
| | 999 | 404 | T9i0j1k2l | | ||
|
|
||
| Scenario Outline: Add new pet - <petName> @<testId> | ||
| # Test creating new pets with different data | ||
| * def petData = { "id": <petId>, "name": "<petName>", "status": "available" } | ||
| Given path 'pet' | ||
| And request petData | ||
| When method post | ||
| Then status <expectedStatus> | ||
| # Validate response contains the pet data | ||
| * if (<expectedStatus> == 200) match response.name == '<petName>' | ||
| * if (<expectedStatus> == 200) match response.id == <petId> | ||
|
|
||
| Examples: | ||
| | petId | petName | expectedStatus | testId | | ||
| | 1001 | Buddy | 200 | Tm3n4o5p6 | | ||
| | 1002 | Rex | 200 | Tq7r8s9t0 | | ||
| | 1003 | Fluffy | 200 | Tu1v2w3x4 | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this does not belong here. It's local.