Skip to content
This repository was archived by the owner on Feb 2, 2022. It is now read-only.

Commit 44a6e35

Browse files
authored
Explain modes in raft-restler-relationship.md (#222)
1 parent 585a5dc commit 44a6e35

File tree

1 file changed

+81
-15
lines changed

1 file changed

+81
-15
lines changed

docs/raft-restler-relationship.md

Lines changed: 81 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -3,31 +3,97 @@
33
The RAFT team owns and maintains the RESTler agent that runs on RAFT platform.
44
In order to provide a consistent platform for supporting multiple test tools there are some differences between how RESTler is configured when running using RAFT versus when running RESTler directly.
55

6-
RAFT supports all of the same configuration values that RESTler supports.
7-
RESTler uses a mix of camel case and underscore parameters that it accepts as part of it's run configurations. On the other hand RAFT only uses camel case parameters. For example RESTler uses parameter `restler_custom_payload` where the same parameter is called `restlerCustomPayload` in RAFT job definition.
6+
RAFT exposes almost all of the same configuration values that RESTler exposes. Some less frequently
7+
used values are not currently exposed but can easily be added if needed.
8+
RESTler uses a mix of camel case and underscore parameters that it accepts as part of it's run configurations.
9+
RAFT by contrast only uses camel case parameters. As an example RESTler uses the parameter `restler_custom_payload`
10+
where as the same parameter is called `restlerCustomPayload` in a RAFT job definition.
811

9-
You can find the RESTler documentation at
10-
https://github.com/microsoft/restler-fuzzer/tree/main/docs/user-guide
11-
12-
When using RESTler documentation for configuring RESTler tasks use the RAFT swagger definition for paramater name conversion.
12+
You can find the RESTler documentation [here](https://github.com/microsoft/restler-fuzzer/tree/main/docs/user-guide).
1313

14+
When using RESTler documentation for configuring RESTler tasks use the RAFT swagger definition for parameter name conversion.
1415
The RAFT swagger definition can be found using this URL : *https://\<my-deployment\>-raft-apiservice.azurewebsites.net/swagger/index.html*
1516

1617

17-
RAFT will do a DNS lookup on the host parameter, specified in the job definition file,
18-
on your behalf and fill in the IP parameter for you. RAFT also defaults the port number
18+
There are a few convenience things that RAFT will do for you. RAFT will do a DNS lookup on the
19+
host parameter specified in the job definition file and fill in the IP parameter for you.
20+
RAFT also defaults the port number
1921
to 443 when using SSL and 80 when not using a secure connection. If for some reason you
20-
find that you still need to specify the IP and port number then manually provided values
21-
in TargetEndpoint configuration will override any lookup or default values.
22+
find that you still need to specify the IP and port number, you can manually provide the values
23+
you need in the `TargetEndpoint` configuration. These values will override any lookup or default values.
2224

2325
## RESTler mode of operation
2426

25-
RESTler needs to compile the Swagger specifications into RESTler grammar as a first step. The output of the compile step is then consumed by any of the following steps: Test, TestFuzzLean, Fuzz.
26-
To enable passing of data from one step to the next RAFT allows any file share in the storage account to be mounted by any task.
27-
This way RAFT jobs can be executed in a "pipeline" manner by passing the output of the Compile job as input to the Test, TestFuzzLean, or Fuzz jobs.
27+
RAFT exposes several modes that can be used with RESTler.
28+
- Compile</br>
29+
This mode is used to convert the OpenAPI (or swagger) spec into a RESTler grammar. You must run `compile` before any of the
30+
other modes can be used as they will use the grammar produced.
31+
- Test</br>
32+
Using this mode will cause RESTler to exercise each API once. Using this mode you can quickly determine if RESTler is able
33+
to call all your APIs without errors, this is reported as spec coverage.
34+
See [How to Improve coverage](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/ImprovingCoverage.md).
35+
- TestAllCombinations</br>
36+
By default, test mode will try to execute each request successfully once. This means that, if there are 5 possible values
37+
for a parameter, and the request is successfully executed when passing the first value, the remaining 4 will not be tested.
38+
In some cases, such as for differential regression testing, it is desired to test all of the specified parameter values in
39+
Test mode. Use the TestAllCombinations mode try all parameter values (up to maxCombinations).
40+
The results for all parameter combinations will be reported in the spec coverage file.
41+
- TestFuzzLean</br>
42+
In this mode, RESTler executes every endpoint+method once in the compiled RESTler grammar with a default set of checkers to see
43+
if bugs can be found quickly.
44+
- Fuzz</br>
45+
In Fuzz mode, RESTler will fuzz the service under test during a longer period of time with the goal of finding more bugs and
46+
issues (resource leaks, perf degradation, backend corruptions, etc.). Warning: The Fuzz mode is the more aggressive and
47+
may create outages in the service under test if the service is poorly implemented.
48+
- Replay</br>
49+
Used to replay a bug found on a previous run.
50+
- FuzzRandomWalk</br>
51+
This mode uses a random walk to traverse the graph of requests.
52+
- FuzzBfsCheap</br>
53+
Using a breadth first search mode similar to what is used in the `Fuzz` mode to traverse the graph of requests
54+
but does it in a "cheap" way.
55+
56+
RESTler needs to compile the Swagger specifications into RESTler grammar as a first step.
57+
The output of the compile step is then consumed by any of the following steps:
58+
Test, TestAllCombinations, TestFuzzLean, Fuzz, Replay, FuzzRandomWalk, FuzzBfsCheap.
2859

29-
A compile job produces a job ID and a file share is created and named using the job ID.
30-
This makes it possible for you to take the output written to a file share of one job and mount it to use as input to another.
60+
There are several ways to connect the RESTler compile task output to the other tasks.
61+
One way is to pass the compile output data via a file share in the storage account.
62+
This is useful when your compile step takes a long time and you don't want to run the compile task every time.
63+
64+
For example, a compile job produces a job ID and by default a file share is created and named using the job ID.
65+
This makes it possible for you to take the output written to that file share and mount it to use as input to another job.
3166
The diagram below illustrates this behavior in the job definition files.
3267

3368
![RESTler ](images/restler-configs-flow.png)
69+
70+
If your compile step executes quickly, it can be much easier to simply in-line the
71+
tasks like this in your job definition file:</br>
72+
```json
73+
"tasks": [
74+
{
75+
"targetConfiguration" : {
76+
"endpoint" : "https://petstore3.swagger.io",
77+
"apiSpecifications": [
78+
"https://petstore3.swagger.io/api/v3/openapi.json"
79+
]
80+
},
81+
"toolName": "RESTler",
82+
"outputFolder": "restler-pestorev3",
83+
"toolConfiguration": {
84+
"tasks": [
85+
{"task": "Compile"},
86+
87+
{"task" : "Test"},
88+
89+
{
90+
"task" : "Fuzz",
91+
"runConfiguration" : {
92+
"Duration" : "00:10:00"
93+
}
94+
}
95+
]
96+
}
97+
}
98+
]
99+
```

0 commit comments

Comments
 (0)