Newer
Older
# KDTesting
Python library for testing applications using IO.
# Test
Test is container that holds name, arbitrary and value provided by said arbitrary (if you use constructor with None, it will generate value using arbitrary). The test also contains functions `shrink` and `to_str` (both just use the functions from arbitrary on the value held by the test).
Tester is abstract class, the most important function `run_test(test: Test[G])` is not defined. Tester defines how shrinking works etc.
`Arbitrary[T]` defines an object capable of providing arbitrary value with type `T`, shrinking value of type `T` to a simpler instance and converting value of type `T` to `str`.
To define your custom Arbitrary, either extend one of the predefined structures (below) or extend Arbitrary. Note that you have to define functions `shrink`, you don't have to define `to_str` but it will probably create undesirable output. The function `get` uses the value generator from the constructor.
The constructor of `Arbitrary[T]` takes ValueGenerator, function with type `Callable[[], T]`, this function is used in get(). There are some predefined value generatos, see ValueGenerator section.
shrinks any value (created by the Arbitrary) to simpler value
### To string
`to_str(value: T) -> str`
converts the value to string
## The arbitraries currently present are:
### NumberA(min_value, max_value)
get returns: number in [min_value, max_value)
shrinks value to: val // 2, val moved towards 0
### ConstantA(constant)
get returns: constant
shrinks value to: nothing
### TupleA(arbs)
get returns: tuple where i-th value uses i-th arbitrary from arbs
shrinks value to: list of different tuples with shrunk values
### ListA(arb, min_size, max_size)
get returns: list with length [min_size, max_size)
shrinks value to: shorter lists
### ChoiceA(choices)
get returns: tuple (index of chosen arb, value generated by that arb), choice is random
shrinks value to: tuples with shrunk values (index of the chosen arb, value shrunk using said arb)
# ValueGenerator
There are some predefined factories. Note that you have to CALL the factory, to recieve value generator.
*- = returns generator that creates
## predefined value generators creators
### Atomic
- `integer_gen(min_value: int, max_value: int)` *- random integer in range [min_value, max_value)
- `pos_integer_gen(max_value: int)` *- random integer in range [1, max_valaue)
- `lower_case_let_gen()` *- lower case letter
- `upper_case_let_gen()` *- upper case letter
### Generic
- `constant_gen(const: T)` *- the constant
- `oneof_gen(constants: List[T])` *- one of the constants
- `vector_gen(gen: Callable[[], T], size: int)` *- list with constant size with elements generated by the generator
- `listof_gen(gen: Callable[[], T], min_size: int, max_size: int)` *- list with random size [min_value, max_value) with elements generated by the generator
- `tupleof_gen(generators: List[ValueGenerator])` *- tuple created from the values of the different generators
- `choices_gen(choices: List[Tuple[int, Callable[[], T]]])` *- choices are list of tuples in form (weight, generator), generates value created by one of the generators, chooses using weighted random
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
# Testing with reference implementation
## RefImlTester
Tester that runs tests on your implementation and the reference implementation, then runs your 'process output tests' on the outputs. More below.
### ProcessOutputTest
Type is `Callable[[ProcessOutput, ProcessOutput], bool]`, it takes the outputs (in format provided by the `subprocess.run`) and returns bool.
There are some presets:
- `stdout_equality()`: returns test, that only compares the raw stdout
- `stdout_processed_equality(test_preprocessor, exp_preprocessor)`: returns test that compares processed stdouts. `test_preprocessor` and `exp_preprocessor` are `Callable[[str], str]` that "process" the stdout (you have to define these, there is `str_id` that returns the input unchaged `x -*> x`)
### Example use
```python
def ignore_numbers(x: str):
return re.sub(r'\d+', '', x)
# `program_tst` and `program_exp` are absolute paths pointing to the executables
# 3 - verbosity (default is 5, too much)
tester = RefImplTester(program_tst, program_exp, 3)
tester.add_output_test(stdout_processed_eqality(ignore_numbers, ignore_numbers))
for i in range(5):
# `command_sequence` is a custom arbitrary that returns sequence of
# `\n` delimitered commands
tester.add_test(Test(f"all_{i}", command_sequence))
# runs the test, returns (reduced test, number of shrinks) tuple
reduced_tests = tester.run()
# prints the failed tests, first the name, the number of shrinks, the test,
# then saves it as .tst file
for test, shrinks in reduced_tests:
print(test.name, shrinks)
print(test.to_str())
tester.save_shrunk(reduced_tests)
# also saves the original tests
tester.save_tests()
```
## Command
Command is very similar to TupleA, it takes list of arbitraries, and provies 'tupled' values. The difference is, that Command's `to_str` returns by space separated values, instead of the standard ().
Command can be provided with symbolic table `Dict[str, List[int]]`, table `symbolic name -> list of indeces`. This table can be later used in `get_symbol(self, symbol: str, value: Tuple[Generatable, ...])` to get value from the tuple by name (this is used in CommandSequence)
## CommandSequence
CommandSequence takes in commands in form of choice-style list of tuples. It is very similar to ListA, but it renders the values separated by '\n'. It also contains the `get_symbol(self, symbol: str, values: List[Tuple[int, Generatable]])` function, for retrieving symbols. This time it returns all the values present in all the commands.
## Example use
```python
# This creates command in the form:
# git commit -m <string[5,10)>
git_commit = Command([
ConstantA("git commit -m"),
StringA(5, 10)
], {
"message": [1]
})
# This creates command in the form:
# git pull
git_pull = Command([
ConstantA("git pull")
])
# This creates sequence with pull being twice as likely as happen as commit
command_sequence = CommandSequence([
(1, git_commit),
(2, git_pull)
], 10, 15)
# Let't get random value from the command sequence
random_command_sequence = command_sequence.get()
print(command_sequence.to_str(random_command_sequence))
# Let's get the symbol message from commits
messages = command_sequence.get_symbol("message", random_command_sequence)
messages = [''.join(m) for m in messages]
print(messages)
```
Output
```
git commit -m nYWMY
git pull
git pull
git pull
git commit -m lpejbIJr
git pull
git pull
git pull
git commit -m ZZNfJHHF
git pull
git commit -m UxkaTibR
['nYWMY', 'lpejbIJr', 'ZZNfJHHF', 'UxkaTibR']
```