Test is container that holds name, arbitrary and value provided by said arbitrary (if you use constructor with None, it will generate value using arbitrary). The test also contains functions `shrink` and `to_str` (both just use the functions from arbitrary on the value held by the test).
Test is a container that holds name, arbitrary, and value provided by said arbitrary (if you use the constructor with value set to None, it will generate value using arbitrary). The test also contains functions `shrink` and `to_str` (both use the functions from arbitrary on the value held by the test).
# Tester
Tester is abstract class, the most important function `run_test(test: Test[G])` is not defined. Tester defines how shrinking works etc.
# Arbitraries
# Arbitraries
`Arbitrary[T]` defines an object capable of providing arbitrary value with type `T`, shrinking value of type `T` to a simpler instance and converting value of type `T` to `str`.
`Arbitrary[T]` defines an object capable of providing arbitrary value with type `T`, shriking a value of type `T` to a simpler instance and converting a value of type `T` to `str`.
To define your custom Arbitrary, either extend one of the predefined structures (below) or extend Arbitrary. Note that you have to define functions `shrink`, you don't have to define `to_str` but it will probably create undesirable output. The function `get` uses the value generator from the constructor.
The constructor of `Arbitrary[T]` takes ValueGenerator, function with type `Callable[[], T]`, this function is used in get(). There are some predefined value generatos, see ValueGenerator section.
The constructor of `Arbitrary[T]` takes ValueGenerator, function with type `Callable[[], T]`, this function is used in get(). There are some predefined value generators, see ValueGenerator section.
### Get
`get() -> T`
returns arbitrary value
### Shrink
`shrink(value: T) -> List[T]`
shrinks any value (created by the Arbitrary) to simpler value
shrinks any value (created by the Arbitrary) to a simpler value
### To string
`to_str(value: T) -> str`
converts the value to string
converts the value to string
## The arbitraries currently present are:
## The arbitraries currently present are:
### NumberA(min_value, max_value)
get returns: number in [min_value, max_value)
shrinks value to: val // 2, val moved towards 0
### ConstantA(constant)
get returns: constant
get returns: constant
shrinks value to: nothing
### TupleA(arbs)
get returns: tuple where i-th value uses i-th arbitrary from arbs
shrinks value to: list of different tuples with shrunk values
shrinks value to: list of different tuples with shrunk values
### ListA(arb, min_size, max_size)
get returns: list with length [min_size, max_size)
shrinks value to: shorter lists
### ChoiceA(choices)
get returns: tuple (index of chosen arb, value generated by that arb), choice is random
shrinks value to: tuples with shrunk values (index of the chosen arb, value shrunk using said arb)
# ValueGenerator
There are some predefined factories. Note that you have to CALL the factory, to recieve value generator.
There are some predefined factories. Note that you have to CALL the factory to receive the value generator.
\*- = returns generator that creates
*- = returns generator that creates
## predefined value generators creators
## predefined value generators creators
### Atomic
### Atomic
-`integer_gen(min_value: int, max_value: int)`*- random integer in range [min_value, max_value)
-`pos_integer_gen(max_value: int)`*- random integer in range [1, max_valaue)
-`lower_case_let_gen()`*- lower case letter
-`upper_case_let_gen()`*- upper case letter
-`integer_gen(min_value: int, max_value: int)`\*- random integer in range [min_value, max_value)
-`pos_integer_gen(max_value: int)`\*- random integer in range [1, max_valaue)
-`lower_case_let_gen()`\*- lower case letter
-`upper_case_let_gen()`\*- upper case letter
### Generic
-`constant_gen(const: T)`*- the constant
-`oneof_gen(constants: List[T])`*- one of the constants
-`vector_gen(gen: Callable[[], T], size: int)`*- list with constant size with elements generated by the generator
-`listof_gen(gen: Callable[[], T], min_size: int, max_size: int)`*- list with random size [min_value, max_value) with elements generated by the generator
-`tupleof_gen(generators: List[ValueGenerator])`*- tuple created from the values of the different generators
-`choices_gen(choices: List[Tuple[int, Callable[[], T]]])`*- choices are list of tuples in form (weight, generator), generates value created by one of the generators, chooses using weighted random
### Generic
# Testing with reference implementation
-`constant_gen(const: T)`\*- the constant
-`oneof_gen(constants: List[T])`\*- one of the constants
-`vector_gen(gen: Callable[[], T], size: int)`\*- list with constant size with elements generated by the generator
-`listof_gen(gen: Callable[[], T], min_size: int, max_size: int)`\*- list with random size [min_value, max_value) with elements generated by the generator
-`tupleof_gen(generators: List[ValueGenerator])`\*- tuple created from the values of the different generators
-`choices_gen(choices: List[Tuple[int, Callable[[], T]]])`\*- choices are list of tuples in form (weight, generator), generates value created by one of the generators, chooses using weighted random
# Testing with a reference implementation
## RefImlTester
Tester that runs tests on your implementation and the reference implementation, then runs your 'process output tests' on the outputs. More below.
Tester that runs tests on your implementation and the reference implementation then runs your 'process output tests' on the outputs. More below.
### ProcessOutputTest
Type is `Callable[[ProcessOutput, ProcessOutput], bool]`, it takes the outputs (in format provided by the `subprocess.run`) and returns bool.
There are some presets:
There are some presets:
-`stdout_equality()`: returns test, that only compares the raw stdout
-`stdout_processed_equality(test_preprocessor, exp_preprocessor)`: returns test that compares processed stdouts. `test_preprocessor` and `exp_preprocessor` are `Callable[[str], str]` that "process" the stdout (you have to define these, there is `str_id` that returns the input unchaged `x -*> x`)
...
...
@@ -104,20 +114,20 @@ def ignore_numbers(x: str):
returnre.sub(r'\d+','',x)
# `program_tst` and `program_exp` are absolute paths pointing to the executables
# runs the test, returns (reduced test, number of shrinks) tuple
# runs the test, returns (reduced test, number of shrinks) tuple
reduced_tests=tester.run()
# prints the failed tests, first the name, the number of shrinks, the test,
# then saves it as .tst file
# prints the failed tests, first the name, the number of shrinks, the test,
# then saves it as .tst file
fortest,shrinksinreduced_tests:
print(test.name,shrinks)
print(test.to_str())
...
...
@@ -127,9 +137,9 @@ for test, shrinks in reduced_tests:
tester.save_tests()
```
## Command
## Command
Command is very similar to TupleA, it takes list of arbitraries, and provies 'tupled' values. The difference is, that Command's `to_str` returns by space separated values, instead of the standard ().
Command is very similar to TupleA. It takes a list of arbitraries and returns 'tupled' values. The difference is that Command's `to_str` returns values separated by space, instead of the standard (,) format.
Command can be provided with symbolic table `Dict[str, List[int]]`, table `symbolic name -> list of indeces`. This table can be later used in `get_symbol(self, symbol: str, value: Tuple[Generatable, ...])` to get value from the tuple by name (this is used in CommandSequence)
...
...
@@ -139,7 +149,7 @@ CommandSequence takes in commands in form of choice-style list of tuples. It is
## Example use
```python
```python
# This creates command in the form:
# git commit -m <string[5,10)>
git_commit=Command([
...
...
@@ -172,7 +182,7 @@ messages = [''.join(m) for m in messages]
