- Property based testing (with hypothesis)
- Fuzz testing (with afl, aka. American Fuzzy Lop)
- Stress testing
- Mutation testing to “test the tests” (with cosmic-ray)
- Takes hours or days to run
- Run periodically as a test audit
- Results need manual review
- Very helpful to do this on a fast test suite
Original video by PyCon 2017:
“Speaker: James Saryerwinnie
Unit, functional, and integration tests are great first steps towards improving the quality of your python project. Ever wonder if there’s even more you can do? Maybe you’ve heard of property-based testing, fuzzing, and mutation testing but you’re unsure exactly how they can help you. In this talk we’ll cover additional types of tests that can help improve the quality and robustness of your python projects: property-based testing, fuzz testing, stress testing, long term reliability testing, and mutation testing.
We’ll also go beyond just covering what these tests are. For each of the test types above, I’ll give you real world examples from open source software that I maintain that shows you the types of bugs each test type can find. I’ll also show you how you can integrate these tests into your Travis CI and/or Jenkins environment.
Slides can be found at: https://speakerdeck.com/pycon2017 and https://github.com/PyCon/2017-slides”