FsCheck

Property based testing

So you might be first wondering what property based testing means. Simply put you describe properties of your functions, and then you bombard your code with data trying to prove or disprove these properties.

Here’s an example, let’s say we have a function like this:

let removeNegatives list =
        let rec removeNegatives' list newList =
            match list with
            | [] -> newList
            | head :: tail when head < 0 -> removeNegatives' tail newList 
            | head :: tail -> removeNegatives' tail (head :: newList)
        removeNegatives' list []

And we want to say this function has the following three properties:

  1. If you pass in a list, it returns the list without any negative numbers in it.
  2. If you pass in a list without any negative numbers in it, you get exactly the same list back (this is implied from the first property, but let’s make it explict anyway).
  3. The function will not re-order the elements.

So, how do we go about testing these properties?

FsCheck to the rescue

FsCheck is an F# port of the Haskell framework QuickCheck. What it’ll let us do is fire a whole bunch of data at our function with the aim of helping us ensure that these properties hold.

Now before we get onto FsCheck itself, let’s look at how we might test this traditionally. Normally using something like xUnit, you’d have to write specific examples such as this:

[<Fact>]
let ``for a non-negative list, returns the same list`` () =
    let list = [1;2;3]
    let result = list = removeNegatives list
    Assert.True result

This does work, but it means you have to manually come up with a few examples of this, a few examples of the other ones. In this particular case the function is simple enough that this would find any bugs but in more complex functions there might be certain numbers that would break it.

So using FsCheck we would instead might define this test like this:

[<Property>]
let ``for a non-negative list, returns the same list`` (list:NonNegativeInt list) =
    let list = List.map (fun (i:NonNegativeInt) -> i.Get) list
    list = removeNegatives list

The Property attribute works just like the Fact in xUnit or the Test in Nunit. There is one major difference however: the function no longer takes unit, it can take any parameter that FsCheck knows how to create (and yes, you can define them yourself). NonNegativeInt is a built in alias for int which is as the name suggests is a int which is not negative.

You may also have noticed that there are no asserts in this test either. That is because as long as your function returns something FsCheck understands (i.e. bool, exceptions, etc), FsCheck will work out whether the test failed or not that way. So in this case, so long as the final line resolves to true, then the test will be considered to have passed.

There is actually one final thing to note here aswell, this test isn’t excuted once. By default FsCheck will try to find 100 parameters that work and run the test that many times. You can change this number by setting the MaxSize like this:

[<Property(MaxSize=1000)>]
let ``my test`` () =
    ...

Anyway, back the tests. If you try and run this test, you’ll see that it fails. Oh noes! What could possibly be wrong?!?! Oh yes, have you noticed that the list is actually reversed during the process, which breaks one of our properties. So lets fix that.

let removeNegatives list =
        let rec removeNegatives' list newList =
            match list with
            | [] -> List.rev newList
            | head :: tail when head < 0 -> removeNegatives' tail newList 
            | head :: tail -> removeNegatives' tail (head :: newList)
        removeNegatives' list []

Ok, so now we have a test covering that second property: if you pass in a non-negative list, you get the same list back again.

So now how about the first property, testing for the removal of elements less than zero. Well first we could write a test that checks that if you pass in a list of negative numbers, it should always return an empty list:

[<Property>]
let ``for a negative list, returns an empty list`` (list:PositiveInt list) =
    let list = List.map (fun (i:PositiveInt) -> -1 * i.Get) list
    [] = removeNegatives list

Ok great and that passes, maybe we should write a test that tests if it actually filters things out. Well that looks something like this:

[<Property>]
    let ``for a mixed list, returns a positive list`` (list:int list) =
        let outputList = List.filter (fun i -> i >= 0) list
        outputList = removeNegatives list

In this case, this test rather points out the uselessness of this function. Maybe that makes it quite a bad example, but I put it here for completeness anyway (after all it does implement the same function in a single line).

Side note

I probably should point out that I have had some problem with using FsCheck, if you’re using NCrunch you might notice the tests don’t get picked up. However, I have found that if you include the following code in your module it works:

[<Fact>]
let ``ignore this test`` () =
    ()

This (for some reason) causes NCrunch to see all the other tests too.

Updated:

Comments