In this post, I’ll give a walk-through of property-based testing in clojure with a few great tools: plumatic’s schema, clojure’s own test.check and another great alternate property-based testing library called test.chuck.
First off, schema is a very nice library. If you have not looked at it yet, you should definitely check it out as a lighter-weight alternative to a full-blown typing solution like core.typed. As of version 1.0, schema now includes support for out-of-the-box generators for any of your schemas. Though experimental, the schema generators have worked really nicely for me so far. In this post, we’ll be using the latest version of test.check (0.9.0 as of this writing). It includes a number of nice new features, so you should definitely upgrade your version if you can.
All the code in this post is on github, so you can check out a repository with the complete example. I’ll just be covering the highlights here. First a few project dependencies:
[org.clojure/clojure “1.8.0”] [org.clojure/test.check “0.9.0”] [com.gfredericks/test.chuck “0.2.6”] [prismatic/schema “1.0.4”]
The example I’m going to use is loosely based on a music application i’m currently porting to clojure. We will create a few schemas to define a very simple music domain:
(def Rest (s/eq REST-NOTE-NUMBER)) (def Note (s/constrained s/Int #(<= MIN-NOTE % MAX-NOTE))) (def NoteOrRest (s/either Note Rest)) (s/defrecord Melody [notes :- [NoteOrRest]])
You can ignore the constants as they are just implementation details for the sake of this example. So we have a Melody schema, which is defined as a sequence of Notes or Rests. The idea behind encoding the rests in the melody is that we can replace the actual notes, while preserving the basic rhythm (though the rhythm details are omitted for this example). In fact, we are going to write just such a function. Namely, given a melody and a collection of new notes, return a new melody with all the notes in the original melody replaced with the provided notes. The function will handle preserving the locations of the rests. Hopefully straight-forward enough. Here is the function, called with-new-notes:
(s/defn with-new-notes :- Melody
[melody :- Melody new-notes :- [Note]]
{:pre [(= (count new-notes) (note-count (:notes melody)))]}
(let [notes (first (reduce (fn [[updated-notes new-notes] note]
(if (rest? note)
[(conj updated-notes note) new-notes]
[(conj updated-notes (first new-notes)) (rest new-notes)]))
[[] new-notes] (:notes melody)))]
(->Melody notes)))
A few other helper functions worth noting help with-new-notes do its job. note-count returns the number of notes in a given Melody. rest? does what you would expect. It returns true if the supplied NoteOrRest is a Rest.
Our with-new-notes function also has a pre-condition we must satisfy: the number of notes in the new-notes collection must be equal to the number of notes in the original melody. As you’ll see later on, we’ll have to make sure our property-based test can generate inputs that meet this requirement.
Now let’s define some basic properties that should hold true for this function:
- The number of notes in the new Melody should match the size of the provided
new-notescollection - The extracted notes from the melody should be equal to the provided
new-notescollection
The interesting part of writing a property-based test for this function is that we have a number of interesting constraints here for our generators:
- We need to generate a Melody with the same number of notes as the
new-notesargument in order to satisfy the method’s pre-condition. - To sufficiently exercise the function, we’re going to need to generate melodies which have at least one rest to ensure the logic of preserving the rest positions works correctly.
So how can we generate random inputs for this test that satisfy these constraints?
We’ll need to generate a random size and then have both our Melody generator and new-notes generator seed off that same value. For this we’ll need a generator which takes the output of another generator as its argument. We could use test.check’s fmap and bind for this, but they can be pretty cumbersome to work with and confusing to keep straight in your head for all but the simplest constructions.
Since version 0.9.0, test.check includes a new let macro which behaves just like clojure.core’s let, but works with generators. This makes it much easier to compose the type of correlated generators we need for our test. Here’s the test using test.check:
(defspec with-new-notes-test-check 1000
(let [test-gens (gen/let [num-notes gen/s-pos-int
melody-num-rests gen/s-pos-int
total-melody-num-notes (gen/return (+ num-notes melody-num-rests))
melody (melody-gen total-melody-num-notes num-notes)
notes (notes-gen num-notes)]
[melody notes])]
(prop/for-all [[melody notes] test-gens]
(let [new-melody (with-new-notes melody notes)]
(and (= (count notes) (note-count (:notes new-melody)))
(= notes (remove rest? (:notes new-melody))))))))
Here were running 1000 iterations of our test. We first generate a positive integer for the number of notes, num-notes. We then generate a random number of rests for our melody. We then get the total size of the melody and pass this as argument to our Melody generator function:
(s/defn melody-gen :- Generator
([size :- PositiveInt
num-notes :- PositiveInt]
(sgen/generator Melody {[NoteOrRest] (notes-and-rests-gen size num-notes)})))
Rather than relying on the default Generator that schema would produce for our Melody schema, we are overriding the part which actually constructs the collection of NoteOrRests to meet our constraints about note count. notes-and-rests-gen is a generator function which produces a collection of notes or rests which match our constraints:
(s/defn notes-and-rests-gen :- Generator
[size :- PositiveInt
num-notes :- PositiveInt]
(gen/bind (notes-gen num-notes) (fn [v]
(let [remaining (- size num-notes)]
(if (zero? remaining)
(gen/return v)
(gen/fmap (fn [rests] (shuffle (into v rests))) (rests-gen remaining)))))))
Here we’re just choosing between our note and rest generators based on the difference in total size and the number of notes. One other interesting thing is worth pointing out here. Recall our schema definition for Note:
(def Note (s/constrained s/Int #(<= MIN-NOTE % MAX-NOTE)))
An out-of-the-box schema generator for Note won’t quite work for us here. The problem is the generator produced is based on randomly generating an Int and then running the validation function on it at run-time. This has the unfortunate side-effect of potentially generating values which will not conform to the schema constraint. What we want here is to ensure that all notes generated fall within the value range. For this we override our NoteGenerator as follows:
(def NoteGenerator (gen/choose MIN-NOTE MAX-NOTE))
This ensures that we always produce a legal value conforming to our schema and avoid the dreaded clojure.lang.ExceptionInfo: Couldn't satisfy such-that predicate after 10 tries. which occurs when test.check cannot generate a constraint-satisfying input within 10 random tries.
OK. Back to our test.check test again:
(defspec with-new-notes-test-check 1000
(let [test-gens (gen/let [num-notes gen/s-pos-int
melody-num-rests gen/s-pos-int
total-melody-num-notes (gen/return (+ num-notes melody-num-rests))
melody (melody-gen total-melody-num-notes num-notes)
notes (notes-gen num-notes)]
[melody notes])]
(prop/for-all [[melody notes] test-gens]
(let [new-melody (with-new-notes melody notes)]
(and (= (count notes) (note-count (:notes new-melody)))
(= notes (remove rest? (:notes new-melody))))))))
While this works, it leaves a little to be desired. Were forced to declare our generators upfront and then pass them to the for-all function. Not bad, but not quite as clean as I would like.
Enter test.chuck. test.chuck is a fantastic little library meant to augment clojure’s test.check. It also has some other nifty features including generating strings based on a regular expression (I haven’t used this feature yet, but I’m sure I will soon). It’s also worth mentioning that the creator of the library is also the maintainer of test.check, so the library is of high quality. In any case, test.chuck gives us the function we are looking for:
(defspec with-new-notes-test-chuck 1000
(tcp/for-all [num-notes gen/s-pos-int
melody-num-rests gen/s-pos-int
total-melody-num-notes (gen/return (+ num-notes melody-num-rests))
melody (melody-gen total-melody-num-notes num-notes)
notes (notes-gen num-notes)]
(let [new-melody (with-new-notes melody notes)]
(and (= (count notes) (note-count (:notes new-melody)))
(= notes (remove rest? (:notes new-melody)))))))
And that’s it. Nice and clean.
Conclusion
I’m having a lot of fun with property-based testing. It takes a bit of getting used to, but it is really powerful and provides a form of test rigor that used to take a lot more test harness code to accomplish. Being able to rely on schema’s generators is a fantastic productivity booster for complex types. It’s also flexible enough to allow you to override points in the generation hierarchy to plug in more robust constraint satisfying generation properties in order to make your tests predictable and repeatable. With the newest version of test.check or usage of test.chuck, we get the power to easily compose generators into ever more complex generators to help seed things properly in more complex tests. The example covered in this post is really just meant to illustrate this type of usage.
One parting thought. I’ve been thinking lately if you carry this form of testing to its logical conclusion, you end up at simulation testing. This feels like it might be the analog on the integration testing side. You essentially push your code through a (presumably accelerated and random) sequence of steps to exercise possibly unexpected data and scenarios you might see in production. At some point, I think i’ll have a look at simulant.