From plumatic’s schema to clojure.spec

In a previous blog post, I showed an example of using plumatic’s schema with test.check and test.chuck. With the introduction of Clojure’s new spec library, I thought it would be interesting to revisit that post and port it from schema to spec. The code from this post is available on github.

Overall, the port was relatively straight-forward, though spec took some getting used to. spec provides similar facilities for what I was using in schema. It integrated with both test.check and test.chuck with no significant modifications!

First, we need to update our `project.clj` to include the latest alpha of Clojure 1.9 (alpha10 as of this writing):

org.clojure/clojure "1.9.0-alpha10"

And in our main file, require spec instead:

(:require [clojure.spec :as s])

The following block of definitions go from this in schema:

(def Rest (s/eq REST-NOTE-NUMBER))
(def Note (s/constrained s/Int #(<= MIN-NOTE % MAX-NOTE)))
(def NoteOrRest (s/either Note Rest))
(s/defrecord Melody [notes :- [NoteOrRest]])

To this in spec:

(s/def ::rest (s/spec #(= % REST-NOTE-NUMBER) :gen #(gen/return REST-NOTE-NUMBER)))
(s/def ::note (s/int-in MIN-NOTE (inc MAX-NOTE)))
(s/def ::note-or-rest (s/or :note ::note :rest ::rest))
(s/def ::notes (s/coll-of ::note-or-rest :kind vector?))
(defrecord Melody [notes])
(s/def ::melody (s/keys :req-un [::notes]))

The first thing you notice is things are slightly more verbose in spec, but there's also more going on. For example, in the case of the `::rest` spec, I need a custom generator which is more restrictive than the one automatically created by spec during generative testing. the `:gen` option on the spec definition allows you to pass in a generating function as part of the definition - making the generator just as re-usable as the spec. This is really powerful.

The `::note` spec is more powerful than it's schema counterpart too. I no longer need a special `NoteGenerator` in my test namespace.

Now, let's compare some of the simple functions. Here are some the schema versions:

(s/defn rest? :- s/Bool [n :- NoteOrRest] (neg? n))
(s/defn note-count :- s/Int [notes :- [NoteOrRest]]
  (count (remove rest? notes)))

And here are the spec versions:

(defn rest? [n] (neg? n))
(s/fdef rest?
        :args (s/cat :n ::note-or-rest)
        :ret boolean?)

(defn note-count [notes] (count (remove rest? notes)))
(s/fdef note-count
        :args (s/cat :notes ::notes)
        :ret integer?
        :fn #(<= (:ret %) (-> % :args :notes count)))

The spec versions look a fair bit more verbose at first glance, compared to their compact-looking schema counterparts. Aside from some indentation, the main difference is that the specification of the functions exists separately of the function. But take a look at the function spec for `note-count`. It actually goes further than the schema version. It encodes an invariant using the arguments and return type of the function. We'll see later on how this is used during testing.

Now let's look at the main function we are testing with test.check/chuck. Here's the schema version:

(s/defn with-new-notes :- Melody
  [melody :- Melody new-notes :- [Note]]
  {:pre [(= (count new-notes) (note-count (:notes melody)))]}
  (let [notes (first (reduce (fn [[updated-notes new-notes] note]
                               (if (rest? note)
                                 [(conj updated-notes note) new-notes]
                                 [(conj updated-notes (first new-notes)) (rest new-notes)]))
                             [[] new-notes] (:notes melody)))]
    (->Melody notes)))

And here's the spec version:

(defn with-new-notes [melody new-notes]
  (let [notes (first (reduce (fn [[updated-notes new-notes] note]
                               (if (rest? note)
                                 [(conj updated-notes note) new-notes]
                                 [(conj updated-notes (first new-notes)) (rest new-notes)]))
                             [[] new-notes] (:notes melody)))]
    (->Melody notes)))

(s/fdef with-new-notes
        :args (s/cat :melody ::melody
                     :new-notes (s/coll-of ::note :kind vector?))
        :ret ::melody
        :fn (s/and #(= (-> % :args :new-notes count) (note-count (-> :ret % :notes)))
                   #(= (-> % :args :new-notes) (remove rest? (-> :ret % :notes)))))

Once again, there's more going on in the spec version. We are able to specify the invariants of the function right in the definition of the function spec. Nice.

Testing

At first I didn't really realize why spec actually wraps test.check. Then I watched this video. The cool part is that the library will not only be able to generate test data based on specs in many cases (like schema does), but it will also use the `:fn` defined in function specs as a property for the tests so that you don't have to write them yourself! So if you define a `:fn` for your `fdef`, it will be used as a property within the test.

Integrating the specs and their respective generators for use with `test.check` or `test.chuck` is straight-forward. In schema we had a few generators we needed to setup our tests:

(def PositiveInt (s/constrained s/Int pos?))
(def RestGenerator (sgen/always -1))
(def NoteGenerator (gen/choose MIN-NOTE MAX-NOTE))

Here are the spec equivalents:

(defn notes-gen [size] (gen/vector (s/gen ::core/note) size))
(defn rests-gen [size] (gen/vector (s/gen ::core/rest) size))

Notice we no longer define a spec analog for `PositiveInt`. We don't need one since Clojure 1.9 includes a predicate in core called pos-int?.

The `notes-and-rests-gen` genator is just as before:

(defn notes-and-rests-gen [size num-notes]
  (gen/bind (notes-gen num-notes) (fn [v]
                                    (let [remaining (- size num-notes)]
                                      (if (zero? remaining)
                                        (gen/return v)
                                        (gen/fmap (fn [rests] (shuffle (into v rests))) (rests-gen remaining)))))))

For the melody generator, we need to pass in our overrides. In schema this looked like this:

(s/defn melody-gen :- Generator
  ([size :- PositiveInt
    num-notes :- PositiveInt]
    (sgen/generator Melody {[NoteOrRest] (notes-and-rests-gen size num-notes)})))

In spec, it looks like this:

(defn melody-gen [size num-notes]
  (s/gen ::core/melody {::core/notes #(notes-and-rests-gen size num-notes)}))

Here we are creating a generator for our melody spec, but with a custom generator for our notes vector. Our actual test.check and test.chuck tests are unchanged!

;;
;; test.check version
;;

(defspec with-new-notes-test-check 1000
         (let [test-gens (tcg/let [num-notes tcg/s-pos-int
                                   melody-num-rests tcg/s-pos-int
                                   total-melody-num-notes (tcg/return (+ num-notes melody-num-rests))
                                   melody (melody-gen total-melody-num-notes num-notes)
                                   new-notes (notes-gen num-notes)]
                                  [melody new-notes])]
           (prop/for-all [[melody new-notes] test-gens]
                         (let [new-melody (with-new-notes melody new-notes)]
                           (and (= (count new-notes) (note-count (:notes new-melody)))
                                (= new-notes (remove rest? (:notes new-melody)))
                                (= (count (:notes melody)) (count (:notes new-melody))))))))
;;
;; test.chuck version
;;

(defspec with-new-notes-test-chuck 1000
         (tcp/for-all [num-notes tcg/s-pos-int
                       melody-num-rests tcg/s-pos-int
                       total-melody-num-notes (gen/return (+ num-notes melody-num-rests))
                       melody (melody-gen total-melody-num-notes num-notes)
                       notes (notes-gen num-notes)]
                      (let [new-melody (with-new-notes melody notes)]
                        (and (= (count notes) (note-count (:notes new-melody)))
                             (= notes (remove rest? (:notes new-melody)))
                             (= (count (:notes melody)) (count (:notes new-melody)))))))

Well, except for one thing actually. During the testing of the code in this post, I actually noticed I was missing an essential property! While I was trying to prove to myself that this still worked, I began intentionally breaking the `with-new-notes` method. I made a change where it just blindly added only the notes, and didn't include the rests, but the tests still passed! It turned out I needed one more critical property for correctness:

(= (count (:notes melody)) (count (:notes new-melody)))

This ensures the the returned melody still has the same number of notes as the original ones passed in, including rest notes.

Finally, I added the spec/check version. Spec includes a `check` function which wraps test.check's `quick-check`, first checking function `:args`, and passing any `:fn` defined along to quick-check as a property. You can also pass in all the options supported by quick-check through a special options map:

(stest/check `with-new-notes {:gen                          {::core/melody     #(melody-gen 5 4)
                                                             ::core/notes-only #(notes-gen 4)}
                              :clojure.spec.test.check/opts {:num-tests 100}})

This is pretty short!

I did get a bit lazy here. Notice I'm hard-coding the sizes for the melody and notes args. The `:gen` overrides map requires that the provided overrides be no-arg functions returning generators. Not exactly sure, but I suspect there is a way to do some kind of `bind/fmap` type of magic to mimic the size generators in the test.check/chuck versions.

Impressions

Compared to spec, schema felt a bit more natural to me at first coming from a more strongly typed background as the schema information is right there next to what it is specifying. But there's an elegant simplicity to the spec approach which I like as well. It's more composable and feels like it fits into the language more naturally, and thus can be leveraged in more ways, without needing to lean too much on the capabilities of macros syntactically.

I love how any predicate can be used as a spec. This makes perfect sense and allows for a more concise way of specifying the same thing without the need for special functions from the library. And of course a lot of existing validation and other code out there can be leveraged immediately in the spec system.

The regex approach to defining sequential specs is perhaps the most impressive aspect of the design to me. This feels really powerful and flexible, allowing arbitrarily complex definitions to be put together using a technique that's already really well understood by programmers.

Of course one of the big wins in spec will be the tooling support afforded by the language itself. Built-in documentation support, etc.

I noticed in 1.9.0 alpha8 a new namespaced map reader syntax was added. Wondering when something equivalent will be defined or somehow supported automatically for records. The `:req-un` and `:opt-un` syntax is awkward.

Clojure namespaced keywords initially tripped me up a bit. I hadn't really used them before. Once you start referencing them in other namespaces you quickly learn a nice little thing that clojure supports. Basically you can alias a namespace as usual and then use the namespace qualifying syntax and it works! I refer `:all` the forms from the core namespace and add an alias so I can refer to the namespaced symbols conveniently, where I make reference to my specs defined in the main source namespace under test.

I've seen some debate about where the best place to put specs are; to me it seems pretty logical to co-locate them directly with the code they spec. This is most analogous to the way typed languages work. Why put them somewhere else? You want them where the code is defined. I suppose one exception might be truly generic specs meant for supporting arbitrary code, say a small library of common specs.

I'm interested to see whether spec can improve Clojure's error messages. I haven't looked at the source for the alpha to see whether any of Clojure itself is spec'd, but I would imagine when (if?) this code is spec'd along with common libraries, that there may be fewer seemingly obscure exceptions happening in code like class cast exceptions and the like.

The experience of porting the code from schema reminds me its important to practice REPL-driven development and build things up so you can really understand where a problem is. In spec's case, use the new tools you get from spec like checking conformance, generating some samples to make sure things work as you expect, etc. as it will make tracking down issues much easier. Not checking your spec's before using them in property-based tests can lead to some strange test failures where it's not immediately obvious what's breaking.

Tracking Alpha Changes

As of this writing, spec is still evolving. The documentation in the guide isn't always keeping pace with the changes. The best way I've found to track it is the announcements in the Clojure google groups forum. There you can find changelogs for each successive alpha.

Happy specing!