Charles Chamberlain

Expect tests are awesome

2021-04-11

Testing software is usually composed of the following steps:

There are many DSLs for making assertions like this, but they usually involve someone hard-coding exactly what is expected to happen. Later on, you might intentionally change the behavior of the code you're testing, and end up breaking the test. Then you have to go back to make the assertions conform with your new view of the world.

"Expect tests" automate this process. Instead of manually writing out assertions, you can print the relavant data and "expect" (i.e. assert) that it doesn't change between runs.

Cram Example

Cram is a tool for writing expect tests in bash. Within cram files (which end in .t), indented lines beginning with a '$' are run as bash. Lines that are not indented are comments. Let's use it to make sure ls is working correctly:

This is the entire cram.t file.
Unindented lines are comments.

Let's first make some files by running 'touch':

  $ touch a b

and then try to list them

  $ ls

'find' should work too:

  $ find .

Notice there are no assertions here yet — cram will add them for us. Running cram test.t writes out a test.t.err file with the output as indented lines below each line of bash. It also print out the diff between our test.t and the new test.t.err:

$ cram test.t
!
--- test.t
+++ test.t.err
@@ -8,7 +8,12 @@
 and then try to list them
 
   $ ls
+  a
+  b
 
 'find' should work too:
 
   $ find .
+  .
+  ./a
+  ./b

# Ran 1 tests, 0 skipped, 1 failed.

If we decide this is indeed the output we want, we can mv test.t.err test.t to "accept" the changes. The file now looks like this:

This is the entire cram.t file.
These unindented lines are comments.

Let's make some files

  $ touch a b

and then try to list them

  $ ls
  a
  b

'find' should work too:

  $ find .
  .
  ./a
  ./b

The new lines after ls and find become the assertions for our test. What happens when we change the inputs and break the test? Let's change touch a b to:

$ touch b c d

and re-run the test:

$ cram test.t
!
--- test.t
+++ test.t.err
@@ -8,12 +8,14 @@
 and then try to list them
 
   $ ls
-  a
   b
+  c
+  d
 
 'find' should work too:
 
   $ find .
   .
-  ./a
+  ./c
+  ./d
   ./b

# Ran 1 tests, 0 skipped, 1 failed.

Output looks good? Again mv test.t.err test.t to accept the test.

Changes in behavior are represented as diffs to expected output. This is great for debugging; instead of an opaque "Assertion failed" you get a picture of how your program changed. And it speeds things up when the new diff is right. No longer do we manually edit assertions!

History

As far as I know, this style of expect tests originated with Mercurial's tests. Later their machinery was extracted into cram. And finally, this sort of testing has been added to OCaml with ppx_expect. Instead of running bash and asserting on the output of whole programs, you can run and test individual OCaml functions! This idea has been integrated into the OCaml build system as a generalized diffing/promoting tool.

Fin

Anyways thanks for reading! Hope you liked hearing about expect tests :)

↵

• introduction
• Cram Example
• History
• Fin