Automatic Unit Test Case Generation

We have millions of lines of source code in all sorts of langauges. A lot of it was written by waterfall projects who did not use Test-Driven Development. In other words we don't have unit tests for much of it. But we do want to move to Agile methods. The problem is that without unit tests refactoring becomes very, very scary for most IT people. So we appear to be snookered.

There are many Unit Test Case Generators which seem to require a human to fill in the core. Agitar have some very sexy technology which can do some of this. However, how can it know what a correct outcome is?

One thing we do have, (even in waterfall projects) is suites of System Tests or UATs (User Acceptance Tests). Sometimes these are manual, sometimes they are automatic. Either way they test the entire system, not just a small component. The other thing we have is working code.

So why can't we use the correctness of the code and the system tests to automatically derive component unit tests?

What is needed is a software tool that will take the source or bytecodes of our applications and instrument them. When we run our system tests, it watches the behaviour of the components and records inputs and outputs. It would use the logged results to generate unit tests. We take the unit tests and add them to out build process. Next time a developer changes a component it will detect if the change will break the observed correct behaviour.

This tool would also solve another hard problem. Much of our software relies on large 'frameworks' and is called up and handed a large environment. The problem is that to re-create the environment outside the framework (in a unit test) is hard. A tool which records system test behaviour would also be able to record the state of the environment before and after the test. It would also record the components' interaction (side effects) with the environment.

This does not sound easy. But such a tool would massively reduce our testing costs because we would re-test small components without having to do a full end-to-end system test.

Caveat: I have not carefully researched solutions to this problem so there already may be something available.

Here's an simple Lispin version which re-writes the multiplication function and logs all calls as test cases.

defun system-tests (n)
   ; test multiplication n times
   * (random) (random)
   if (not (equal n 0))
      system-tests (- n 1)

defun test-apply (func args expected-return)
   ; unit test harness...
   cond
      (equal (apply func args) expected-return)
         format t "OK~%"
      else
         format t "FAILED ~S ~S ~S~%" func args expected-return


df test-o-matic (&rest fn)
 ; replace function with instrumented version that logs correct behaviour
   setq fn (car fn)           ; select first parameter
   put fn 'olddef (eval fn)   ; save old definition
   set fn 
      subst fn 'fn
         quote
            lambdaq (&rest *x)
               setq *x (mapc *x eval)
               format t "(test-apply #'~S '" 'fn
               prin1 *x
               setq *x (apply (get 'fn 'olddef) *x)
               format t " '~S)~%" *x
               the *x
   fn

test-o-matic *

system-tests 3

; generates: 
;(test-apply #'* '(25972 868) '22543696)
;(test-apply #'* '(22980 2223) '51084540)
;(test-apply #'* '(22549 23592) '531976008)

Doh!

Out in the real world of business you will find that most people are not rocket scientists. Most of us are fighting a constant battle with idiocy, our own and our collegues. Myself, I am a signed up Induhvidual. On a good day I achieve quite good competency in my work, but even on those days I might spill coffee on my shirt.

There is an ebb and flow of competency vs idiocy in any large organisation. Over the years the result is not perfect. To quote Alain De Botton: "What is declared obvious and 'natural' rarely is so. Recognition of this should teach us to think that the world is more flexible than it seems, for the established views have frequently emerged not through faultless reasoning but through centuries of intellectual muddle. There may be no good reason for things to be the way they are."

So if you have just finished University don't expect things to work properly in the real world, or even rationally. Some business opportunities are simply about helping people do things better. Look around, you'll be amazed at what you find.

:wq

DOH! DOLOR, DOLLAR

This blog is related to Paul Graham's essay Why Smart people Have Bad Ideas. Essentially the academic life does not expose young smart hackers to the world of business so their business ideas are often unlikely to fulfill a need.

This blog describes some of the recurring real-world problems I have personally experienced or have been told about. Sometimes I will make suggestions about how to solve these problems. Maybe some of these postings might produce a viable business model. Maybe not. It's up to you.