Automatic Unit Test Case Generation

We have millions of lines of source code in all sorts of langauges. A lot of it was written by waterfall projects who did not use Test-Driven Development. In other words we don't have unit tests for much of it. But we do want to move to Agile methods. The problem is that without unit tests refactoring becomes very, very scary for most IT people. So we appear to be snookered.

There are many Unit Test Case Generators which seem to require a human to fill in the core. Agitar have some very sexy technology which can do some of this. However, how can it know what a correct outcome is?

One thing we do have, (even in waterfall projects) is suites of System Tests or UATs (User Acceptance Tests). Sometimes these are manual, sometimes they are automatic. Either way they test the entire system, not just a small component. The other thing we have is working code.

So why can't we use the correctness of the code and the system tests to automatically derive component unit tests?

What is needed is a software tool that will take the source or bytecodes of our applications and instrument them. When we run our system tests, it watches the behaviour of the components and records inputs and outputs. It would use the logged results to generate unit tests. We take the unit tests and add them to out build process. Next time a developer changes a component it will detect if the change will break the observed correct behaviour.

This tool would also solve another hard problem. Much of our software relies on large 'frameworks' and is called up and handed a large environment. The problem is that to re-create the environment outside the framework (in a unit test) is hard. A tool which records system test behaviour would also be able to record the state of the environment before and after the test. It would also record the components' interaction (side effects) with the environment.

This does not sound easy. But such a tool would massively reduce our testing costs because we would re-test small components without having to do a full end-to-end system test.

Caveat: I have not carefully researched solutions to this problem so there already may be something available.

Here's an simple Lispin version which re-writes the multiplication function and logs all calls as test cases.

defun system-tests (n)
   ; test multiplication n times
   * (random) (random)
   if (not (equal n 0))
      system-tests (- n 1)

defun test-apply (func args expected-return)
   ; unit test harness...
   cond
      (equal (apply func args) expected-return)
         format t "OK~%"
      else
         format t "FAILED ~S ~S ~S~%" func args expected-return


df test-o-matic (&rest fn)
 ; replace function with instrumented version that logs correct behaviour
   setq fn (car fn)           ; select first parameter
   put fn 'olddef (eval fn)   ; save old definition
   set fn 
      subst fn 'fn
         quote
            lambdaq (&rest *x)
               setq *x (mapc *x eval)
               format t "(test-apply #'~S '" 'fn
               prin1 *x
               setq *x (apply (get 'fn 'olddef) *x)
               format t " '~S)~%" *x
               the *x
   fn

test-o-matic *

system-tests 3

; generates: 
;(test-apply #'* '(25972 868) '22543696)
;(test-apply #'* '(22980 2223) '51084540)
;(test-apply #'* '(22549 23592) '531976008)