Faster WebSphere wsadmin Jython scripting with imports

When using wsadmin to run Jython scripts I'm seeing a 30 sec startup time, which is too long for my edit-run-test cycle. Here's a way of avoiding the startup delay.

This method uses a simple boot script to launch a read-execute-print-loop (REPL) which you use for your own work. wsadmin does not set up the modules it's imported properly. This you need to fix in your top-level script. Here's the invocation:

    ${path-to-wsadmin}/wsadmin.sh -lang jython -f ./repl.py

Here's repl.py

import sys
sys.modules['AdminConfig'] = AdminConfig

import code
repl = code.InteractiveConsole()
repl.interact("jython %s console" % (sys.version))

When you run this, you get a prompt:

jython 2.1 console
>>>

Your scripts need to import the relevant Admin module. e.g. 'foo.py' starts with

import AdminConfig
class Foo:
 ...

So when you the script with execfile, it picks up the wsadmin modules:

>>> execfile('foo.py')

or

>>> import foo
>>> reload foo

Now you can edit and run 'foo.py' without relaunching wsadmin.

Announcement: Genyris Language

An implementation of this new language inspired by the Semantic Web is now available. Genyris is a Scheme derivative language with classes. An object can belong to multiple classes at once. (This is not multiple inheritance - each object has not just one class but a set of them.) In summary:

• Anything can be "tagged" as a member of a class (including cons cells, atoms). Once tagged, programmers can use OO-style methods, inheritance etc to play with the objects.
• Fully OO
• Objects may be members of multiple classes
• Classes are assigned to objects after construction
• Indentation syntax reduces parentheses.

A tutorial and Java binary executable are available for download.

Rest In Peace: RefLisp, Lispin

After decades of hacking my C++ Lisp interpreter I just got sick of the state of the code. I don't like most of the code any more. I had some advice from Jon Eaves (http://www.eaves.org), who said "Are you a typist or a programmer?". Hmmm. The value of the work is actually in my head, not the code. His advice - walk away - rewrite it since you will write it much better the next time.

Soooo I have started rewriting it in Java. After three days in, I have got it to this state:

JLispin > ((lambda (x) x) (cons (quote Hello) "World"))
(Hello . "World")

Woohooo !

I'm using SICP as a guide for implementation.

Why Java, I hear you ask? Lots of reasons. The other main candidate was Mono/C#. Java has Eclipse whereas Monodevelop is not more than a text editor. (No debugger). I'm dependant on the Eclipse refactoring tools nowadays. Only Vi$ual $tudio has refactoring tools for C# ;-(. Mono will also compile Java, so Java wins. JLispin will also come in handy in my Java day job.

Let me know if you're interested in dabbling in the project.

Why I demand Agile development.

You can rely on Agile development projects to produce good code and unit tests, regression tests and continuous integration environments. Waterfallers can do this too, but don't expect it.

My last couple of Agile projects have used Thoughtworks developers. We've been using Java to build both J2EE web applications and an fat client. When the projects are finished, we are handed not just a fine application, but also a complete world of fully automated test suites and JUnit or Selinium tests. The code itself is highly refined since by now it has been refactored many times.

Big Deal, I hear you say, what does this have to do with architecture? Well as an architect this suite of code and tests means a lot to me. I don't need to design every foreseeable integration point up front. Because the code is well-made I can be sure that adding integration points in future will be feasible and safe.

It's feasible because of the large number of unit tests has forced the devs to use dependency injection, so adding interfaces doesn't engender a re-write. It's safe because when we refactor to add (say) a web service, I know the automated regression tests will pick up any breakages. So changing is less painful in after-project maintenance. Not surprising when you understand that all coding is maintenance in an Agile process.

So when I consider change cases for the system design I can be very relaxed about future changes. This is very valuable to me and my customer. It reduces my need to create a Big Design Up Front. The less you do up front, the better.

Deferring detailed design decisions allows you to use the lessons learned during prototyping and software development. In engineering terms its a way of reducing delay in the feedback loop from posited design to actual experience.

As each iteration unfolds we create production-ready software. We run this an make sure it works in a production-like environment. So any architectural nasties are found early. On one memorable project I deployed the software into the production environment well before the system was to go live. The system performance was abysmal due to WAN latency. We were able to make fixes in the next iteration without breaking stride. If it had been a waterfall project we would have never seen the issue until the end.

Your Agile development team can give you early warning if you have architectural issues.

All good reasons architects should prefer Agile.

Automatic Unit Test Case Generation

We have millions of lines of source code in all sorts of langauges. A lot of it was written by waterfall projects who did not use Test-Driven Development. In other words we don't have unit tests for much of it. But we do want to move to Agile methods. The problem is that without unit tests refactoring becomes very, very scary for most IT people. So we appear to be snookered.

There are many Unit Test Case Generators which seem to require a human to fill in the core. Agitar have some very sexy technology which can do some of this. However, how can it know what a correct outcome is?

One thing we do have, (even in waterfall projects) is suites of System Tests or UATs (User Acceptance Tests). Sometimes these are manual, sometimes they are automatic. Either way they test the entire system, not just a small component. The other thing we have is working code.

So why can't we use the correctness of the code and the system tests to automatically derive component unit tests?

What is needed is a software tool that will take the source or bytecodes of our applications and instrument them. When we run our system tests, it watches the behaviour of the components and records inputs and outputs. It would use the logged results to generate unit tests. We take the unit tests and add them to out build process. Next time a developer changes a component it will detect if the change will break the observed correct behaviour.

This tool would also solve another hard problem. Much of our software relies on large 'frameworks' and is called up and handed a large environment. The problem is that to re-create the environment outside the framework (in a unit test) is hard. A tool which records system test behaviour would also be able to record the state of the environment before and after the test. It would also record the components' interaction (side effects) with the environment.

This does not sound easy. But such a tool would massively reduce our testing costs because we would re-test small components without having to do a full end-to-end system test.

Caveat: I have not carefully researched solutions to this problem so there already may be something available.

Here's an simple Lispin version which re-writes the multiplication function and logs all calls as test cases.

defun system-tests (n)
   ; test multiplication n times
   * (random) (random)
   if (not (equal n 0))
      system-tests (- n 1)

defun test-apply (func args expected-return)
   ; unit test harness...
   cond
      (equal (apply func args) expected-return)
         format t "OK~%"
      else
         format t "FAILED ~S ~S ~S~%" func args expected-return


df test-o-matic (&rest fn)
 ; replace function with instrumented version that logs correct behaviour
   setq fn (car fn)           ; select first parameter
   put fn 'olddef (eval fn)   ; save old definition
   set fn 
      subst fn 'fn
         quote
            lambdaq (&rest *x)
               setq *x (mapc *x eval)
               format t "(test-apply #'~S '" 'fn
               prin1 *x
               setq *x (apply (get 'fn 'olddef) *x)
               format t " '~S)~%" *x
               the *x
   fn

test-o-matic *

system-tests 3

; generates: 
;(test-apply #'* '(25972 868) '22543696)
;(test-apply #'* '(22980 2223) '51084540)
;(test-apply #'* '(22549 23592) '531976008)

Dolor

We're all hurting, but every business has a unique threshold of pain. Some business can continue doing things in expensive ways for years, hemorrhaging money on useless activity or parasites. Most cannot. What matters is whether the business can afford the costs and whether the spend required exceeds the pain itself. This a
business case.

If you have smart idea for a new business, guess what, you actually need two business cases. One for yourselves (are you going to make a profit?) and one for your customer (is the cost of your product less than their current pain?).

Cost is probably the most obvious way businesses feel pain, but there are other hurts. Customer lossage, sloth, staff turnover, low reputation, legal assault, competitors, poor quality, it's all there. For each kind of pain there are armies of MBAs and consultants all willing to provide a fix, and often they can.

The cure for many business ills are simply not derived from technology. Looking for an IT solution to a business problem before considering business solutions first just ain't right. A good IT product would attack something that has no common-sense business solution. Many IT 'solutions' aren't. Which creates another kind of business pain: failed IT projects. Use your powers for good, not evil.

Doh!

Out in the real world of business you will find that most people are not rocket scientists. Most of us are fighting a constant battle with idiocy, our own and our collegues. Myself, I am a signed up Induhvidual. On a good day I achieve quite good competency in my work, but even on those days I might spill coffee on my shirt.

There is an ebb and flow of competency vs idiocy in any large organisation. Over the years the result is not perfect. To quote Alain De Botton: "What is declared obvious and 'natural' rarely is so. Recognition of this should teach us to think that the world is more flexible than it seems, for the established views have frequently emerged not through faultless reasoning but through centuries of intellectual muddle. There may be no good reason for things to be the way they are."

So if you have just finished University don't expect things to work properly in the real world, or even rationally. Some business opportunities are simply about helping people do things better. Look around, you'll be amazed at what you find.

:wq