World's buggiest web server?

This being Australia, ants built a nest in my web server.  This might explain the random system resets. Ants must conduct electricity!

Toshiba Portege M400 Debian Linux Squeeze 6.0.3 Install Note #1 - Touchscreen

The M400 tablet PC has a Wacom touch screen electronics. This appears as a serial device. Because of this, there is no need to install wacom kernel drivers! This means there's nothing found by any of dmesg , lshw nor discover !

However Squeeze does not recognize the touch screen of the M400 tablet PC. So the command

xinput list

does not show the device.

This down to an upstream bug as follows https://bugs.launchpad.net/ubuntu/+source/xf86-input-wacom/+bug/522318

This fix is to apply the above patch to the file /lib/udev/rules.d/69-xserver-xorg-input-wacom.rules. Then reboot.

Debug aids:

You can test the device by running

cat /dev/ttyS? | od

then move the pen around. On my machine I found that running the command

cat /dev/ttyS0 | od

produced output when my pen was moved. You may have to try all the /dev/ttyS* devices.

When patched and working, you will see this:

$ xinput list

⎡ Virtual core pointer id=2 [master pointer (3)]

⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]

⎜ ↳ Microsoft Microsoft Basic Optical Mouse v2.0 id=9 [slave pointer (2)]

⎜ ↳ PS/2 Mouse id=11 [slave pointer (2)]

⎜ ↳ AlpsPS/2 ALPS GlidePoint id=12 [slave pointer (2)]

⎜ ↳ Serial Wacom Tablet eraser id=13 [slave pointer (2)]

⎜ ↳ Serial Wacom Tablet id=14 [slave pointer (2)]

And hwinfo reports:

# hwinfo

10: udi = '/org/freedesktop/Hal/devices/pnp_WACf004_serial_platform_0'
linux.device_file = '/dev/ttyS0'
linux.hotplug_type = 2 (0x2)
linux.subsystem = 'tty'
info.capabilities = { 'serial' }
info.category = 'serial'
info.subsystem = 'tty'
info.product = 'Wacom Serial Tablet PC Pen Tablet/Digitizer'
linux.sysfs_path = '/sys/devices/pnp0/00:0c/tty/ttyS0'
info.parent = '/org/freedesktop/Hal/devices/pnp_WACf004'
info.udi = '/org/freedesktop/Hal/devices/pnp_WACf004_serial_platform_0'
serial.originating_device = '/org/freedesktop/Hal/devices/pnp_WACf004'
serial.device = '/dev/ttyS0'
serial.port = 0 (0x0)
serial.type = 'platform'

Top Gear is Dr Who

I think Top Gear and Dr. Who are the same show. Look at the similarities:

• The Stig = The Doctor, who dies and regenerates in slightly different forms and is played by different actors. First black, then white. What colour Stig next? The Stig is not human and can drive any time machine (er, car) perfectly. Ever notice how he is always discussed formally as "The Doctor" or "The Stig", never some other nomenclature?
  

• The Car = The TARDIS, much abused but moves people and them some. http://www.youtube.com/watch?v=N77GAsv8X-Q

• Jerremy Clarkson, James May and Richard Hammond are The Assistants, each echoing a portion of the Doctor's personality.

• The Stopwatch = The Sonic Screwdriver, part of the cast almost.
  

• The Producers = Terry Nation et. al. who put the characters in strange challenges each episode.

• The Guests = Guest stars who drop into the series to add to the ratings.

• The company? The BBC.

Can you spot more equivalences?

The Configuration Management Manifesto

We follow these principles:

We use source control tools to record changes to our software and configuration during development so that our source code and source data is uniquely identifiable.

Our deployable software is created by repeatable and automated builds.

We identify every item of source code and data used to create our deployable artefacts.

We provide a cryptographic checksum of all files we release.

We generate deployable software that is uniquely identified and self-identifying. No two artefacts we generate have the same identification.

We create software products whose environment configuration is externalised hence absent from the deployable software artefacts.

We provide a baseline of the unique identifiers for all parts of a delivery to our customers including software releases, test or production environments, hardware, data or any other delivery.

We record the all changes we make to our test and production environments, including software deployments, configuration changes, and hardware changes.

We only test software artefacts that are uniquely identifiable.

We only test software in environments whose hardware. and software components configurations are fully documented.

Faster WebSphere wsadmin Jython scripting with imports

When using wsadmin to run Jython scripts I'm seeing a 30 sec startup time, which is too long for my edit-run-test cycle. Here's a way of avoiding the startup delay.

This method uses a simple boot script to launch a read-execute-print-loop (REPL) which you use for your own work. wsadmin does not set up the modules it's imported properly. This you need to fix in your top-level script. Here's the invocation:

    ${path-to-wsadmin}/wsadmin.sh -lang jython -f ./repl.py

Here's repl.py

import sys
sys.modules['AdminConfig'] = AdminConfig

import code
repl = code.InteractiveConsole()
repl.interact("jython %s console" % (sys.version))

When you run this, you get a prompt:

jython 2.1 console
>>>

Your scripts need to import the relevant Admin module. e.g. 'foo.py' starts with

import AdminConfig
class Foo:
 ...

So when you the script with execfile, it picks up the wsadmin modules:

>>> execfile('foo.py')

or

>>> import foo
>>> reload foo

Now you can edit and run 'foo.py' without relaunching wsadmin.

Announcement: Genyris Language

An implementation of this new language inspired by the Semantic Web is now available. Genyris is a Scheme derivative language with classes. An object can belong to multiple classes at once. (This is not multiple inheritance - each object has not just one class but a set of them.) In summary:

• Anything can be "tagged" as a member of a class (including cons cells, atoms). Once tagged, programmers can use OO-style methods, inheritance etc to play with the objects.
• Fully OO
• Objects may be members of multiple classes
• Classes are assigned to objects after construction
• Indentation syntax reduces parentheses.

A tutorial and Java binary executable are available for download.

Rest In Peace: RefLisp, Lispin

After decades of hacking my C++ Lisp interpreter I just got sick of the state of the code. I don't like most of the code any more. I had some advice from Jon Eaves (http://www.eaves.org), who said "Are you a typist or a programmer?". Hmmm. The value of the work is actually in my head, not the code. His advice - walk away - rewrite it since you will write it much better the next time.

Soooo I have started rewriting it in Java. After three days in, I have got it to this state:

JLispin > ((lambda (x) x) (cons (quote Hello) "World"))
(Hello . "World")

Woohooo !

I'm using SICP as a guide for implementation.

Why Java, I hear you ask? Lots of reasons. The other main candidate was Mono/C#. Java has Eclipse whereas Monodevelop is not more than a text editor. (No debugger). I'm dependant on the Eclipse refactoring tools nowadays. Only Vi$ual $tudio has refactoring tools for C# ;-(. Mono will also compile Java, so Java wins. JLispin will also come in handy in my Java day job.

Let me know if you're interested in dabbling in the project.

Why I demand Agile development.

You can rely on Agile development projects to produce good code and unit tests, regression tests and continuous integration environments. Waterfallers can do this too, but don't expect it.

My last couple of Agile projects have used Thoughtworks developers. We've been using Java to build both J2EE web applications and an fat client. When the projects are finished, we are handed not just a fine application, but also a complete world of fully automated test suites and JUnit or Selinium tests. The code itself is highly refined since by now it has been refactored many times.

Big Deal, I hear you say, what does this have to do with architecture? Well as an architect this suite of code and tests means a lot to me. I don't need to design every foreseeable integration point up front. Because the code is well-made I can be sure that adding integration points in future will be feasible and safe.

It's feasible because of the large number of unit tests has forced the devs to use dependency injection, so adding interfaces doesn't engender a re-write. It's safe because when we refactor to add (say) a web service, I know the automated regression tests will pick up any breakages. So changing is less painful in after-project maintenance. Not surprising when you understand that all coding is maintenance in an Agile process.

So when I consider change cases for the system design I can be very relaxed about future changes. This is very valuable to me and my customer. It reduces my need to create a Big Design Up Front. The less you do up front, the better.

Deferring detailed design decisions allows you to use the lessons learned during prototyping and software development. In engineering terms its a way of reducing delay in the feedback loop from posited design to actual experience.

As each iteration unfolds we create production-ready software. We run this an make sure it works in a production-like environment. So any architectural nasties are found early. On one memorable project I deployed the software into the production environment well before the system was to go live. The system performance was abysmal due to WAN latency. We were able to make fixes in the next iteration without breaking stride. If it had been a waterfall project we would have never seen the issue until the end.

Your Agile development team can give you early warning if you have architectural issues.

All good reasons architects should prefer Agile.

Automatic Unit Test Case Generation

We have millions of lines of source code in all sorts of langauges. A lot of it was written by waterfall projects who did not use Test-Driven Development. In other words we don't have unit tests for much of it. But we do want to move to Agile methods. The problem is that without unit tests refactoring becomes very, very scary for most IT people. So we appear to be snookered.

There are many Unit Test Case Generators which seem to require a human to fill in the core. Agitar have some very sexy technology which can do some of this. However, how can it know what a correct outcome is?

One thing we do have, (even in waterfall projects) is suites of System Tests or UATs (User Acceptance Tests). Sometimes these are manual, sometimes they are automatic. Either way they test the entire system, not just a small component. The other thing we have is working code.

So why can't we use the correctness of the code and the system tests to automatically derive component unit tests?

What is needed is a software tool that will take the source or bytecodes of our applications and instrument them. When we run our system tests, it watches the behaviour of the components and records inputs and outputs. It would use the logged results to generate unit tests. We take the unit tests and add them to out build process. Next time a developer changes a component it will detect if the change will break the observed correct behaviour.

This tool would also solve another hard problem. Much of our software relies on large 'frameworks' and is called up and handed a large environment. The problem is that to re-create the environment outside the framework (in a unit test) is hard. A tool which records system test behaviour would also be able to record the state of the environment before and after the test. It would also record the components' interaction (side effects) with the environment.

This does not sound easy. But such a tool would massively reduce our testing costs because we would re-test small components without having to do a full end-to-end system test.

Caveat: I have not carefully researched solutions to this problem so there already may be something available.

Here's an simple Lispin version which re-writes the multiplication function and logs all calls as test cases.

defun system-tests (n)
   ; test multiplication n times
   * (random) (random)
   if (not (equal n 0))
      system-tests (- n 1)

defun test-apply (func args expected-return)
   ; unit test harness...
   cond
      (equal (apply func args) expected-return)
         format t "OK~%"
      else
         format t "FAILED ~S ~S ~S~%" func args expected-return


df test-o-matic (&rest fn)
 ; replace function with instrumented version that logs correct behaviour
   setq fn (car fn)           ; select first parameter
   put fn 'olddef (eval fn)   ; save old definition
   set fn 
      subst fn 'fn
         quote
            lambdaq (&rest *x)
               setq *x (mapc *x eval)
               format t "(test-apply #'~S '" 'fn
               prin1 *x
               setq *x (apply (get 'fn 'olddef) *x)
               format t " '~S)~%" *x
               the *x
   fn

test-o-matic *

system-tests 3

; generates: 
;(test-apply #'* '(25972 868) '22543696)
;(test-apply #'* '(22980 2223) '51084540)
;(test-apply #'* '(22549 23592) '531976008)

Doh!

Out in the real world of business you will find that most people are not rocket scientists. Most of us are fighting a constant battle with idiocy, our own and our collegues. Myself, I am a signed up Induhvidual. On a good day I achieve quite good competency in my work, but even on those days I might spill coffee on my shirt.

There is an ebb and flow of competency vs idiocy in any large organisation. Over the years the result is not perfect. To quote Alain De Botton: "What is declared obvious and 'natural' rarely is so. Recognition of this should teach us to think that the world is more flexible than it seems, for the established views have frequently emerged not through faultless reasoning but through centuries of intellectual muddle. There may be no good reason for things to be the way they are."

So if you have just finished University don't expect things to work properly in the real world, or even rationally. Some business opportunities are simply about helping people do things better. Look around, you'll be amazed at what you find.

:wq

DOH! DOLOR, DOLLAR

This blog is related to Paul Graham's essay Why Smart people Have Bad Ideas. Essentially the academic life does not expose young smart hackers to the world of business so their business ideas are often unlikely to fulfill a need.

This blog describes some of the recurring real-world problems I have personally experienced or have been told about. Sometimes I will make suggestions about how to solve these problems. Maybe some of these postings might produce a viable business model. Maybe not. It's up to you.