Pither.com

Every year or so I am reminded that yet another year has gone by without the HMRC and Companies House fixing their awful electronic CT600 corporation tax return. If you're a company you don't really get a lot of choice about using this form once a year and these days it pretty much has to be done electronically.

I'm going to mostly overlook the fact that this form is awfully designed, forces you to type in blocks of standard text and insists on you entering and re-entering identical information several times throughout the form. Even the use of the horrendous Adobe Acrobat forms technology could be forgiveable, if only it actually worked!

However, it sadly does not. There is (and has been every year that I've been involved with submitting this form) a problem with Adobe's Acrobat reader accepting the SSL certificates of the Companies House submission site.

This year, I thought I'd actually write up the process of fixing this. Let's start with the error message:

SSL Error!!! Please install the CA Certificate(s) for SSL
Communication if certificate resides on local disk, try
"acroread -installCertificate [-PEM|-DER] [pathname]" on
the command line. if certificate resides on the server try
"acroread -installCertificate xmlgw.companieshouse.gov.uk 443"
on command line.

It helpfully suggests a solution right there in the error message. Further more if you open a terminal and run the second command that is suggested, it will appear to do the right thing. The only trouble is, that it doesn't! What it will actually do is install just the first certificate in the chain and the full verification of the chain will still fail.

So, whistle stop tour of how to actually fix it (using the command line obviously, none of that nonsense web browser, pointing and clicking stuff that other guides out there already suggest!)...

Check that you can actually verify the certificate (you may need a different CApath, this one is suitable for Ubuntu/Debian):

$ openssl s_client -CApath /etc/ssl/certs/ -connect xmlgw.companieshouse.gov.uk:443
...
    Verify return code: 0 (ok)

Assuming openssl can verify the chain, you have everything you need. First, check how many certificates are involved. Look at the first few lines of output from the command above and you should see a certificate list, something like:

depth=2 C = US, ST = UT, L = Salt Lake City, O = The USERTRUST Network, OU = http://www.usertrust.com, CN = UTN-USERFirst-Hardware
verify return:1
depth=1 C = IE, ST = Dublin, L = Dublin, O = Digi-Sign Limited, OU = Terms and Conditions of use: http://www.digi-sign.com/repository, CN = Digi-Sign CA Digi-SSL Xp
verify return:1
depth=0 C = GB, ST = Wales, L = Cardiff, O = Companies House, OU = IT Infrastructure, OU = Provided by Digi-Sign Limited, OU = Digi-SSL Xp, CN = xmlgw.companieshouse.gov.uk
verify return:1

To collect the certificates we can from s_client, add "-showcerts" to the openssl command (and save a copy of the output):

$ openssl s_client -CApath /etc/ssl/certs/ -showcerts -connect xmlgw.companieshouse.gov.uk:443 | tee certs.txt

Within the output, you should see some certificates, I get two during this example. The important bits are wrapped with begin and end lines like:

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

For each certificate, copy the output file to a new file and then edit it to remove everything except the certificate. Leaving just the begin/end lines and all the characters between them. Make sure each file only ends up with one certificate (and that they aren't the same one!). I called my two files cert1.pem and cert2.pem:

$ cp certs.txt cert1.pem
$ vi cert1.pem
$ cp certs.txt cert2.pem
$ vi cert2.pem

Then tell acroread about them:

acroread -installCertificate -PEM cert1.pem
acroread -installCertificate -PEM cert2.pem

Each time you should check the issuer displayed and confirm the import.

For me there is one certificate left, which is the top level CA that openssl does not list. To find this file, we'll need the issuer hash of the second certificate from above:

$ openssl x509 -issuer_hash -noout -in /tmp/cert2.pem 
b13cc6df

We can then use that hash directly to import the CA:

acroread -installCertificate -PEM /etc/ssl/certs/b13cc6df.0

Now, finally, we can get can to filling in the CT600!

A few weeks ago I ran the East Grinstead 10k. I managed it slightly slower than I was hoping (I blame the non-road surfaces and hills!), but it still wasn't too bad at 57:44.

My official position was 200/335 with my fellow runners John managing an impressive 139th and Chris just behind me in 202nd place. We all looked a bit worn out afterwards though (I was so bad I even ate the free banana they gave us!):

Quite a long time ago I talked about setting envelope sender addresses so that I could send mailshots and get back any bounces in a useful way. The missing component to it was to process those bounces within my application. Here, finally, is my write up of how I did this.

After much searching I eventually found the excellent SubEtha SMTP project which made including an SMTP server within my Grails application incredibly easy. Just four simple steps are required...

Include the library (grails-app/conf/BuildConfig.groovy):

dependencies {
    compile('org.subethamail:subethasmtp:3.1.4') {
        excludes 'mail'
    }
}

Write a SubEtha message listener (for me: src/groovy/com/supajam/smtp/MessageListener.groovy):

package com.supajam.smtp;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.subethamail.smtp.TooMuchDataException;
import org.subethamail.smtp.helper.SimpleMessageListener
import com.supajam.BounceRecord

public class MessageListener implements SimpleMessageListener {
    private final static Logger log = LoggerFactory.getLogger(MessageListener.class);

    public boolean accept(String from, String recipient) {
        return true;
    }

    public void deliver(String from, String recipient, InputStream data) throws TooMuchDataException, IOException {

        // I use this to process bounce messages, but it's just an
        // email message that could be processed however you like

        BounceRecord.withTransaction {
            if (log.isDebugEnabled())
                log.debug("Processing mail from " + from + " to " + recipient);

            // The regex on the next line is specific to my VERP scheme
            def origAddr = recipient =~ /-(.+=.+)@/

            if(origAddr && origAddr[0].size() > 1) {
                // here I extract the source address, ensure it's
                // valid, record the bounce and remove the address
                // from the mailing list.
            }
        }
    }
}

I've included an outline of the code that I actually use to process incoming email - all of which I know will be bounce messages. There's nothing to force this use though, it's just a generic way to receive emails into an application.

Wire it together with Spring (grails-app/conf/spring/resources.groovy):

beans = {
    // Create an in-process SMTP server for processing bounces
    smtpMessageListener(com.supajam.smtp.MessageListener)

    smtpMessageListenerAdapter(org.subethamail.smtp.helper.SimpleMessageListenerAdapter, ref('smtpMessageListener'))

    smtpServer(org.subethamail.smtp.server.SMTPServer, ref('smtpMessageListenerAdapter')) {
        port = 2500
        hostName = "my.server.name"
        disableTLS = true
        maxConnections = 10
    }
}

To avoid running my grails application as root, I set the SMTP port to a high number (2500 in this example). Getting emails delivered to such a port is beyond this post (but personally I use iptables redirects).

Finally, make it run at application start (grails-app/conf/BootStrap.groovy):

class BootStrap {
    def smtpServer

    def init = { servletContext ->

        log.debug("Starting SMTP server")
        try {
            smtpServer.start()
        }
        catch(Exception e) {
            // Starting the web application is more important
            // than SMTP, so just log the failure and carry on
            log.error("SMTP server could not be started")
        }
    }

    def destroy = {
        log.debug("Stopping SMTP server")
        smtpServer.stop()
    }
}

I guard the SMTP server start just in case there's a problem. No SMTP server is a nuisance but no web application is very bad. Just for cleanliness I also attempt to stop it in the destroy method.

That's it! It really is that simple to have a full SMTP server running within grails and ready to accept and process incoming email.

I administer (mostly through SEOSS) quite a few OpenVZ virtual machines and I recently wanted to create a clone of one onto a local system for some testing. Unfortunately I didn't have an OpenVZ machine locally, so decided I'd have a go at migrating the OpenVZ virtual to a local KVM instance. The process I went through is below.

Create a local disk image for the clone.

qemu-img create -f raw clone.raw 10G

Setup a loop back device so the image can have a partition table created.

sudo losetup /dev/loop0 clone.raw
sudo fdisk /dev/loop0

Create swap (1st primary partition, 2G in size and set type to Linux swap - just the fdisk key presses are listed).

n
p
1
<enter>
+2G
t
82

Create root (2nd primary partition, rest of disk - just the fdisk key presses are listed).

n
p
2
<enter>
<enter>

Finally save and quit fdisk.

w

Remove the simple loopback device.

sudo losetup -d /dev/loop0

Create loopback devices for each of the new partitions.

sudo kpartx -av clone.raw

Initialise the swap partition and create a suitable file system on the root.

sudo mkswap /dev/mapper/loop0p1
sudo mke2fs -j /dev/mapper/loop0p2

Mount the clone's root file system locally and rsync the remote source machine into it. In my case I connect to the actual, running VM as there is nothing running that will break my copy. In other cases it may be necessary to shutdown the source VM and rsync the underlying directory from the host machine.

mkdir mnt
sudo mount /dev/mapper/loop0p2 mnt
sudo rsync -avzx --numeric-ids root@your.source.machine:/ ./mnt/

An OpenVZ VM doesn't have terminals but some will be needed for KVM.

sudo vi mnt/etc/inittab
# Re-enable the tty terminals

Having a DHCP client will also make life easier with KVM, so I install a common one.

sudo chroot mnt
apt-get install isc-dhcp-client
exit

Finished with the root file system and the cloned image loopbacks.

sudo umount mnt
sudo kpartx -d clone.raw

For the next bit, you'll need to have a suitable kernel (to run the VM) installed locally (outside the VM).

sudo kvm -m 512 -kernel /boot/vmlinuz-2.6.38-11-generic -initrd /boot/initrd.img-2.6.38-11-generic -append "root=/dev/vda2" -drive file=clone.raw,if=virtio -net nic,model=virtio -net user

For some reason I had to change VT (ie Alt+F2) within the KVM console in order to get a login prompt. Make sure you login as (or become) root for the next bit.

For me the KVM networking interface appeared as eth3 (you can check with "ip a"), which needs to be used in the next commands.

echo "auto eth3" >> /etc/network/interfaces
echo "iface eth3 inet dhcp" >> /etc/network/interfaces

ifup eth3

Install a kernel (choose one suitable for your VM) and a boot loader (probably grub).

apt-get install linux-image-amd64 grub

The normal grub configuration questions will need answering during this. To tidy up you probably want to enable the swap space and create a simple fstab as well.

vi /etc/fstab

cat /etc/fstab 
/dev/vda2 / ext3 rw,relatime 0 0
/dev/vda1 none swap defaults 0 0

swapon -a

shutdown -h now

The machine can now be run without sudo and with a simpler command line

kvm -m 512 -drive file=clone.raw,if=virtio,boot -net nic,model=virtio -net user

That's it! You may want to reconfigure the networking, perhaps import the VM into libvirt/virt-manager or perform other configuration specific to your setup. For me, I just needed a local clone of the machine for testing so I stuck with the existing config and just ran it from the command line when needed.

Much earlier this year Liz and the children planted a few daffodil bulbs along our fence. Not quite so long ago they came out rather well, and I took some photos. My favourite three daffodil close-ups are below (click on each for a larger version).

Please see Part 1 for the start of this series and links to all parts.

For the SupaJam ticket outlet, we didn't need anything too fancy, just enough to present people with a couple of ticket types to choose from and a way to pay for them. Of course we still had the extra requirement of collecting a real name for each and every ticket as well though.

We kept the structure online pretty simple, sticking to a single web page, either re-displaying with errors or sending the user straight to PayPal (our chosen payment processor purely for reasons of technical experience).

So far this was all fairly basic coding, a command object, controller, a couple of views and a few new domain objects to keep it all. This is where Grails really shines, making all of this simple and quick.

There is already a PayPal plugin for grails, but in this case I didn't feel it fitted, so the PayPal integration was custom coded. This included building and passing a cart, based on user selections as well as an IPN endpoint to make sure all the purchases were dealt with automatically.

Once a user has visited one of our ticket pages and decided to take up an offer, been to PayPal and paid, it's generally useful to then provide a ticket. This was the part that presented the first interesting challenge.

We hadn't finalised how we were going to run our festival entrance yet but it seemed like a fast method to check unique ticket codes might be handy. The quick and simple answer to this seemed like a barcode. So each person would need a physical ticket to bring along that we could scan and quickly check validity. We didn't want to add postage costs to what we charged, so email was the simplest solution. As for the most controllable and widely readable document format to send people - PDF of course!

So we needed to make PDFs that would contain some useful instructions to the owner, a few critical ticket details plus the essential unique code and its barcode representation. To make them look nicer and also to better fill an A4 page we also decided to add an image about the festival.

I had read about Jasper Reports before and knew there was a Grails plugin for it, so I decided it would be a good place to start. I soon had iReport downloaded, running and a simple ticket report designed.

While the basics of the ticket looked simple enough, there were two elements that proved slightly tougher. The most critical was the barcode. When adding a barcode in iReport you first get a choice of barcode library. So I researched each and picked the one I preferred.

This worked fine in iReport, but it soon became apparent that the Grails Jasper Reports plugin deliberately excludes most of the optional components and dependencies of Jasper. This included both of the barcode libraries. So off I went trying to add Barcode4J to my grails project.

After an hour or so of fighting to include one layer of dependencies while desperately excluding the next layer (mostly to avoid XML parser conflicts), I admitted defeat and had a crack at the Barbecue barcode library instead. Thankfully this turned out to be trivial in comparison and just needed:

compile('net.sourceforge.barbecue:barbecue:1.5-beta1') {
  excludes 'portlet-api', 'servlet-api', 'junit'
}

The only downside was that I couldn't get the code to be automatically included under the barcode (even in iReport). Just including the code in a text field was a simple work around though.

Next up was the included images - for branding, page filling and information. I wanted to change these for different events and fetching each image from an event configured URL seemed like a good way to go. At least I thought that until I tried it!

For quite some time Jasper Reports stubbornly refused to load the images into the report. Test images from another site would work, but not those from my development grails server.

After much debugging and eventually resorting to tcpdump(!), I tracked this down to the zipped-resources plugin which currently does two bad things (as documented). It doesn't check to see if a client actually supports gzip and it gzip's images! So this meant that Jasper was being given gzipped images which it couldn't deal with.

Once I realised this, simply removing the zipped-resources plugin solved the problem. Although in the end I actually re-instated the zipped-resources plugin and decided to host the images on one of our external (ie not Grails) web servers.

My final little niggle with Jasper was font based. I could not seem to find a font that both Jasper could measure (for alignment purposes) and clients could find when rendering the PDF. I ended up solving this by using the Jasper-fonts package and choosing to embed the fonts within the PDF. This needed another line in my grails-app/conf/BuildConfig.groovy file:

compile('net.sf.jasperreports:jasperreports-fonts:4.0.0')

And that was it.

In less than a (short, due to bank holidays) week I built an e-commerce ticket agent system that created pretty, barcoded e-tickets. This promptly went live on SupaJam and sold several hundred (up to thousands by now) festival tickets.

Towards the end of May, SupaJam launched the first round of several festival ticket give aways planned for this summer. This wasn’t the first time SupaJam has given away festival tickets, but it is the first time that we've done it in this way.

Technically, it all went well. The days afterwards did include a rather large number of support emails though!

In hind sight, this isn't entirely surprising, we do make life a little difficult for ourselves. These give aways are providing a limited number of quite valuable tickets, so to protect ourselves and the festivals we want to make sure that the tickets we give away are not resold. We don't want a ticket tout to end up benefiting. Hence we always collect a real (ID-able) name for each ticket. If that wasn't enough to ask of people claiming these tickets, we gave them away first-come-first-served, so everyone typing in their details was typing in a hurry!

So in flooded the requests to change names, correct spellings, return duplicates that were got for or by friends and assorted other amendments.

It quickly became apparent that giving away tickets in this way had a fairly high support cost associated with it.

So we put our thinking caps on and decided that we should charge a small administration fee to cover some of the support costs that seemed to be inevitable. Of course, this meant we needed some kind of e-commerce setup in order to process these fees.

Part of the deal that we made with the festival for these tickets meant that we had to run our own box office, on site at the festival, to process and verify each of our ticket holders. A process for making that happen, both reliably and quickly was also a challenge we hadn't solved yet.

By the time we'd revised our approach, we had just three weeks until the first festival - the London Feis in Finsbury Park. This story (which I've split into four further parts to ease my writing as much as your reading) is my journey through the technology solutions I developed and built to enable hundreds of people to enjoy a free festival courtesy of SupaJam.

(Each part below will become a link as I write the articles.)

Part 2 - Ticket sales
Part 3 - Customer support and search
Part 4 - Creating and running the technology for a box office
Part 5 - Lessons from becoming a ticket agent and box office in 3 weeks