Richard - it really depends how much extra poke you want, and how much of the work you'll be able to do yourself.
Dave Andrews site is THE best resource for K series tuning available so an absolute must see site if you're considering upgrades. Also tune into the mailing list on se7ens.net (you can just read the archives to cut down on mail traffic). Lots of upgraditis exhibited on there.
BUT, if you have to get much of the work done for you and aren't starting with a Superlight type car then you may be better off (financially and spec wise) trading the car in against a used SLR.
Then again, finding one of the exact spec, colour etc. may prove difficult.
I did the upgrade thing last year so if you want more specifics, drop me a mail and I can let you have prices etc. for pretty much everything you may (or may not!) want. Plus some dos and don'ts!
Graham's idea may also be sensible, but I'm not sure that the Exige engine uses the "normal" VHPD ECU or induction setup (I have some recollection that they're specific to the Exige) so it might be a bit of a headache getting it up and running in a Caterham.
Sid - I've seen used Supersports kits advertised for about 500 quid in the past - not sure if they sold at this level though as there isn't always a market for them...if it's anything like mine a good holiday, promises of curtains etc. usually appeases the Finance Dept.
C7 AJM