This is a personal view and opinion and should not be taken as hard facts, not everybody would have the same experience. I’m sharing this because I think it might be interesting to others.

Some background first…

I’ve been toying with Cocoa development since around 2003 and back then it was difficult getting any information on how to use the tools and it all came down to doing a LOT of reading and a LOT of pain to figure things out. Since then there’s been a lot of great resources that showed up on the scene and a lot of books has been written on the subject matter. Given that I am a sucker for punishment (hey I have to be; I’m a developer right!?) I ended up learning about a lot of the pitfalls and quick wins in the Apple development world. I’ve since trained other developers in the art of not shooting yourself in the foot etc etc…

Now for the bit you actually want to read…

When Apple announced the iPhoneOS 2.0 SDK I was registered and ready to go as soon as I was able to do so, living in New Zealand meant that at that point in time we did not have access to official iPhones that’s supported by Apple – and this made life interesting. I paid my subscription fees as soon as Apple would take my money because this was the next gold rush! With every new platform it’s normally a case of using it for a while to realise what is missing form it so you can develop the required software. Unfortunately it took me a while to realise what I’d like to have on it and low and behold there was nothing like that on the App store (at this point in time there was only around 500-1000 apps on the store so it wasn’t exactly hard). Being a consultant the thing that I’m the worst at and is the most important, is keeping track of my billable hours – this is also my least favourite thing to do but my life depends on it so to speak.

Time Tracker was born (if you’re that way inclined you can search for it on the app store) and it was meant to be the “be all and end all” of my time keeping woes. I designed and built it to function the way I do, and believe me when I tell you I function in strange ways. Given that I’m a unix geek at heart the application reflects this, it’s minimalistic does exactly what it needs to with the least amount of effort possible (get in and out quickly so you can carry on with the important stuff right!?). Of course as life would have it, during the development a few other applications showed up on the scene doing the same thing, and they were priced cheaper than what I intended on pricing Time Tracker (More on sales numbers later on). This almost made me give up and just buy one of those applications to help me keep my time and forget about Time Tracker.

So what kept me going, firstly I had a bit of down time in my consulting – it wasn’t enough for me to rush out and find more work so I figured I’d just wait it out till my regular client’s were ready to pass more work onto me. This was also the perfect excuse I needed to apply Cocoa in anger – as I’ve always wanted to. I must admit that it only took me around 60 hours of development to get a functioning version up and running, but it took about another 60 hours to deal with all the other supplementary requirements to have an app up on the iTunes App Store. Since it’s been on the app store I’ve not done too much to it but fix the occasional bug and breaking changes that Apple introduced with each new release of iOS. I’ve also spent a fair bit of time with customers having issues (and sometimes these are as a result of their device not Time Tracker – this could be hours wasted). To make life easy we can call it 250 hours spent on Time Tracker and these house are still counting.

Given these numbers I would have had to sell around 5500 units at an average of $4 per application just to break even, granted my pricing varied a little over the last 2.5 years but on average it would be about $4 per unit.

So the burning question is “How did the sales go?”.

To be honest I didn’t even recover half of my investment (I’m not complaining about this, I had fun doing it and it taught me a little bit more about Cocoa so I write it down to training). Next you’d probably ask me why I still have Time Tracker listed on the App store and that would be a fair question. I’m still getting a few payments from Apple once in a while for sales on the few small Apps I have on the store, it’s not much but it covers the fees I need to pay Apple every year to keep my subscription active, so it doesn’t appear to do any harm. This assumption is of course wrong because all it takes is one person to email me with a strange issue they are experiencing and I can waste 8 – 16 hours testing and debugging (sometimes there’s a real bug, but most of the time it’s not, as every good developer should; I always assume that I’m to blame and start looking at my creation to figure it out).

Another point of interest is that I created a Lite version of Time Tracker that I intended on giving away free. I quickly reversed that idea and have since been selling it mostly a the lowest price possible due to the other developers downloading it so they can give it a horrible rating and review (I’m not saying all the bad reviews are by the competition, Time Tracker is certainly not for everybody and I’m not ashamed to admit this either). I’ve also experimented with giving away the Apps for free for a few days to see if it would increase my user base, I’ve learnt that people will download anything just because it’s free but never use it. I could tell this from my upgrade statistics every time I’ve released and update.

Now it’s not all bad, I sometimes get emails from users who really get what Time Tracker is meant to be, here’s an snippet from one I received recently:

I have evaluated like 3-4 Time Tracking application and I think your version is really the best. It’s the cleanest and simplest – quick and fun to use. All others had too much really unncessary buttons and stuff – with the cost of usability.

I also get a lot of feature requests, but unfortunately it’s not possible to implement these as I’m already behind and spending more time on it is ultimately just stupid – new feature does not translate to more sales as it’s already a niche market.

So what have I learnt from this experience and what can you take away from all of this:

• The cost of product development is far more than building the product itself.
• If you’re thinking of building any old random application for the App store and expect to make a lot of money out of it, you’re probably wrong (then again you may get lucky – it does happen).
• Knowing how to build iOS software is actually valuable, as I briefly mentioned I train developers and I also build software for other companies (this is far more lucrative).
• People like free, but that doesn’t mean they’ll use it.
• People actually expect a lot for very little payment, some feature requests would take more than the 250 hours spent on Time Tracker alone.
• I’m not the best use case designer for Time Keeping tools – it might make perfect sense to me, but not many others think so!

So naturally you might ask if I’d do it all over again? The answer is Yes. The simple fact of having an App up on the App store has lead to a fair bit of other business. I’ve built some software on the iPad that will hopefully earn me a bit of money and make some people in the enterprise space very happy, it’s not up on the App store yet, but it will be and it will be free – the business model is completely different for this and I might even blog about it in the future. I’m also about to embark on another App store project with a team of great people and an idea that sounds like it’s got much better legs than Time Tracker – also this time I would not be designing and building for myself which will make a huge difference. I still think there’s a lot of untapped potential in the iOS space hopefully I can tap into some of this and do better in the future.

The bottom line is that even though I’ve just outlined a less than favourable experience, it’s not completely put me off the idea of building App Store Apps. It’s better to have tried something and failed than to never have tried at all.

If you’ve read this far and I’ve completely put you off building iOS applications, don’t be put off, build it regardless.

The absolute worst case scenario would be that you’ve learnt something new, this is never a bad thing!

Life got very interesting one and a half months ago. I had a meeting with my employer where he sat me down and asked me if GridSpy was ready to survive without my salary. Essentially he asked me if I was ready to leave and take GridSpy full-time. Being an VC funded entrepreneur himself he knew how hard it is to make that initial leap into full-time start-up and gave me a gentle nudge “out of the nest.”

After that meeting my mind was racing. My wife and I agreed that taking GridSpy full-time was absolutely exciting but equally frightening. We’ve been tightening our belts for some time to get ready for bootstrapping but even we don’t know how soon GridSpy will transition from a ‘sure thing’ to a trading business. I assured her that I could land on my feet with a contracting job should the money run out.

So with much trepidation I decided that it indeed was the time to leave.

With a month’s notice given, I served out the notice period with tense anticipation, mostly worrying about the money. It turned out that the money problem soon solved itself. A friend of the family runs a stealth start-up which now had need of a contract programmer, and it was suggested to him that I would fill the role..

So the plan was to serve out my notice and start the contract after a well deserved full-time month on GridSpy. It soon came to be that my new client had an urgent need for my services – arguably more urgent than our own start-up for at least a couple of weeks. As a founder, it was a real heart wrenching experience. With complete control over my time I have decided that cash in the hand now to build initial stock is worth delaying our product just that little bit longer. I’ve also had a nice little splurge, for instance getting the required noise cancelling headphones.

It has been four weeks now since my last day as a salaried employee and I’ve had a great time. For the first time in ages overtime has been optional, working conditions are in my control, and opportunities are endless. However, the opportunities being endless thing has taken me away a little, well a lot, from GridSpy. These last four weeks have been great for my client but not so great for GridSpy. I have really enjoyed the programming challenges of the contract work, but I really haven’t spent any more time on GridSpy than I would have before. I seem to have exchanged one full-time job for another. Again I appear to be working for ‘the man.’

However, my working conditions now are much better. I’m working from home on my favourite sort of problems with my four screens. After a 5 step commute from the kitchen table to my office in the morning I work solid until lunchtime. The afternoon starts late after lunch and a play with the kids and is occasionally interrupted by one of my gorgeous children demanding a cuddle. Dinner is at 5pm and since that tends to work out as a 6.5 hour day I often work an odd evening too. Even though it seems like a short day I have found my output is huge. I am really able to totally concentrate and complete tasks in a way I enjoy. I’m much happier and I’m spending much more time with my wife and kids – that’s great!

Back when I was a Part time entrepreneur I constantly dreamed of the day that I went full-time. How much easier it would be when I could work on GridSpy and not have to stop. When looking at schedules of when the next feature would be ready I could wistfully say “well, it will take a long time but only because I can’t work on it full-time.” Although it was true, it was also a very convenient excuse.

Now I have exactly the opposite problem. With complete control of my time comes an equal amount of guilt when progress on GridSpy doesn’t suddenly take off. Since late last year I have been eternally thinking that we are only two months away from a product that is worth selling, and today I would make the same claim. What we do have today is a fantastic product and a handful of very happy clients. Today our Minimum Viable Product is pretty refined. It is focusing all my energies into those two months, or what it takes to get GridSpy out there, I am now struggling with. This has been a surprise for me.

Each week I can choose how much to work – somewhere on the slider from “Instant Cash Today” to “Successful business tomorrow.” It doesn’t help that I love my contracting job. I had better set the slider back to “Successful Business” before we run out of time. To do that requires the ability to say “No.” No to fun contracting tasks, No to quick cash, No to partnerships that require a lot of time on our part, and No to distracting marketing. But how do I break away from ‘the man’ when I actually really enjoy the work (and the cash too.)

Reflecting on what I have written I can see that I need to reduce the contracting work to the point where it pays some bills only, and spend as much time as possible putting GridSpy as a finished product into the marketplace. Well, after this particular contracting task is over that is.

Further reading:

• Part time entrepreneur, Fulltime Employee
• How our devices have evolved

See Also:

• Gridspy Homepage
• Live demo of GridSpy

Part Time Entrepreneur, Full-time Employee

Most startups seem to embrace 60-80 hour weeks, you keep reading about them and begin to believe that this is normal. After 12 hour marathon coding sessions ‘typical entrepreneurs’ walk 10 metres from desk to bed and collapse in their shared accommodations. Living in such lean conditions makes those crucial first months of business far cheaper. This work ethic and minimial living costs maximises the runway before the seed money runs out.

Paul Graham is one of my favourite bloggers, and his essay entitled The Other Road Ahead he paints a clear picture of how lean a start-up can be, stating “You can literally launch your product as three guys sitting in the living room of an apartment, and a server collocated at an ISP. We did.”

No matter how much I’d love to take the lean ramen noodle approach it’s simply not practical for me. I’m a father of two, a husband of one, and an employee. I’ve got a beautiful family with young children, a modest house with a mortgage, and a full-time job to pay the bills. In short, I have a standard existence that I imagine many of my readers share. Living off noodles in a cheap apartment with my co-founders is not the only way that start-ups are built. I’ve taken an alternative route that involves just as much hard graft (harder maybe?) and allows me to remain employed full-time. The food sure is better!

I simply have to fit Gridspy into the gaps between the hours at work outside of the home and the start-up and the time put into fatherhood and being an available partner. It is a compromise position that I have had to maintain for the last year until Gridspy was ready. Along with the mandatory overtime, maintaining a full-time job limits my flexibility. I cannot travel to meet potential clients as easily or pick up the phone to talk to them. It markedly slows down progress. However, without the job there would possibly be no Gridspy, or less so than there is now, as there would be no base to develop from.

Despite my full-time commitments I still consistently work on Gridspy. As I mentioned in Selling the Dream I spend four evenings a week working on Gridspy (3-4 hours each one) plus all of Sunday and manage to make a lot of progress. Sometimes I spend my four hours on a blog entry, sometimes on some firmware or a little bit of Django code. Lately I have been creating the setup process for new users. There is a ton of documentation to do on our website – it all takes so much time.

Since my baseline existence is ‘family-man’ and a full-time job, I have had to pace myself with Gridspy. If I am not careful I will burn myself or my loved ones out (and trust me, we’ve been close at times) so I have to be the tortoise rather than the hare. At least doing a small chunk every night gives me a lot of time to think through technology decisions and plan my development.

Of course Gridspy is all that I want to work on and all that I can think about. This creates constant tension/wishful thinking in wanting to jump in both feet first and get everything up and running now, yesterday! It doesn’t help that this fully dedicated full-time start-up founder is also a widely published expectation.

But there are no savings to live off, and even though the almost three year old in my life would like to live on noodles for a bit, I don’t think I could for long. Being employed isn’t as simple as just turning up to my ‘day job’ and punching keys so I can pay the bills. I want, need, to be a ‘good employee.’ It takes conscious effort to honour my commitment to my employer. I remind myself everyday to focus on the work I do for them and to leave Gridspy at home. I’m sure you can see how after a long day at the office and after the chaos of the early evening with the kids, it can take a while to get down to the business of Gridspy. Sometimes all I want to do is grab a beer and turn on the TV.

I like my job, it has loads of benefits. Not least that I can afford to buy beer and have nice dinners and provide my wife and children with all the comforts of home. I have been given some great introductions to potential business partners through my well connected colleagues. I’ve had a group of fellow engineers to discuss my ideas with and sound out my features. Plus I’ve had lots of free coffee and idle banter around the water cooler to enjoy. The benefits of a stable routine and a reliable injection of cash into our bank account each month are not to be frittered away lightly.

But when it comes to developing my own gig, it is a chicken and egg problem. The sooner I take the leap and go fulltime with Gridspy, the sooner we can complete the features that are preventing us from doing a large scale release. But the moment I take that leap, the clock (and the bank account) starts ticking down. I have an extremely short runway and I worry that I could scuttle this opportunity before it has had a chance to take off.

Reading entrepreneurial blogs such as Paul Graham’s can make you think that you have to go fulltime or give up. I am taking an alternative route. By creating Gridspy part-time I have been able to give it the time and breathing room required to succeed. I’m starting to think that this path is the most courageous since it requires serious commitment to devote all your free time to your startup. In the end, I am sure that my compromises will pay off.

Further reading:

• Selling the Dream, putting Sales before Software
• Introducing the Nexus

I’d love to hear more about what you are all doing in your valuable free time and how it is going.

Thanks Eben for the chance to share this blog entry.

A: You know, I can’t deal with code that’s not correct.
B: But what is correct code? Isn’t the correctness of the code defined by the usage?
A: Yeah, that’s for sure, but you know, there’s still the ‘correctness’ in the code. I can’t leave a code behind that is not correct. Something that’s broken

Who hasn’t had this conversation? Of those who haven’t, who hasn’t heard it from other people?

There are more than two kinds of programmers in the world. Today, though, I’d like to pretend that there are the two kinds. Those who are happy with the code when it’s good enough, and those for whom enough is never enough.

And no – this is not going to be one of the fit-for-purpose arguments. The whole notion is self explanatory, and to be completely honest, I find it slightly condescending. As professionals in this field, it’s not like we don’t understand the concept of boundaries, parameters, conditions, constraints, and other things that inherently limits us and our code.

What I’m interested in, is where we derive this notion of ‘correct code’ from. There are many books written, many more guide lines set and countless hours of code reviews done, all for the search of the correct code. All this, without really ever having a satisfying explanation – not to mention the definition – of the correctness of the code.

In the beginning,

there was, or as it seems; there was, the correct code. Shown to us earthly beings by the higher powers, revealed through the dreams of the highest, righteous ones such as Ada and Dijkstra. They, our forefathers and priests, were the bridge builders between the world of us – the programmers, and the realms of pure logic and computation.

The computation was platonic. Numbers were integers. Computers were ideas.

‘How might we add numbers?’ Asked a money lender.

‘You enter one number here, and the other number here, and turn this handle.’ The righteous ones answered.

The money lender did his calculations. There was much rejoicing.

Then an astronomer came along, and entered the distance between the sun and the earth, then the distance between the earth and the moon, and turned the handle.

The great computing machine, at once, started steaming, then crashed and burned. This event marked the birth of something known as the ‘representation’. The computation had fallen from the realms of pure and accurate integers.  Unsavoury things such as precision, FLOATING precision, LOSS OF FUCKING precision, and others, came forth from the mouth of the ‘representation’ in a torrent of vomit.

The numbers we meant was, if you will, the will of the computation. The numbers we entered and the numbers that came out, were the representation. This idea, naturally, was met with much disgust. Many people tried to pretend that this was merely a passing idea, and that once we get finer cogs and smaller gears and stronger handles, representation will get closer and closer to the will, and all will be good, and we’ll be able to enter the Garden of Eden once more.

This day, alas, never came. In the meantime, though, the duality of the computation became the norm. The representation was not going away, and even though it was still vomiting forth the torrent of unsavouriness, people had grown to accept them.

The will, or the requirement, was correct. Representation that faithfully conveyed the will, was a correct program. Requirements were romantic.

Then came the War, and everything changed. It was almost like a back pedalling by the community, almost as if they were gripped by a nightmare. The will was not enough. It was not correct enough, they said, because it’s not done in UML.

What is UML? Asked a few. Most of those who asked were sent to the Camps. Those who escaped – the lucky ones – were captured soon and promptly burnt at stakes. Lucky bastards.

The tyranny of FORM and DIAGRAM and ARCHITECTURE and PATTERNS and CAPITAL LETTERS swept across the world, sometimes like the Black Death, and sometimes like the Pneumonic Plague. Sometimes, less often, it was like Cholera.

For those who remembered the days of the good will and not-so-bad representation, it was a sad time. This dark, dark period, was referred to as the Modern period, because that’s when they started chronicling the changes, and of course it was Modern back then.

The madness hailed all things strict and formal. Specification was the new correctness.

But as we approached the turn of the century for the second time in the history of the Code, there came to be a new school of thought – one that said, the specification might as well exist, but it really doesn’t matter if it’s not going to RUN.  If they didn’t run, said these Post Modernists, they aren’t codes at all. Do they even exist? How do we measure the correctness of something that exists, against something that doesn’t? Isn’t that like a cruel trial where you’re not even told what you’re being accused of?

So these reckless, shitless pimple-faced kids, who were now in their late-20s, started making up their own ‘correct answers’, or also known as ‘tests’. Of course they weren’t really correct, but they were steadfast. They said it didn’t matter whether those correct answers were really correct or not. The correctness, they argued, could only be found in the constant interplay between the code and the test.

The pure and platonic logos of the computation was pushed back to the back room. The line between the code and the correct test, between the correct code and the wrong test, and vice versa, blurred. The line between art and trash, promiscuity and chastity, vice and virtue, high-tech and low-tech, hipsters and yuppies, phone and computers and cameras and TVs and portable navigation devices, blurred.

And we find ourselves in this blurry time. Are we any closer to finding out what the correct code is? Probably not.

Not all is lost, however – I take a great relief in thinking that the era of the Black Death, UML and other horrors is now over. The era of bad music, bland building and startlingly ugly furniture is finally over.

The quest of finding out what the correctness of code is, is probably a bit more arduous than the quest of writing the correct code. But only because it’s hard, it does not mean that we get to give up. The best we can do is to question. Keep questioning, whether the code we’re writing is correct, and whether how we judge the correctness of the code is correct.

And by the end of our times, or by next Tuesday, we might catch a glimpse of the next mighty ones.

About three years ago, I was working for this company ‘A’. I had been working for this company for more than 4 years by then, which was also my first full-time employer.

At first, I really enjoyed it. Colleagues were good, the boss was really good, the work was interesting, and I got recognition for the work I did. Also, the money was good. I really had very little to complain about.

Then slowly but surely, I got sick of maintaining the same software month after month. Who wouldn’t? So I asked the boss-man that I be given something new to work on. He was most accommodating, and I was given a new product to develop and a whole new team (which I quickly populated with my close friends) to build something completely new, from scratch. I was ecstatic. It was tremendous amount of fun, while it lasted.

After a while, though, things started to go sour in the company ‘A’. We had a new CEO, whom I felt really didn’t fit into the company culture. We also relocated, and the new working environment did not allow my ‘team’ to work effectively as we did previously. The upper management (the new CEO and the boss-man) felt that we were not delivering our product on time, we had marketing woes, and so on.

The new product lacked a clear direction, either technical or otherwise. No one in the company exactly knew what it did and whom we intended to sell it. The management changed direction whenever there was a potential customer. It went like this:

A potential customer would require a customised demo, but the suggested sample would cause some performance issues on our pre-alpha version. So we were told to improve the performance across the board, and we made some strange technical decisions to make it go fast. Then the next week, another potential customer would come along and tell us that they’d like to run it on .NET rather than Java, so we spent a week porting all of our code, like maniacs.

At this stage, it was impossible to keep the development team focused. We got frustrated. None of the customised demos were spectacular. We didn’t win any of those contracts, and the upper management got frustrated as well.

Feeling that our team was not performing, they ordered a feature-freeze on our product. The team was effectively disbanded, save about one and a half FTE (full time equivalent) for the maintenance and even more demo-building. I was taken off the product team, and put to something else.

That position is what I recall as the least fun time I had in the industry. I was not able to touch any of the code for our core products, I was on maintenance of a side project, and I spent my days answering calls from our clients. I felt that I was in tech-support.

Frustrations piled up, and so did the personal friction between various colleagues. My ex-team members left  the company almost all at the same time – within a window of a couple of months. I really couldn’t get along with the CEO. Although at the time I attributed all my problems to other people, now it’s clear to me that I was the morose bastard who was being difficult.

Then I got into a discussion with a friend, who also has been in the I.T. industry for a few years. She told me that she intended to leave the industry for once and for all, because it wasn’t right for her. She was going to pursue a whole new career, starting over. Encouraged by her actions, I decided to leave the industry as well. I didn’t set any concrete targets. I thought may be about three months before I started looking for a new job.

I went travelling. After about a month of not working, I stopped thinking about work completely. I stopped touching computers at ALL. I read books, climbed, cooked, ate, drank, played with friends, and so on. My perception of time became.. strangely real. I started counting time with the passage of the moon and the length of the day, instead of working weeks.

This state of mind lasted, more or less, for about eight months. I say again: for the eight months, I did not sit in front of a computer for more than an hour at a time. In a typical month, my computer-time would be around 30 minutes a week.

I stopped thinking in terms of problem solving, modelling, structuring, simulating, all the mental tools that I used for ‘work’ went out the door. I got to the point where, when I was making a cup of tea, I was able to concentrate all my effort in making that one cup of tea, and drinking it.

I became single-minded. If I wished to do something, I did it with a complete focus. Mind you, I was not managing difficult problems. I was trying to get things done such as ‘eat something’ and ‘drive to the next village, where I can eat something’. Occasionally I had to convert longitude-latitude into map grids, but that was a simple task with a calculator.

Then all of a sudden, when I was driving to the next village so I could eat something, in the expanse of the Mongolian desert, I had an epiphany. A part of my brain that was turned off eight months ago, spontaneously rebooted. Wherever I saw, I saw patterns. They arranged themselves into problems, which I’d try and solve. Mole-rat holes on the ground would become 8-Queens. Longitudes and latitudes were prime numbers – or were they?

And I liked it. It was like finding an old friend. Even though I loved the simple, clear thought process devoid of any background threads analysing everything, I realised, in the end, that I did those things for living because I Liked It. I went to a university to study it because I Liked It.

From there, returning to the industry was a brain-dead decision. I may have lost a year or so in my career, but I regained something that so few people in this industry seem to retain, especially when they become worldly and wise after a few years out of the university – the knowledge that I do what I do for no other reason than that I Like It.

I’m now a few years older, and a few years wiser; and I’ve found myself in a far off country trying to organise developments in a country 26 hours by plane away.

So, I thought I’d share some of the tips I’ve picked up from the experience so far. And for the sake of lazyness, I’ll write it as an “all time top 5″ list, a-la High Fidelity.

5) It helps if the two ends are culturally similar. I don’t mean culturally similar with regards to race, relgion etc… rather it helps when the two organisations are similar in ethos. Do you both have a similar level of commitment to clients? Do you both work similar hours? Have similar expectations of what constitutes an acceptable level of testing/documentation/speed of development?

4) Try and get the ability to allow both sides to access source. This gives you the ability to check progress… fix bugs and give suggestions that you wouldn’t otherwise be able to early on in the development cycle.

3) You need to clearly define the responsabilities at each end.. which country is responsible for tracking the project state? code deficiencies? documentation etc… Preferably there needs to be one designated person at either end ultimately responsible for deliverables. Without such a person, deliverables have a habit of languishing in the bowels of the development shop without progress.

2) Personal relationships are a key factor to making things work. Having met the other person at the other end of the phone can make up for a lot of process deficiencies. Put some people on a plane to meet the folks at the other end… if you haven’t met the side, your business is likely to suffer. Try and foster relationships not just with the senior members of the other side,  as the developers will likely be more open to going the extra yard for you if they have a face to put the work to.

1) Communicate. Daily if possible. Emails good, IM is better, Phone calls are the best. Make sure each side is aware of what’s going on…

Log[...] things that you [...] care about[...].

I couldn’t agree more.

I’ve come across this guy, who wrote in this book that we should all stop commenting. Instead, if you needed to comment, log!

What an interesting idea, I thought, so I started doing it for a little project. Of course, comments on classes and methods remained, but all in-line comments were converted into log messages.

A few things became very apparent very quickly:

1. Logging messages were littered with obscenities.
2. The normal operation of the program generated a LOT of logging messages. This was OK, since the code was actually doing some complicated thing that I wasn’t fully used to.
3. All the ‘easy’ part, that we don’t normally comment on, don’t get logged at all. This caused a few problems. Without those messages, though, the logging messages didn’t ‘flow’. That is, it would jump from one difficult part, without giving me any closure on what was happening, to another difficult part.
4. If you include the working variables into your logging statements, you practically have effectively compiler-checked comments.

After a while, though, it became evident that this approach had its limitations and I relented, and my boss complained about the massive giganto logging file, but that’s a whole another story.

.J

At my current job I’ve been fortunate enough to work on three projects covering each of these cases and I’ve learnt a few things along the way.

The first is that writing lots of code and making decisions is fun! But when you only work on that part it’s easy to look down on those who work on the project later. Ha! I wrote 90% of that in 4 weeks, now you’re spending months just adding trivial enhancements around the edges without really understanding what’s going on.

What you miss though, if you never have to take applications into live and beyond, is that there are a whole lot of things you can do that would make other people’s lives a whole lot easier if only you knew. You’ve got to see the issues that operations have to deal with in order to get the “yeah, there’s no way you could know that” empathy.

Logging

When you’re doing development, you don’t need to have very good logging. There’s a problem? Well, I’ll just read the code or step through a debugger. Once the application goes to testing it gets harder – um, I can’t reproduce that, can you give me more information? Production is worse still, with weird intermittent issues that operations need to detect and deal with even before talking to developers.

The first rule of logging is that when the application is running normally, the logs should be “clean”. Right from the beginning of development, we should keep an eye on what’s being logged. Do we get errors or warnings under normal operation? Fix them. You don’t want to be in a situation where you’re looking at a log full of errors and trying to figure out which ones are “normal” and which indicate a problem.

What do the logs contain when run at the level they’re going to be in production. We’re not running at DEBUG level – is there still enough information to see what’s going on. Are there lots of messages about trivial things and very little about other important parts of the system? Are there multiple lines of code that log the exact same message for different reasons?

Logging is one of those things that you need to care about right from the start. If you don’t keep an eye on it you’re never going to get it right. Functional (end-to-end) tests are useful here – check the output when they’re run.

Environment issues

I get really sick of dealing with P1 “defects” due to incorrectly set up test environments. The application fails with an error? What error’s that then. “table FOO doesn’t exist”. Or the reference data isn’t set up correctly – “just don’t do that” and everything will be ok.

What I do these days is keep a list of environment issues in a table on the wiki. Everytime someone screws up an environment because they forgot to apply a database patch, I add a row to the table with a query and expected output that you can use to verify that the error hasn’t occured again. You can bet that if a mistake is made setting up one environment that it’s pretty likely to happen again in others.

For reference data the application really needs to apply validation rules to the configuration at load time. It’s not good enough to say that “we don’t support configuring those things together” if the application could just as easily detect the situation and give an error.

One of my favourite head-smacking issues with testing is when multiple deployments of the application end up pointing at the same database. This is especially fun when they’re different versions, and you spend hours trying to figure out how on earth this behaviour is happening in version 2.5.2 when we fixed this back in 2.5.1. This happens during development as well when you “temporarily” tell a colleague to use your development database. One way to avoid this problem is to have the application post regular updates to a ‘heartbeat’ table including a unique runtime identifier (e.g. random number generated at startup), last update timestamp and version information. Then you can run a query to find those identifiers that have updated in the last 5 minutes and see who’s really touching those tables.

Requirements changes

Requirements churn is a good thing. As we find out more about an application and how it’s used we will naturally want to change earlier decisions. The problem is that it becomes difficult for anyone to know how the system is supposed to behave – or actually behaves. The requirements have changed, but did we ever actually have a chance to implement those changes? How did we “interpret” those parts where there was hand-waving? Where did we intentionally deviate without telling anyone What was that conversation where I spoke to a business analyst and they “clarified” some details of the requirements but never wrote it down?

On one of my projects back in NZ I was a bit laissez-faire about coming up with system behaviour that “made sense” without telling anyone. It wasn’t until we hit testing and I realised that testers had no better source of information than the original business requirements that I realised that although the software functioned well, there were a whole bunch of people who needed to understand how the application worked without reading the code.

What I do these days is to write up wiki pages whenever I’m implementing a major piece of functionality, describing precisely the behaviour I’ve actually implemented. If a requirement was ambiguous and I had a conversation with an analyst, I write down the conclusion in my document. Don’t leave it up to them to do it.

The great thing about this approach is that it gives you a safe way forward when presented with imperfect requirements. You can go ahead and get most of the work done and when it gets to testing and they interpret the requirements in a different way, go and mark it as “Functions as Specified” with a link to your document. I did that twice today and it makes life just that bit more pleasant. Quite often when there are multiple ways of designing some functionality it doesn’t matter all that much which one you pick, so long as people can easily find out which one that was. Even if it later turns out you mis-interpreted the requirements, presented with a clear description of the actual system behaviour the business may decide that the implementation is good enough, and that the requirements should be retrospectively changed to match. Won’t Fix!

Bugs in live

If you hit a bug outside of production, there’s not too much of a problem. We’ll just fix the code, re-run the test, sorted. When you hit a bug in live though, you have to recover from it.

Bugger.

This is the hard part of writing “business applications”. Making them reliable. Most of the time we just don’t bother and then there’s a major panic when anything goes wrong. You’ll have requirements that cover ‘negative flows’ by saying “create an alert”. Great, so the live system has just logged an alert saying we’re fucked. Time to go on holiday. Otherwise you’re sitting in a room full of worried managers who want you to find a workaround and fast.

The best aproach I’ve found for creating reliable applications is to expect them to fail. Expect that there will be bugs. Expect that hardware will fail. Expect that exceptions will occur. Expect that we’re going to have to recover.

How do we do that?

There’s no such thing as a database table that just contains internal application state that no-one other than the developers needs to understand. In some cases operations are going to have to hack this data – the structure and contents need to be well documented and understood. Functional tests are useful here; although some people claim that these tests should only assert against external interfaces and not against the database, I argue that the database is a semi-public interface. It’s not as formal and fixed as a web service, but developers, testers and operations all have a need to know how the database changes under processing.

Have hooks for inserting workarounds into live without deploying a new release (aka legitimate backdoors). For example, if the system applies validation rules to input messages, have some support for conditionally turning some of the rules off. One application I worked on was composed of dozens of little callable services with XML interfaces. The web interface included a developer-only page that had a textfield (service name) and textarea (xml) allowing you to send arbitrary internal messages to the live system. Dangerous but handy. Make sure you try your call in a test environment first! And don’t forget to secure it.

Expect that processing is going to fail, manual workarounds be applied, and we’ll then want the system to continue. The project with the callable services was at a Telco and integrated with billing systems and other monstrosities. Naturally some customers would have accounts that were … unusually set up, and would contradict any assumptions we could possibly make. To deal with this I turned the processing flow into a state diagram and then made it so that if an error occured at any state the system would set a “manual” flag on that customer’s record, generate an alert message, allow for any manual fixups and have a “reprocess” button to try again.

Taking that approach introduces some design contraints into the system that you simply can’t add as an afterthought. It’s easy and convenient to say, well at this processing stage we really need this piece of information, so instead of working it out again we’ll just persist it to the database at an earlier stage. That’s all well and good until the earlier stage didn’t actually go through automated processing – someone hacked it together by hand and then continued.

Whenever anyone pontificates about “loosely-coupled components” that’s what I think of. If Component A doesn’t actually run, will Component B still work, or does it depend on some of the internal behaviour of A?

Don’t try too hard to enumerate all the different things that can go wrong. Just assume that any component can fall over half way in some unexpected way, possibly leaving data in a corrupted state. If you can handle that you can handle anything. Data corruption is an interesting one, actually. Try to segment data to that if say the data related to one customer is corrupted, processing can still continue for other customers. This is similar to sharding that’s done for performance reasons.

Functional Tests

Ok, so I’ve talked about functional tests in a previous post. That’s because they’re great. When a problem occurs in live and you can in ten minutes write a test to replicate the problem and experiment with workarounds – I’ve done that many times. When a new developer joins the project and can actually understand and discover functionality and avoid breaking half the application – yup.

The rest

There are plenty of other issues to consider. How do we deal with upgrades – can we migrate old data? So you’re changing the database schema – are you aware that people have written systems that perform queries directly against your database? They’re going to break and you don’t even know it.

Reporting is a great one. It seems that “reports” are always something that get implemented at the last minute and are a nightmare because, surprise surprise, they want some information that, sorry, we just aren’t representing in our domain model. It gets even worse when you have performance challenges and you’d really like it if the application would’ve persisted certain intermediate statistics as it’s processing rather than us having to work everything out afterwards. Actually, I could do a whole ‘nother post on performance, and on system integrity checking (reconciliation), deployment (clustering) ….

I’d be interested in hearing any ideas or war stories other people have about dealing with live systems. Leave them in the comments!

Today a rather strange thing happened to me, I typed one another dev’s DAS Keyboard so my mind switched into full on touch typing mode, this lasted about 20 seconds before I started typing DVORAK instead of QWERTY (I can touch type both layouts – ironically I’m typing this on QWERTY). I’ve also been trying since around 1999 or 2000 to switch to the DVORAK layout but I almost without fail revert to QWERTY after a few weeks. I don’t revert because DVORAK is simply too hard or too slow because I’ve actually managed to build up my DVORAK touch typing speed to something that is almost useable. 

There are certain trends in the IT industry that just bugs me (and believe me that’s an understatement, ask anybody who’s had a similar conversation with me!) for instance the worst possible technology will be the industry standard! There are many examples of this, I’ll leave this as an “exercise” for you “the reader” to research. Back to the topic at hand, QWERT vs DVORAK is an excellent example – now I realise that some of you will claim that my statement is false and there is no scientific proof blah blah blah, I’m drawing on personal experience and in MY OPINION QWERTY sucks! Personally I like the way DVORAK feels while typing and that is why I believe that the worst layout is the standard. If you’ve read this far you might think that this is going to be another lecture on why you have to switch to DVORAK, let me assure you it is not!

So why am I mad as hell? There is a force that keeps me from my favorite keyboard layout and I wish I could eliminate it, unfortunately I do not think I’ll be able to. What’s my current options to get onto the DVORAK layout? I can use a soft switch by changing my keyboard layout on the OS layer, this might be fine for your average user that lives on a single machine or at worst case have a machine at home and a machine at work. In my case my home machine is also my work machine so theoretically I should have my cake and eat it? Another alternative is to buy a hardwired DVORAK keyboard, not only are these hard to find, some of them are just plain silly looking, tacky or very expensive . If there’s another one I’ve missed let me know in the comments I’d like to investigate it!

Interestingly neither of these actually solve my problem. I’m a laptop user, I’ve got a MacBook Pro to be exact, I’m also a Microsoft developer which means I spend a lot of my time using VMWare to run windows, not only that I work on many server environments via remote desktop and I even remote to some of my other Mac’s. The servers I use are shared machines which other developers use too and they are not DVORAK users either. So the first problem is that when I switch my Mac to the DVORAK layout it’s all fun and games until I move to windows VMWare will treat my keyoard as QWERTY regardless of what the Mac thinks it’s using, and right fully so the hardware is wired for QWERTY so it makes perfect sense, it’s virtualised and not dependent on any Mac settings. My real issue starts with the remote desktop connections, if Windows is set to use DVORAK remote desktop will blatantly ignore this and treat my keyboard as QWERTY – this does not make sense since it’s NOT virtualised and I’m merely sending the remote machine keystrokes! Suddenly I have to switch every machine I connect to to DVORAK when I only want my own machine to be switched and send the keystrokes as DVORAK instead of QWERTY to the machines I connect to. This works well when I’m using a terminal and ssh into a remote machine, why does this not work for other remote connections?

Another problem with the DVORAK layout is that short cut keys are usually chosen for their location on they keyboard or due to their relation to each other, undo, cut, copy and paste for example. DVORAK scatters these keys all over the keyboard so it makes no sense using them anymore and you loose all the benefits of using DVORAK instantly. There’s a partial solution to this with DVORAK layouts that switch to QWERTY when you hit the command key on the apple (I’ve not been able to find the windows equivalent yet, but it does not matter since windows produces too many other headaches besides this one). Don’t even get me started on trying to use EMACS or Vim (I use both I’m not religious!) on DVORAK keyboard, things go bad VERY quickly!

So my solution to this, and this is a theoretical solution I highly doubt I’d be able to find a practical implementation of it anywhere:

Change my laptop keyboard layout on a firmware (possibly even hardware, but I don’t believe it’s required) level, but I don’t want a standard DVORAK layout because of the short cut keys issue. Come to think of it if windows copied more of Mac OS X you might actually be able to globally remap all short cut keys to DVORAK equivalents. This might only get you part way there since there’s other issues on Mac OS X that will prevent you from doing this since Cocoa provides this magic and Carbon apps will give you the proverbial finger! The only way this solution will actually function properly will be when implemented on a lower level than the OS and even then it will be far from perfect.

So in conclusion DVORAK will ultimately be an EPIC fail, this sucks and gets my blood boiling, I really do like the DVORAK layout and in a perfect world there would be no QWERTY short cut keys would have been designed around DVORAK so all the counter arguments would also go away. Personally I feel DVORAK is better because on a subconscious level I automatically switch to it even though I spend 99.9% of my time typing on QWERTY. So if you’re thinking about switching to DVORAK I can honestly not recommend it unless you’re living on an isolated computer (which might be fine for most people).

DVORAK FAIL </ Rant>

P.S. Please prove me wrong! I’d like nothing more!

P.P.S. You might have noticed that I did not mention Linux in this post, I’ve given up on Linux many years ago due to the amount of pain associated with it (I did try DVORAK on it though and it sucked back then), my linux pain argument could fill a book so I refuse to go into it!

… or ‘OO is hurting the industry because encapsulation’

The ability to abstract is a powerful one. I have seen my share of uni-freshers. There are a million different kinds of students and they all have their different ways of understanding. Obviously, some are better than others, and some are worse.

The worse kind of students are those who cannot abstract. Of course all human beings can abstract to some degree or other, otherwise no one would be able to drive a modern motor vehicle, or eat a pie. But some students insist on ‘knowing all the details’ because ‘that’s how they understand things’.

To those students, the concept of an API is difficult to explain.

‘You call this method, and fun things happen.’
‘But how does it happen?’
‘You shouldn’t care.’
‘But I need to know!’

And there are sighs. They most likely won’t pass 101, or if they do get through, they’ll be miserable for the rest of the course, unless you’re a hot chick and have a full set of 14 geeky Asian guys to help you out with the assignments.

There are a couple of explanations that can be offered for this. Most people, even some experienced developers, tend to side with one explanation: they are bad at abstraction. Without it, you won’t get far.

I beg to differ.

The reason why they won’t do well is that they are asking clearly the wrong person. They should not be asking lab demonstrators, who in all their infinite (well, 2 extra years’ worth of) wisdom, have no clue about how that API works. They should be asking the professors, the writers of that API, internet forums, or read the documentation, whatever. The information is available, but not from the lab demonstrators who get paid 13 dollars an hour including tax.

Their weakness is lack of confidence that stops them from asking difficult questions to the proper authority, and laziness in their quest for information.

Look at the top students on the other hand. The chances are, they fully well know how their favourite APIs work underneath the covers. They probably know how the chips are manufactured, how gates are made, circuits are designed, machine code is executed, assembly code is written, compilers are architected, and so on, and so forth. All the way up, all the way down.

My point is this: abstraction is a powerful tool, but it’s only a tool and it’s only one of the tools.

To pretend that the students ‘shouldn’t care’ about how an API works is at best ignorant and at worst, well, just ignorant. Of course they should fucking care, you know? Like Knuth or Dijkstra or someone rather really important once said in one of their immortal essays, although what we produce is computer code, we should never forget that what we’re really crafting is the behaviour of the machine. The runtime matters. It’s the thing that we’re trying to get the computer to do.

How else, other than finding out what your function call does, are you going to determine what the computer will do? Reading documentation is fine and dandy, but that guarantees neither the completeness nor the correctness.

You will never learn how to fix a car by reading the owner’s manual, or even the repair manual, or even the blueprints of the car straight from the manufacturer. You have to open it up. Take the wheels off. Lift the car from its axle. Pry open the diff. Feel how broken that bearing is. Shake it and see what sound it makes. That is how you learn how to fix something. That is how you learn _anything.

Furthering my point, I make this claim: Currently, the ‘abstraction’ is severely overrated, especially in OO- and component-oriented programming circles.

What they write is this: the blue button makes it go. The red one makes it go faster. Do not press them at the same time. That’s the extent of the 90% of the API documentation there are.

And they expect us to work with it. What if you want to make it go slower? The trick is to press the two buttons at the same time, except for in the next minor version update, pressing the both buttons at the same time makes it blow up, and you’re punished for using an undocumented feature.

‘We told you not to’, says the smug author. And we agree with him. All the OO people agree that this is the way it should happen. Don’t do things that you’re not told to.

But hang on a minute. Don’t you think that’s even slightly weird? Why such change in behaviour? Why did it make it go slower in the first version? What deficiency in the original design prompted the authors to make such changes? Wouldn’t you want to know? Wouldn’t you be curious?

Most of all, wouldn’t you be pissed off if it broke your app?

But no. We’re told it’s just like a contract between two entities. You keep to the words, and don’t do things that are not specified it in the contract, or there be consequences.

Bullshit, I say: in the real world, if pressing those two buttons made things go slower for more than 12 months, it would be in the de-facto go-slow-mode contract, which would be enforceable. The wordings of the contract itself be damned.

The proper way to make things would be to let people know why and how the things work. Abstract it, document it, that’s all fine, but do not hide how and why things happen. Craft every bit, every level of the application correctly, thoughtfully. Then let people see it. Let people see your fucking craftsmanship. If you ain’t got one, this is a very good time to get one.

This brings around me to the final point: although I started off this article by talking about abstraction, it’s not about abstraction as such. It’s about hiding behind it and thinking that kludge is OK as long as you stick to the contract, the API, the interface, whatever.

People don’t think in terms of contracts. They think in terms of behaviours, hows and whys. When you ring up your ISP, you don’t have the service contract in your mind. You want the internet to work, and you want them to keep quiet about your midget-porn download usage. Fuck the contract. Who gives a shit? Who even reads them? If they didn’t have the explicit midget-porn-secrecy clause in their contract, and if they spill the beans, you’d be furious, wouldn’t you?

So there. Be honest, write good code, let people see what’s going on. Do not hide behind the encapsulation. If you do, you’re a midget-porn lover.

.J