Chris James - Software developer and other things

Why Test Driven Development

26 October 2018

It's difficult to write about Test Driven Development (TDD) without rehashing what others have said but it helps me to organise my thoughts around the matter. So in a way this is a selfish endeavour but I do hope this will at least get readers thinking about TDD and the important role it has in software development.

Software

The promise of software is that it can change. This is why it is called soft ware, it is malleable compared to hardware. A great engineering team should be an amazing asset to a company, writing systems that can evolve with a business to keep delivering value.

So why are we so bad at it? How many projects do you hear about that outright fail? Or become "legacy" and have to be entirely re-written (and the re-writes often fail too!)

How does a software system "fail" anyway? Can't it just be changed until it's correct? That's what we're promised!

In 1974, a long time before I was born, a clever software engineer called Manny Lehman described

The Law of Continuous Change

Any software system used in the real-world must change or become less and less useful in the environment

It feels obvious that a system has to change or it becomes less useful but how often is this ignored?

Many teams are incentivised to deliver a project on a particular date and then moved on to the next project. If the software is "lucky" there is at least some kind of hand-off to another set of individuals to maintain it, but they didn't write it of course.

People often concern themselves with trying to pick a framework which will help them "deliver quickly" but not focusing on the longevity of the system in terms of how it needs to evolve.

Even if you're an incredible software engineer, you will still fall victim to not knowing the future needs of your system. As the business changes some of the brilliant code you wrote is now no longer relevant. Software must change

Lehman was on a roll in the 70s because he gave us another law to chew on.

The law of increasing complexity

As a system evolves, its complexity increases unless work is done to reduce it

(emphasis mine)

What he's saying here is we can't have software teams as blind feature factories, piling more and more features on to software in the hope it will survive in the long run.

We have to keep managing the complexity of the system as the knowledge of our domain changes.

Refactoring

There are many facets of software engineering that keeps software malleable, such as:

I am going to focus on refactoring. It's a phrase that gets thrown around a lot "we need to refactor this" - said to a developer on their first day of programming without a second thought.

Where does the phrase come from? How is refactoring just different from writing code?

Factorisation

When learning maths at school you probably learned about factorisation. Here's a very simple example

Calculate 1/2 + 1/4

To do this you factorise the denominators, turning the expression into

2/4 + 1/4 which you can then turn into 3/4.

We can take some important lessons from this. When we factorise the expression we have not changed the meaning of the expression. Both of them equal 3/4 but we have made it easier for us to work with; by changing 1/2 to 2/4 it fits into our "domain" easier.

When you refactor your code, you are trying to find ways of making your code easier to understand and "fit" into your current understanding of what the system needs to do. Crucially you should not be changing behaviour.

When refactoring code you must not be changing behaviour

This is very important. If you are changing behaviour at the same time you are doing two things at once. As software engineers we learn to break systems up into different files/packages/functions/etc because we know trying to understand a big blob of stuff is hard.

We don't want to have to be thinking about lots of things at once because that's when we make mistakes. I've witnessed so many refactoring endeavours fail because the developers are biting off more than they can chew.

When I was doing factorisations in maths classes with pen and paper I would have to manually check that I hadn't changed the meaning of the expressions in my head. How do we know we aren't changing behaviour when refactoring when working with code, especially on a system that is non-trivial?

Those who choose not to write tests will typically be reliant on manual testing. For anything other than a small project this will be a tremendous time-sink and doesn't scale in the long run.

In order to safely refactor you need automated tests because they provide

Why Test Driven Development (TDD)

Some people might take Lehman's quotes about how software has to change and overthink elaborate designs, wasting lots of time upfront trying to create the "perfect" extensible system and end up getting it wrong and going nowhere.

This is the bad old days of software where an analyst team would spend 6 months writing a requirements document and an architect team would spend another 6 months coming up with a design and a few years later the whole project fails.

I say bad old days but this still happpens!

Agile teaches us that we need to work iteratively, starting small and evolving the software so that we get fast feedback on the design of our software and how it works with real users; TDD enforces this approach.

TDD addresses the laws that Lehman talks about and other lessons hard learned through history by encouraging a methodology of constantly refactoring and delivering iteratively.

Small steps

As you become proficient, this way of working will become natural and fast.

You'll come to expect this feedback loop to not take very long and feel uneasy if you're in a state where the system isn't "green" because it indicates you may be down a rabbit hole.

You'll always be driving small & useful functionality comfortably backed by the feedback from your tests.

Common objections with pithy responses

Tests don't help me refactor. Every time i refactor loads of tests stop passing/compiling

Remember what refactoring is supposed to be? Just changing the way your program is expressed, not changing behaviour. Now ask yourself why your tests are failing. It will be because your tests are too coupled to implementation details.

You're probably mocking too much and testing irrelevant detail. Remember a unit test is not only on functions/classes/whatever.

A unit of behaviour can be tested and it may have a number of internal collaborators to make that behaviour work; just don't test them!

Listen to your tests and act on what they're telling you.

I don't like writing tests as I want to explore the design first, then I write my tests afterward.

It is hard/time-consuming to write your first test; if your first test is "make a website to rival twitter".

Irrespective of whether you practice TDD or not it is an important skill as a software developer to be able to break problems down into small pieces.

This lets us work in a smaller problem space and deliver small pieces of value quickly, letting us validate our assumptions as we work. This is all about learning from the mistakes of the past with too much work on upfront design.

The beauty of TDD is it forces us to start small - unless you enjoy spending loads of time writing a big test without the endorphin rush of seeing a test pass.

With the constraint of starting small it will challenge your assumptions because you'll get feedback quicker.

Writing tests after the fact is usually harder and more error prone. You are more likely to write code that isn't easy to test because your code has been driven by assumptions in your head rather than tests demanding a specific behaviour.

In addition an important step in TDD is the first one; see how your test fails and see if the error makes sense. This forces you to write ergonomic tests that explain what has gone wrong to the developer reading it.

Too much of my career has been wasted debugging tests that fail with false was not true

It takes too long

You should read GeePaw's TDD & The Lump of Coding Fallacy as it explains brilliantly why this line of thinking is wrong (at least once you become proficient with TDD).

If you're too lazy my TL;DR version is

The "studying" part becomes easier because as GeePaw says

it’s almost like the test code forms a kind of Cliff’s Notes for the shipping code. A scaffolding that makes it easier for us to study, and this makes it far easier to tell what’s going on. This will cut our code study time in about half.

All the examples are unrealistic compared to "real" software

This comes back to being able to break problems down. As you gain practice with TDD and software development you'll learn how to break down problems so that they look like the simple examples you learned with.

Generally if your code is too hard to test; it's not "realistic" - it's poorly written.

Wrapping up

The Web I Want

20 August 2018

I originally posted this at Dev.to, where you can see some comments

This post will very much sound like I want you all to get off my lawn - because I do.

How has the web become like this?

From this very website

The other day my partner was browsing the web on her two year old Chromebook, it struggled to run a number of sites as they loaded ads, gifs, videos and tons of JavaScript to do absolutely nothing of value.

Every day I am frustrated at my phone as it chugs along trying to display a blog post. The post is there behind a sea of JavaScript and auto-playing videos.

This all sounds a bit first world problems, but it's far, far worse for those in developing nations. Do you know how bad satellite Internet is compared to what most of us have here in the west? It is awful.

Graph of mobile connection speeds

The World Wide Web is supposed to be a leveler, something that brings knowledge everywhere and yet developers every day are making it harder for those who need the Internet to work well the most.

Let's take a look at how things have "progressed" with the Internet.

20ish years ago

I made my first website about 20 years ago and it delivered as much content as most websites today. It was more accessible, ran faster and easier to develop then 90% of the stuff you'll read on here.

20 years later I browse the Internet with a few tabs open and I have somehow downloaded many megabytes of data, my laptop is on fire and yet in terms of actual content delivery nothing has really changed.

Technologies used: - HTML

10-15 years ago

People were fed up of nested tables and spacer images. The web was losing its roots of being a content delivery platform.

I was working on websites like the above in my placement year on my degree. I started reading articles on A list Apart about how we should be pushing for semantic markup, where HTML simply describes a document of content and then it is styled using this thing called CSS.

Eventually I ran into CSS Zen Garden which is a website that showcases what you could do with CSS.

The idea is the markup is the same and the website has submissions from developers showing different designs purely using CSS.

The HTML remains the same, the only thing that has changed is the external CSS file. Yes, really.

This website had a profound impact on my attitude to web development. It clicked. HTML for content, CSS for style. No JavaScript required, no tables needed for layout, no spacer images.

Just pure HTML decorated with CSS.

It felt exciting to be part of a community that took real pride in delivering beautiful looking content in the leanest, simplest and most accessible way possible.

Not only did this make websites more accessible and fast to run but in some ways made them easier to develop. Suddenly generating the markup from the server wasn't horrible!

Still things were quite difficult. CSS support wasn't amazing and we still didn't have a lot of semantic elements with HTML4 so there were a lot of divs. Firebug had just come out which was a huge boost but it was still hard to make a consistent experience.

I seem to recall Gmail coming out around this time and it was amazing. Suddenly the web could be actual applications. This is interesting and groundbreaking, but had a real negative impact too. Soon enough people thought that the promotional website for their local pub had to have as much JavaScript as Gmail.

5 years-ish till present

Here is a not exaggerated summary of today's attitudes

Scores of people who just want to deliver their content and have it look vaguely nice are convinced you need every web technology under the sun to deliver text.

You're not a frontend developer unless you know Vue/React/whatever. You're mocked if you dare to just write the bare minimum JavaScript you need for your website

The page refreshing is seen as a massive problem for users, and it must be avoided at all costs.

For some reason people are building tons of SPAs (single page web apps). The reasons stated are for speed and ease of developer use. (I don't buy sending tons of JavaScript to a browser will ever be as fast as just some damn HTML. Also it's not as easy as putting some HTML files on the Internets)

You see these laughable posts where developers jump through dozens of hoops to make their website "fast and performant". They struggle because of the underlying technical choices and then I'm still downloading half a megabyte of data to read 500 words. It's embarrassing.

There are a few things I want you to take away from this post

So what is the The Web I want

So no JavaScript, really?

JavaScript of course still has its place and when used tastefully can improve the usability of a website. However think carefully about the libraries and frameworks you pull in. Maybe you can accomplish what you need without bringing in JQuery, modern browsers have excellent APIs built in these days. Aim for progressive enhancement with JS. Your website should still work without JS turned on.

Regarding single page apps (SPAs), I genuinely believe too many people are making them. GMail is a web application and as such deserves a framework. Your blog platform? Not so much. Remember these frameworks not only put a lot of strain on user's experience but it's also just a lot to learn. Maybe your time could be served better.

But my product owner says we need all these bells and whistles!

As a professional, it is up to you to take a stand. You are the expert, not the customer.

If I told the builder of my home to make it out of straw, I would hope she would convince me otherwise.

Of course, circumstances are tricky and sometimes people will ignore you but when you look at the state of some of the popular websites today you can only conclude that people either don't know what they're doing or are just not pushing back on bad requirements enough.

Wrapping up

Take a look at the performance tab in developer tools for your website. Does it make you proud? If not, take some pride and start cutting away the cruft you thought you needed.

Let's take pride in making lean, accessible, simpler to execute websites by using simple technologies that work everywhere. There are moral and technical arguments for taking this approach.

Remember that the goal of most websites is delivering useful textual content, and all you really need for that is HTML.

Then we might not have to buy new laptops every 2 years and people less fortunate than you have a chance of actually using the web; the way it was intended.