Chris James - Software Engineer and other things

Work 2018

30 December 2018

This post is a summary of work stuff over the past year which covers

The team and the way we work

I rejoined Springer Nature (SN) August 2017 on the exciting premise of being in an "innovations team" where we would attempt to maybe make scientific publishing a bit better using technology.

I was a little sad to be moving from a role mainly writing Go to Kotlin but programming languages aren't everything.

What mainly drew me to the role was the understanding that we would be empowered to work in the way we felt was best.

I am a big believer in "agile"; the actual agile, not the project management nightmare. Stuff like

Individuals and interactions over processes and tools

and

Responding to change over following a plan

We are a small team of 3 developers, UX and a BA. We work very collaboratively with little ceremony. To give an idea our "stand up" is having a chat about what were working on and our general thoughts on the work. By that definition we have at least 3 standups a day.

The responsibilities within the team are more or less shared and we rely on each other's relevant expertise to fill in the blanks when needed. I wont be blocked if our UX is away at a meeting, I'll just take a punt knowing it probably wont be that bad.

We talk a lot and we iterate quickly and actually talk to real users, rather than fighting through many layers of hierarchy.

With all of us feeling empowered and involved it means ideas come to us quickly and if we think we have a good idea we'll discuss it and try it out. We dont worry about 2 week long inceptions and prioritising backlogs; if we think its the best thing for us to try now, we'll do it.

If we get things wrong, we dont sweat it; we can change software. Someone very clever said

A prototype is worth a thousand meetings

The other key to us moving quickly is really acknowledging the rule that the last 10% of work takes 90% of the time and just avoid doing it (at least in the short term).

We often say

Perfect is the enemy of good

We would rather ship something that we hope is good, get feedback and iterate on it rather than tying ourselves in knots over perfection.

In terms of the way we work, I couldn't be happier.

But what are we working on?

The work

We were connected with a journal who had some exciting ideas around data and community building. We thought we could create a minimal viable product (MVP) peer-review system (PRS) to experiment around these topics to hopefully learn how SN can help in these areas.

We thought Trello was cool and we had actually interviewed someone who managed a journal entirely in Trello so it seemed like a nice way to bootstrap a "system" quickly.

Our first iteration was a simple HTML form for an author to submit a manuscript which would create a card in Trello. An editor could then inspect the details and then manage the whole process of peer review manually themselves (for now); when they were happy with the manuscript if they dragged it into the column "Ready for publication" our web hook would listen to that event and then create an XML payload for publishing. We turned this around in about a week and felt really rad.

From there we kept iterating, adding a simple peer reviewer search and inviting mechanism (still in Trello).

Eventually though Trello's limitations in terms of UI were becoming burdensome. It was great to prototype and we learned some interesting things from the users about their processes but ultimately we needed to control the UI. We didnt throw it all out and start again though; we just kept iterating, creating new web pages to address our new needs until eventually we didnt need Trello at all.

We "launched" with our MVP submission system for the journal in September. Since then we have carried on iterating based on user feedback and have been working to try and make the arduous quality assurance process for a manuscript a more pleasant experience.

Innovation vs invention and politics

What exactly is an innovations team? Some would think that we should be using blockchain terraform kubernetes machine-learning wizzpops to change the world. I have often wondered why what we build isn't massively transformative.

The talk Building a winning product and UX strategy from the Kano Model by Jared Spool gives a lot of clarity for me.

The real tension is between invention and innovation. Innovation is where we improve a customer's experience so that they are "delighted". It's moving the needle from basic expectation to something that's awesome.

Invention is when you are launching moonshots at ideas that are wild and likely to fail but if they work, could radically change the business.

When I think about how software teams should be working, they should all be empowered to be innovating. This to me is what agile is all about. You should be shipping a product, collecting feedback and as a team understanding it as best you can and then trying out new ideas to move the needle.

This is exactly how we work and I feel this is how every team should work. You dont work on a project (because there's no such thing as projects in software), you work on a product. You start small and keep iterating for as long as the product lives.

As your product grows so does your team, in terms of improving the way it works, how it collects information and ideas around a domain so that it can deliver better ideas quicker.

Maybe software wouldn't be so expensive if we treated it properly rather than shipping on a particular date and then letting it rot; then re-writing it all over again forever and ever.

My conclusion for all of this, is the whole concept of an innovations team is an anti-pattern. Which is a bit weird to say given that was what was sold to me when I rejoined SN.

The joy of monolith

Until this year I have mainly worked on microservices systems for a good 4-5 years. I really like the approach of microservices in terms of creating small systems within bounded contexts and then gluing them together.

I've always understood that the cost of this is in terms of increased complexity in terms of continuous integration (CI), deployments and making sure your system actually works with unreliable network calls and evolving APIs so they dont break each other (e.g. consumer driven contracts)

This year we monolith'd our system and it has been incredibly liberating. I sometimes sit in meetings with other teams as they discuss their technical stuff and I had forgotten how much of a pain building distributed systems is.

This has really cemented the idea that you should always start monolith when building a system.

YAGNI is getting ignored too much these days, every recruitment email tells me how its a "greenfield microservices project" - sounds awful!

It's far easier and quicker to build your monolith and then you get to see the actual areas in your system that could be separate services when the API is a bit more mature.

A lot of the perceived downsides of monoliths are often fixed with just good programming practices.

At some point a system does get too big and you'll need to break it up, but maybe we should worry about building a successful system before breaking it up?

Kotlin

After writing Kotlin for about 18 months I conclude it's alright.

It feels like a nice happy medium between Scala and Java, but I do miss a number of Scala features such as for comprehensions and pattern matching.

These features being missing often makes Kotlin code more complicated than the Scala equivalents which flies in the face of the opinion of Kotlin being simpler than Scala. Abstractions can and should bring simplicity when used correctly.

Scala for better or worse gives you a lot of abstraction power which most developers will make a mess with if I'm being honest; often myself.

The Kotlin compiler is still quite slow; especially compared to Go but its much faster than Scala.

Ad-hoc Kotlin-isms

I'm personally not a fan of the ad-hoc syntax it has added for commonly known problems. It feels like to me they have really tried to avoid acknowledging monads and the like but have invented different things to learn instead.

I've run into occasions where I've wanted to refactor code from a T? (nullable T) to a List<T> which is harder than it should be.

In Scala the abstraction is clearly modeled so you can call map on them in a consistent way. So if you change the type there's very few code changes to make.

In Kotlin if it is nillable you do things like myThing?.let{it + 1}. But with lists its myList.map{it + 1}. To me it's annoying that they're different. A nullable thing should be seen as a collection of one or zero items in it.

Ecosystem

From what I read a lot of people writing Kotlin still want to feel the comfort of Java land in respect to relying heavily on frameworks like Spring. I really dont want to use a massive framework and neither does anyone in our team.

Thankfully SN has invested a lot of time and effort in allowing some developers to create a lovely library called http4k which offers a simple API to create web servers.

I still feel Go is probably my weapon of choice for a general purpose language but it's certainly not a deal-breaker for me to write Kotlin.

Writing Learn Go with tests

SN is nice enough to have 10% time. In March I thought it would be fun to do a TDD guide of a "Hello, world" function in Go.

I stumbled on a nice approach of starting with a simple function and iterating on it, adding new features of Go as the software grows and gets refactored which results in quite a nice flow of learning.

It got a lot of attention on the social media thing, getting top of hacker news and lots of internet points in Github stars.

This led to me feeling very motivated and since then I've written a lot of content covering most of the Go programming language syntax by TDD-ing useful functionality.

You can find it all on Github

For me the most amazing thing is some members of Chinese Go community have translated it into Chinese!

A few chapters have been contributed by other members from the community and a lot of my terrible grammar has been fixed. Overall there are 39 different contributors to the project.

Using gitbook it has a website and I used Pandoc to generate an epub file from the markdown that the content is written in.

The feedback from the community has been invaluable and very rewarding. Committing knowledge to paper is challenging and trying to explain what you know helps solidify your ideas and I am sure the whole experience has improved me as a developer.

I am immensely proud of this work and intend to press on with it to add more content in 2019.

Summary

The Tests Talk

18 December 2018

This month I had the opportunity to present some ideas floating around in my head around tests, refactoring and design at the Go London User Group.

Here's a link to the video

Here are the slides

If you're not into videos, here's wordy version of it.

Software

The promise of software is that it can change. This is why it is called soft ware, it is malleable compared to hardware. A great engineering team should be an amazing asset to a company, writing systems that can evolve with a business to keep delivering value.

So why are we so bad at it? How many projects do you hear about that outright fail? Or become "legacy" and have to be entirely re-written (and the re-writes often fail too!)

How does a software system "fail" anyway? Can't it just be changed until it's correct? That's what we're promised!

A lot of people are choosing Go to build systems because it has made a number of choices which one hopes will make it more legacy-proof.

Even with all these great properties we can still make terrible systems, so we should look to the past and understand lessons in software engineering that apply no matter how shiny (or not) your language is.

In 1974 a clever software engineer called Manny Lehman wrote Lehman's laws of software evolution.

The laws describe a balance between forces driving new developments on one hand, and forces that slow down progress on the other hand.

These forces seem like important things to understand if we have any hope of not being in an endless cycle of shipping systems that turn into legacy and then get re-written over and over again.

The Law of Continuous Change

Any software system used in the real-world must change or become less and less useful in the environment

It feels obvious that a system has to change or it becomes less useful but how often is this ignored?

Many teams are incentivised to deliver a project on a particular date and then moved on to the next project. If the software is "lucky" there is at least some kind of hand-off to another set of individuals to maintain it, but they didn't write it of course.

People often concern themselves with trying to pick a framework which will help them "deliver quickly" but not focusing on the longevity of the system in terms of how it needs to evolve.

Even if you're an incredible software engineer, you will still fall victim to not knowing the future needs of your system. As the business changes some of the brilliant code you wrote is now no longer relevant.

Lehman was on a roll in the 70s because he gave us another law to chew on.

The Law of Increasing Complexity

As a system evolves, its complexity increases unless work is done to reduce it

What he's saying here is we can't have software teams as blind feature factories, piling more and more features on to software in the hope it will survive in the long run.

We have to keep managing the complexity of the system as the knowledge of our domain changes.

Refactoring

There are many facets of software engineering that keeps software malleable, such as:

I am going to focus on refactoring. It's a phrase that gets thrown around a lot "we need to refactor this" - said to a developer on their first day of programming without a second thought.

Where does the phrase come from? How is refactoring just different from writing code?

I know that I and many others have thought we were doing refactoring but we were mistaken

Martin Fowler describes how people are getting it wrong

However the term "refactoring" is often used when it's not appropriate. If somebody talks about a system being broken for a couple of days while they are refactoring, you can be pretty sure they are not refactoring.

So what is it?

Factorisation

When learning maths at school you probably learned about factorisation. Here's a very simple example

Calculate 1/2 + 1/4

To do this you factorise the denominators, turning the expression into

2/4 + 1/4 which you can then turn into 3/4.

We can take some important lessons from this. When we factorise the expression we have not changed the meaning of the expression. Both of them equal 3/4 but we have made it easier for us to work with; by changing 1/2 to 2/4 it fits into our "domain" easier.

When you refactor your code, you are trying to find ways of making your code easier to understand and "fit" into your current understanding of what the system needs to do. Crucially you should not be changing behaviour.

An example in Go

Here is a function which greets name in a particular language

func Hello(name, language string) string {

  if language == "es" {
     return "Hola, " + name
  }

  if language == "fr" {
     return "Bonjour, " + name
  }

  // imagine dozens more languages

  return "Hello, " + name
}

Having dozens of if statements doesn't feel good and we have a duplication of concatenating a language specific greeting with , and the name. So I'll refactor the code.

func Hello(name, language string) string {
    return fmt.Sprintf(
        "%s, %s",
        greeting(language),
        name,
    )
}

var greetings = map[string]string {
  es: "Hola",
  fr: "Bonjour",
  //etc..
}

func greeting(language string) string {
  greeting, exists := greetings[language]

  if exists {
     return greeting
  }

  return "Hello"
}

The nature of this refactor isn't actually important, what's important is I haven't changed behaviour.

When refactoring you can do whatever you like, add interfaces, new types, functions, methods etc. The only rule is you don't change behaviour

When refactoring code you must not be changing behaviour

This is very important. If you are changing behaviour at the same time you are doing two things at once. As software engineers we learn to break systems up into different files/packages/functions/etc because we know trying to understand a big blob of stuff is hard.

We don't want to have to be thinking about lots of things at once because that's when we make mistakes. I've witnessed so many refactoring endeavours fail because the developers are biting off more than they can chew.

When I was doing factorisations in maths classes with pen and paper I would have to manually check that I hadn't changed the meaning of the expressions in my head. How do we know we aren't changing behaviour when refactoring when working with code, especially on a system that is non-trivial?

Those who choose not to write tests will typically be reliant on manual testing. For anything other than a small project this will be a tremendous time-sink and does not scale in the long run.

In order to safely refactor you need unit tests because they provide

An example in Go

A unit test for our Hello function could look like this

func TestHello(t *testing.T) {
  got := Hello(“Chris”, es)
  want := "Hola, Chris"

  if got != want {
     t.Errorf("got '%s' want '%s'", got, want)
  }
}

At the command line I can run go test and get immediate feedback as to whether my refactoring efforts have altered behaviour. In practice it's best to learn the magic button to run your tests within your editor/IDE.

You want to get in to a state where you are doing

All within a very tight feedback loop so you don't go down rabbit holes and make mistakes.

Having a project where all your key behaviours are unit tested and give you feedback well under a second is a very empowering safety net to do bold refactoring when you need to. This helps us manage the incoming force of complexity that Lehman describes.

If unit tests are so great, why is there sometimes resistance to writing them?

On the one hand you have people (like me) saying that unit tests are important for the long term health of your system because they ensure you can keep refactoring with confidence.

On the other you have people describing experiences of unit tests actually hindering refactoring.

Ask yourself, how often do you have to change your tests when refactoring? Over the years I have been on many projects with very good test coverage and yet the engineers are reluctant to refactor because of the perceived effort of changing tests.

This is the opposite of what we are promised!

Why is this happening?

Imagine you were asked to develop a square and we thought the best way to accomplish that would be stick two triangles together.

Two right-angled triangles to form a square

We write our unit tests around our square to make sure the sides are equal and then we write some tests around our triangles. We want to make sure our triangles render correctly so we assert that the angles sum up to 180 degrees, perhaps check we make 2 of them, etc etc. Test coverage is really important and writing these tests is pretty easy so why not?

A few weeks later The Law of Continuous Change strikes our system and a new developer makes some changes. She now believes it would be better if squares were formed with 2 rectangles instead of 2 triangles.

Two rectangles to form a square

She tries to do this refactor and gets mixed signals from a number of failing tests. Has she actually broken important behaviours here? She now has to dig through these triangle tests and try and understand what's going on.

It's not actually important that the square was formed out of triangles but our tests have falsely elevated the importance of our implementation details.

Favour testing behaviour rather than implementation detail

When I hear people complaining about unit tests it is often because the tests are at the wrong abstraction level. They're testing implementation details, overly spying on collaborators and mocking too much.

I believe it stems from a misunderstanding of what unit tests are and chasing vanity metrics (test coverage).

If I am saying just test behaviour, should we not just only write system/black-box tests? These kind of tests do have lots of value in terms of verifying key user journeys but they are typically expensive to write and slow to run. For that reason they're not too helpful for refactoring because the feedback loop is slow. In addition black box tests don't tend to help you very much with root causes compared to unit tests.

So what is the right abstraction level?

Writing effective unit tests is a design problem

Forgetting about tests for a moment, it is desirable to have within your system self-contained, decoupled "units" centered around key concepts in your domain.

I like to imagine these units as simple Lego bricks which have coherent APIs that I can combine with other bricks to make bigger systems. Underneath these APIs there could be dozens of things (types, functions et al) collaborating to make them work how they need to.

For instance if you were writing a bank in Go, you might have an "account" package. It will present an API that does not leak implementation detail and is easy to integrate with.

If you have these units that follow these properties you can write unit tests against their public APIs. By definition these tests can only be testing useful behaviour. Underneath these units I am free to refactor the implementation as much as I need to and the tests for the most part should not get in the way.

Are these unit tests?

YES. Unit tests are against "units" like I described. They were never about only being against a single class/function/whatever.

Bringing these concepts together

We've covered

What we can start to see is that these facets of software design reinforce each other.

Refactoring

Unit tests

(Well designed) units

Is there a process to help us arrive at a point where we can constantly refactor our code to manage complexity and keep our systems malleable?

Why Test Driven Development (TDD)

Some people might take Lehman's quotes about how software has to change and overthink elaborate designs, wasting lots of time upfront trying to create the "perfect" extensible system and end up getting it wrong and going nowhere.

This is the bad old days of software where an analyst team would spend 6 months writing a requirements document and an architect team would spend another 6 months coming up with a design and a few years later the whole project fails.

I say bad old days but this still happens!

Agile teaches us that we need to work iteratively, starting small and evolving the software so that we get fast feedback on the design of our software and how it works with real users; TDD enforces this approach.

TDD addresses the laws that Lehman talks about and other lessons hard learned through history by encouraging a methodology of constantly refactoring and delivering iteratively.

Small steps

As you become proficient, this way of working will become natural and fast.

You'll come to expect this feedback loop to not take very long and feel uneasy if you're in a state where the system isn't "green" because it indicates you may be down a rabbit hole.

You'll always be driving small & useful functionality comfortably backed by the feedback from your tests.

Wrapping up