The Web I Want
20 August 2018
I originally posted this at Dev.to, where you can see some comments
This post will very much sound like I want you all to get off my lawn - because I do.
How has the web become like this?
This all sounds a bit first world problems, but it's far, far worse for those in developing nations. Do you know how bad satellite Internet is compared to what most of us have here in the west? It is awful.
Graph of mobile connection speeds
The World Wide Web is supposed to be a leveler, something that brings knowledge everywhere and yet developers every day are making it harder for those who need the Internet to work well the most.
Let's take a look at how things have "progressed" with the Internet.
20ish years ago
I made my first website about 20 years ago and it delivered as much content as most websites today. It was more accessible, ran faster and easier to develop then 90% of the stuff you'll read on here.
20 years later I browse the Internet with a few tabs open and I have somehow downloaded many megabytes of data, my laptop is on fire and yet in terms of actual content delivery nothing has really changed.
10-15 years ago
People were fed up of nested tables and spacer images. The web was losing its roots of being a content delivery platform.
I was working on websites like the above in my placement year on my degree. I started reading articles on A list Apart about how we should be pushing for semantic markup, where HTML simply describes a document of content and then it is styled using this thing called CSS.
Eventually I ran into CSS Zen Garden which is a website that showcases what you could do with CSS.
The idea is the markup is the same and the website has submissions from developers showing different designs purely using CSS.
The HTML remains the same, the only thing that has changed is the external CSS file. Yes, really.
Just pure HTML decorated with CSS.
It felt exciting to be part of a community that took real pride in delivering beautiful looking content in the leanest, simplest and most accessible way possible.
Not only did this make websites more accessible and fast to run but in some ways made them easier to develop. Suddenly generating the markup from the server wasn't horrible!
Still things were quite difficult. CSS support wasn't amazing and we still didn't have a lot of semantic elements with HTML4 so there were a lot of
divs. Firebug had just come out which was a huge boost but it was still hard to make a consistent experience.
5 years-ish till present
Here is a not exaggerated summary of today's attitudes
Scores of people who just want to deliver their content and have it look vaguely nice are convinced you need every web technology under the sun to deliver text.
The page refreshing is seen as a massive problem for users, and it must be avoided at all costs.
You see these laughable posts where developers jump through dozens of hoops to make their website "fast and performant". They struggle because of the underlying technical choices and then I'm still downloading half a megabyte of data to read 500 words. It's embarrassing.
There are a few things I want you to take away from this post
- In practice your website would do everything it needed to with some HTML files linking to some CSS.
So what is the The Web I want
- Semantic, accessible, HTML. This means it works for everyone, is fast to download and is less effort than writing React components.
- Small images, when needed. Enough of these hilarious 4mb GIFs.
- Most websites should be ready to read after a 10kb download.
Regarding single page apps (SPAs), I genuinely believe too many people are making them. GMail is a web application and as such deserves a framework. Your blog platform? Not so much. Remember these frameworks not only put a lot of strain on user's experience but it's also just a lot to learn. Maybe your time could be served better.
But my product owner says we need all these bells and whistles!
As a professional, it is up to you to take a stand. You are the expert, not the customer.
- You should be informing them how Amazon learned that every 100ms in page load cost them a 1% of sales.
- You should be informing them that making a carousel of their product catalogue is slow and annoying and expensive to make compared to just listing them in a
- You should let them know that if they really want their groundbreaking poetry to reach around the world that they better make sure their website works on old hardware with slow Internet connections.
If I told the builder of my home to make it out of straw, I would hope she would convince me otherwise.
Of course, circumstances are tricky and sometimes people will ignore you but when you look at the state of some of the popular websites today you can only conclude that people either don't know what they're doing or are just not pushing back on bad requirements enough.
Take a look at the performance tab in developer tools for your website. Does it make you proud? If not, take some pride and start cutting away the cruft you thought you needed.
Let's take pride in making lean, accessible, simpler to execute websites by using simple technologies that work everywhere. There are moral and technical arguments for taking this approach.
Remember that the goal of most websites is delivering useful textual content, and all you really need for that is HTML.
Then we might not have to buy new laptops every 2 years and people less fortunate than you have a chance of actually using the web; the way it was intended.
Why you should deploy on Friday afternoon
28 May 2018
I originally posted this at Dev.to where lots of people agreed and disagreed.
Don't deploy on Friday afternoons!
This expression is taken as programmer wisdom but I hate it. I'm going to try and kill it, with words and experience.
The motivation behind it is sound. I don't want to spend my Friday nights debugging a production problem either.
To me the expressions smacks of unprofessionalism. Software development as an industry has a poor reputation and phrases like this do not help.
If you went to an ER on a Friday afternoon and was turned away because the doctors don't trust their own tools and know-how, how would you feel?
If we want people to take our craft seriously we need to own it and not give the impression that we dont understand the systems are we making enough to make changes at the end of the week.
Why a lot of people don't want to deploy on Fridays
- You don't understand your system. You're making changes but you're not 100% sure it'll be OK. Ask yourself why this is. Why are you scared of changing your software?
- Poor monitoring. If the users are the first to tell you if something is wrong, that feedback loop spills into time at the pub, rather than when you deploy.
- Overly-complicated rollback/branching. How is your disaster recovery? How easy is it to push a fix once you solve a bug?
Continuous Delivery (CD)
I have worked on teams that have deployed new versions of various services in a distributed system multiple times at 4:30pm and not broke a sweat.
Why? Because deployment is fully automated and is a non-event. Here is the groundbreaking process.
- Write some code
git commit -am "made the button pop"
git pull -r && ./build && git push
- Make a cup of tea
- It's now live.
Not so long ago it was considered normal for there to be releases every 6 months, or even just one at the end of a project.
The forward thinkers in that age saw problems with this
- Poor feedback loops
- Stressed development teams
- Catastrophic failures.
So the industry as a whole worked on lots of tooling, techniques and best practices to allow us to release software far quicker.
Recognising that releasing often reduces risk is generally accepted nowadays but teams still often settle on weekly or fortnightly releases; often matching the cadence of their sprints.
What are the problems with weekly/fornightly releases?
- The feedback loops are still not great. If you do your release there can be quite a lot of commits going live and if something is wrong it can be challenging to figure out exactly what broke. Especially if you wrote it 2 weeks ago.
- Still overly reliant on manual processes. I have seen teams actually skip a release because a QA was on holiday. This is surely unacceptable in 2018. Manual testing does not scale into the future. People leave, things get forgotten, etc.
- Let's you fall into the trap of writing stories that are dependant on other stories being finished in a "sprint". When they aren't things can get very complicated.
With CD we recognise that we can go further, deploying new software to live every time the build is green. This has some amazing benefits,
- Extremely fast feedback loops. No longer do you have to think about code you wrote 2 weeks ago when there is a problem in live.
- Forces best practices. In order to be able to deploy to live on green you need excellent monitoring and tests. These are all good things in their own right.
- Reduces stress. "Releasing" is no longer a thing any more. You can be confident in writing your software again!
- Vastly improves agility. Found a bug? Just fix it! This encourages a more lean way of working vs lots of upfront planning. There isn't even an option for a convoluted release process, you have to keep it simple.
- Forces you to work on stories that are actually releasable. Not dependent on story x y and z. Forces the best practices on user stories that everyone acknowledges but often people ignore.
But what if you break things?
Often people say with CD
Yeah it's nice but what if it breaks? We should have a QA check things over
Here's the thing, no process in the world prevents bugs. You will ship broken code. What's really important is how quickly you can detect and recover from it. Hoping manual testing will catch everything is wishful thinking.
How to CD on a new project
It is much easier to do CD on a new project since you can start small and evolve.
Generally your work should be centered on delivering the most valuable user journeys first, so this is an excellent chance to practice how to ensure that feature works without any humans checking anything.
- Write an end to end test. These are expensive to write and run and should only be reserved for your most important journeys
- Have monitoring with threshold alerts for when things go wrong
- Set up your pipeline so that when your code is pushed all the automated tests are run, if they pass go to production.
- Have some kind of green/blue release mechanism. Run your automated tests on the deployed release candidate and if they dont pass, dont ship it.
For each subsequent story ask yourself
- How will we know this is working? (monitoring)
- What tests do I need to have enough confidence this will work without any humans checking. Not every story needs a full end-to-end test on a huge distrubuted system but obviously you'll need some tests.
- Is this story as small as it can be? If your user stories are massive they are more likely to go wrong. If the story takes a week then that's back to slow feedback loops.
- If you cant answer these questions then you need to rethink the story. Notice these are all just basic agile principles for user stories. Continous delivery forces you to adhere to the principles that so often get ignored
How to CD on an existing project
Peel away at the manual process
- You may have some kind of "run book" that is used when shipping the software. See what you could do to automate it.
- Find out all manual processes are happening. Ask why they are needed and what could be done to automate them.
CD up to staging.
Some companies have many environments in their delivery pipeline. A good first start is to automatically ship all the way up to the environment before live. A better step is remove as many of them as you can. It's ok to have some kind of "dev" environment to maybe experiment with but ask yourself why cant just test these things locally in the first place.
Identify a sub-system you could work with as a starting point
If you're working with a distributed system you might be able to identify a system which is easier to CD than the rest. Start with that because it'll give your team some insights into the new way of working and can help you begin to break the cultural barriers.
CD is a cultural issue as much as a technical one
Roles and responsibility
Often a product owner or project manager wants to be the one who is in charge of releasing.
There are circumstances where exposing features to users should be controlled by a non-technical member of your team, but this can be managed with feature toggles.
But the copying of code from one computer to another is the responsibility of the developers on the team. After all we are the ones who are responsible for making sure the system works. It is a technical concern, not a business one.
What do QAs do now?
CD is actually liberating for QAs
- Rather than spending their time manually testing poorly tested systems they can now focus on a more holistic view of the system, trying to facilitate an environment for CD so that the whole team can be confident things are working working
- QAs spend more effort helping developers define what needs to be tested and monitored for a story to be written.
- More time for exploratory testing
Re-evaluate your tolerance for defects
Lots of companies think they cannot have any defects and will spend a lot of time and effort on complicated, time consuming (and therefore expensive) processes to try and stop them.
But think about the cost of all this? If you push a change to production that isn't covered by tests, perhaps a CSS change; consider if it's really catastrophic if there's a small visual fault for some browsers
Maybe it is, in which case there are techniques to test specifically for this too.
Each release you do with CD will have the following qualities
- Plenty of tests
- Good monitoring
- Small scope
- Still "fresh" in the developer's mind
So in my experience fixing anything that falls through the cracks is easy. It's much less complicated than trying to look through 2 week's worth of git history.
I would recommend in most cases not rolling back (unless it's really bad), but just fixing the issue and releasing it. Rollback is sometimes not an option anyway (e.g database migrations) so the fact that your system is geared to releasing quickly is actually a real strength of CD.
Other quick tips
- Fast feedback loops are key. Measure your
git push to live time and keep it low.
- If things are getting slow re-evaluate your end-to-end tests. If you removed a test or refactored it to be a unit test, would you be any less confident? If not, then refactor away
- You may need to invest some time in making your components more testable to avoid writing lots of slow end-to-end tests. This is a good thing.
- Feature toggles are a useful tool but can become messy, keep an eye on the complexity
- Celebrate it. Moving from one release every 2 weeks to 50 a day feels great.
This has been a small introduction to CD, it's a huge topic with plenty of resources to investigate.
Continuous delivery is a big effort both technically and culturally but it pays off massively.
I have worked on distributed systems with over 30 separately deployable components with CD and it has been less stressful than another project I've worked on that had just a handful of systems but a ton of process and ceremony.
Being able to release software when it's written puts a higher emphasis on quality and reduces risk for the team. It also forces you into using agile best practices like testable, small, independently releasable user stories.
Maybe most importantly, it demands that you understand your system and that you're not deploying to production and crossing your fingers. That feels more professional to me.