Sunday, July 2, 2017

[Theory] Software craftsmanship Part 3 - Deployment pipeline

Part 1 - Parallel vs united sequential vision
Part 2 - Microservices
Part 3 - Deployment pipeline
In this post I am going to describe common techniques for the deployment process.

Multi-user development leads to a problem of individual errors. Some developers say the phrase "But it works on my machine", although it does not work on another computer.
To overcome this problem we can create a mythical independent developer, that will build the project and run unit tests. This mythical developer usually presented as Continuous Integration server, i.e. a robot that runs build and tests on every update of a common code base. In case the build is failed or unit test runner shoes the red flag CI server notifies the team about the problem.

Difference between environments 

Any difference between applications leads to a bunch of subtle bugs. There is a need to have one version of the application for the development environment, at the same time we need a small modification of the application for production environment. Let us write this as the contradiction:
Contradiction (1)
Application should be the same and application should be different.

Solution (1)
Divide application on two blocks: one part is immutable and the other is mutable.

The application can be divided in two parts: immutable code binaries and mutable configuration. Multiple combinations can produce different versions of the same application: [immutable app binaries] + [development config] = dev app; while the same 
[immutable app binaries] + [production config] = prod app. This approach allows us to eliminate the bugs that can occur between different binaries for different environments, for example hard coded strings (#ifdef DEBUG). The only difference exists in a configuration file (database connections, log level and so on).

Thus we can archive our application binaries into some immutable package called artefact, e.g. or These artefacts can be placed to a common repository for a later usage.


There are two main rules for testing:
1) Automate tasks as much as possible and to reduce the manual human work. Leave interesting bugs to testers and leave boring job to computers.
2) Fail fast. 
If there is a bug in the application it is better to know about it as earlier as possible. Unit tests are the fastest tests, so we should run them at first. Then go module testing and integrations testing, and lastly the slowest UI tests and end-to-end tests.

Unit tests does not require application installation and are very fast, so these tests can be run by CI tool right after the build process.

Integration and UI tests require a distinctive environment, often with its own database. Before showing the work to QA engineers we can run all these tests in a special environment for each commit to our service.


Creating environments

Any person (developer, tester, manager, stakeholder) should be able to pick the exact version of the services (artefacts), apply configuration and add the service to a standalone environment. By default the system should propose the latest artefacts for the new environment.
Each environment should be autonomous and be almost identical to production. This principal can be achieved by usage of Virtual Private Cloud (AWS VPS) or local Docker runner with multiple images. The deployments should be saved the same as production is not destroyed between deployments with its database. If there is a need in a new database, maybe even seeded with tests data, a person can create a new environment. The rest mimics production environment:
The same as blue-green deployment is done in production we can do it in the test environment. After switching to the newly deployed service we will get the next picture:
The switch can be done manually or automatically via some registry service, where every service registers its new address after the deployment.


Let me shed light on the configuration and its relationship to the application. For proper versioning and history all configuration files should be placed inside version control system near the service code. Continuous Delivery tool can apply updates to config files with secret codes. Configuration file might have different properties for external interfaces like payment settings, SMS sending setting, which are not part of the application. Another settings might be specific to the environment, for example, exact email settings for developer or tester.

During deployment a Continuous Delivery tool might remember previous settings and ask about confirmation for applied configuration patching.
 The overall deployment pipeline is presented on the next scheme:
 Happy coding! 😊

Saturday, July 1, 2017

[Theory] Software craftsmanship Part 2 - Microservices

Part 1 - Parallel vs united sequential vision
Part 2 - Microservices
Part 3 - Deployment pipeline
In the previous post I wrote about the importance of small independent projects. 
The most important rules for building micro or nano services are:

1) The service should be autonomous.
The service and the small team should follow the idea of Bounded Context. Some examples: Email sending, SMS sending, Logging, Order service, Best buy suggestion, Products review, Search, Login and registration.
2) The service should be independently deployable.
The team should be able to deploy its service whenever it is desired.The compatibility can be supported with the ideas that were discussed in the previous part. 
If, for example, email sending team decides to work with some logging library version 1.2.5, while others use fancy v.1.6.0, then they must go with their choice (v.1.2.5), even if it is a product of their own company. The team is responsible for fixing bugs and makes reasonable decisions, logging package v.1.6.0 might not be properly tested or has undesirable behaviour. 
Creating a microservice with some HTTP RPC API usually does not rise many questions. The different picture is with GUI.
UI monolith that calls some HTTP API endpoints breaks the second law. The idea about a team being able to deploy its service independently from others means that the service should contain everything it needs for properly functionality, including UI controls if it is required:

The website shell should be able to collect independent services from some registry. Notice that autonomy also means that each team should be able to apply different technologies for their services, even different UI technologies (React, Angular, Vue.js, pure HTML, bootstrap, jquery, pure js). Of course it is likely that UI will be done in one technology stack, nevertheless such possibility should exist. The same is true for other technologies (data storage type, runtime and programming language).

A new question appears: how is it possible to control such a mess of technologies? Plus companies need consistent design.

As a team we want to have consistent design across the website.
As a team we want freely modify different parts of the website.
One solution is to separate service and not combine them at all.
Solution (1)
Give each service its own microsite.
Take a look at, in addition to the main "news" site there are many sub sites: weather, education, radio, TV and others. Each sub site has its own design, purpose and can be implemented with its own technology stack and its own team.

Solution (2)
Create the UI shell service that will dynamically collect different UI parts from the services.

Today this is scarcely done, however there are some attempts:
1) Scout 24 and Thoughtworks blog post and video
2) Zalando blog post and video
3) Amazon website itself

And what is about design?
Follow strict guide rules for design, share common styles.

An excellent example of such implementation is Building a Visual Language at Aribnb.

Designers specified common guides for sizes, colours and forms. Besides, their teams reuse common blocks across different types of applications. Headers, subheaders and rows have a consistent style in the website, mobile applications and emails.

[Theory] Software craftsmanship Part 1 - Parallel vs united sequential vision

In the series of posts I am going to show important conceptual points for software development.
Part 1 - Parallel vs united sequential vision
Part 2 - Microservices
Part 3 - Deployment pipeline

The whole purpose of a software application is to provide valuable services to its consumers. These valuable services can be constructed from features.

In such a way every backwards-compatible change to the application can be one of the next points:
1) adding a new feature (technical or business request)
2) fixing a bug for the previously added feature

If the modification is not done in a backwards-compatible manner then we can think about the software as a new application. This idea is supported by semantic versioningMAJOR.MINOR.PATCH

The software from a consumer point of view is seen as a presentation of parallel and independent feature blocks:
On the other side coding corresponds to text, and text updating is a sequential operation: some symbols are replaced, some are added, removed or just moved across the text. In addition to this the code from the small "Feature a" often intersects with the small "Feature b" without having strong separation. Thus coding leads to the united and sequential picture of a software, where every bit is appended to the end of a common block: We can think of it as a process of book writing, with all the chapters and the references between its different parts. The idea of a simultaneous text editing by several authors is a source of lots of problems. Let us specify our first problem

Contradiction (1)
As a writer I would like to change a piece of a text without affecting other editors of the same text.

Contradiction (1 strengthen)
As a writer I need to apply changes to a text that is no longer exist (moved, changed or removed).
A text should be changed and should not be changed.

The simple solution is to apply golden Divide and Conquer method and totally eliminate simultaneous editing problem.

Solution (1)
Divide text blocks in such a manner that every writer has its own independent part. 
Books are often divided by chapters where there is no more than one author per chapter.
Applications can be divided in the same manner with a developer being assigned to support a specific software library. Removing the multi-user editing mode can be also done by usage of small services, for example, when one developer is responsible for a bunch of related cloud functions.

Files renaming, moving files across the project, spacious code refactoring, all these steps lead to later painful code merging. If Solution 1 is not accessible then there are some weak solutions, which do not solve the problem but minimize its pain.

Weak solution (1a)
Divide people in small groups, so the teams can provide autonomous and independent deployable services. This approach is called usage of Microservices. We can minimize the impact of a huge sharable code base by presenting small groups of people, which manage small independent micro projects with small code bases. Also the team becomes primarily responsible for the service business logic and will have deeper understanding of the processes than large teams do.

As long as a code merging is inevitable we also can reduce the pain of conflicts resolving by minimizing applied changes.
Weak solution (1b)
The team can make a rule: to commit only small changes to sharable code base and do it often, at least every day. Thus the developers will have only small merge conflicts, which must be simple to resolve.

This weak solution actually leads us to a new contradiction
Contradiction (2)
As a developer sometimes I work on a large task during several days. 
As a developer I have to make small commits to the common code base at least every day, so the other developers can simply merge my changes.
In other words: changes should be large and be applied rarely, changes should be small and be applied often.

Solution (2)
Wrap large changes in a switchable toggle. Make this toggle 'off' by default. Commit small changes to the common code base for the disabled feature. Simply enable the feature when it is ready.
This is a very common principle in software building, which is called Feature Toggle. Now a developer is able to do small and often changes to a common code base.

Sometimes there is a need to do a breaking change that affects a large group of people. For example, a change in the logging library that is used across the whole application; or take, for instance, a field renaming in a live database. If we check the problem carefully then the contradiction is the same as the contradiction (1):
Contradiction (3)
A part of something should be changed and at the same time should not be changed.

One of the solution is to make long-lasting switch to a new breakable change. I call this principle 3-phase switching:
Solution (3)
1) Create the enhanced version of the block. If there is a sharable state (database, cache, external API) then the new version should also support the old state presentation. 
2) Allow or force clients to switch to the new version of the block during some time.

3) Stay with the new version and remove the old block; or stay with the old solution (rollback).
For example, if a developer makes a breakable change in the logging library like method renaming or drastically change method parameters, then he or she can create a new class (a new library) based on the old one. So there will be two blocks: the old one and the new one. The consumers can slowly switch to the new library.

The concept of 3-phase switching is very powerful and is used widely in different aspects:
  • Canary release, which is a process of rolling up a new software to a small group of users and checking whether it works as expected, thus not affecting all customers by a big bang of failure. Rolling up is slowly moving: [10% of customers redirected to the new software, 90% to the old], [20% to new, 80% to the old], ..., [100% to the new software].
  • Beta release or a release for private groups, beta testers. 
  • A-B testing
  • Green/Blue deployment
  • Feature toggle (related)
  • Software package lifecycle: npm, pypi, nuget, ruby gems, php composer and others.

Let's take another example with a table field renaming in a relational database. The process might look as:

1) Add the new field with a proper name to the table.
2) Create a new endpoint for clients (API) in such a manner that data is written to both: the new and the old field. Wait for clients being switched to the new version.
Due the fact that the new API writes data in compatible way to the old field, clients that are using the old field are not affected at all.

3) When there are no more clients using the old API then simply remove it with the old field. Finally there will be only new renamed field.