[Theory] Software craftsmanship Part 3 - Deployment pipeline

Part 1 - Parallel vs united sequential vision
Part 2 - Microservices
Part 3 - Deployment pipeline
----------------------------

In this post I am going to describe common techniques for the deployment process.

Multi-user development leads to a problem of individual errors. Some developers say the phrase "But it works on my machine", although it does not work on another computer.
To overcome this problem we can create a mythical independent developer, that will build the project and run unit tests. This mythical developer usually presented as Continuous Integration server, i.e. a robot that runs build and tests on every update of a common code base. In case the build is failed or unit test runner shoes the red flag CI server notifies the team about the problem.

Difference between environments 

Any difference between applications leads to a bunch of subtle bugs. There is a need to have one version of the application for the development environment, at the same time we need a small modification of the application for production environment. Let us write this as the contradiction:
Contradiction (1)
Application should be the same and application should be different.

Solution (1)
Divide application on two blocks: one part is immutable and the other is mutable.

The application can be divided in two parts: immutable code binaries and mutable configuration. Multiple combinations can produce different versions of the same application: [immutable app binaries] + [development config] = dev app; while the same [immutable app binaries] + [production config] = prod app. This approach allows us to eliminate the bugs that can occur between different binaries for different environments, for example hard coded strings (#ifdef DEBUG). The only difference exists in a configuration file (database connections, log level and so on).

Thus we can archive our application binaries into some immutable package called artefact, e.g. projectA.v.1.zip or serviceB.v3.zip. These artefacts can be placed to a common repository for a later usage.

Testing

There are two main rules for testing:
1) Automate tasks as much as possible and to reduce the manual human work. Leave interesting bugs to testers and leave boring job to computers.
2) Fail fast. 
If there is a bug in the application it is better to know about it as earlier as possible. Unit tests are the fastest tests, so we should run them at first. Then go module testing and integrations testing, and lastly the slowest UI tests and end-to-end tests.
Unit tests does not require application installation and are very fast, so these tests can be run by CI tool right after the build process.

Integration and UI tests require a distinctive environment, often with its own database. Before showing the work to QA engineers we can run all these tests in a special environment for each commit to our service.

 

Creating environments

Any person (developer, tester, manager, stakeholder) should be able to pick the exact version of the services (artefacts), apply configuration and add the service to a standalone environment. By default the system should propose the latest artefacts for the new environment.
Each environment should be autonomous and be almost identical to production. This principal can be achieved by usage of Virtual Private Cloud (AWS VPS) or local Docker runner with multiple images. The deployments should be saved the same as production is not destroyed between deployments with its database. If there is a need in a new database, maybe even seeded with tests data, a person can create a new environment. The rest mimics production environment:
The same as blue-green deployment is done in production we can do it in the test environment. After switching to the newly deployed service we will get the next picture:
The switch can be done manually or automatically via some registry service, where every service registers its new address after the deployment.

Configuration

Let me shed light on the configuration and its relationship to the application. For proper versioning and history all configuration files should be placed inside version control system near the service code. Continuous Delivery tool can apply updates to config files with secret codes. Configuration file might have different properties for external interfaces like payment settings, SMS sending setting, which are not part of the application. Another settings might be specific to the environment, for example, exact email settings for developer or tester.

During deployment a Continuous Delivery tool might remember previous settings and ask about confirmation for applied configuration patching.
 The overall deployment pipeline is presented on the next scheme:
 Happy coding! 😊

[Theory] Software craftsmanship Part 2 - Microservices

Part 1 - Parallel vs united sequential vision
Part 2 - Microservices
Part 3 - Deployment pipeline
----------------------------

In the previous post I wrote about the importance of small independent projects. 

The most important rules for building micro or nano services are:

1) The service should be autonomous.
The service and the small team should follow the idea of Bounded Context. Some examples: Email sending, SMS sending, Logging, Order service, Best buy suggestion, Products review, Search, Login and registration.

2) The service should be independently deployable.

The team should be able to deploy its service whenever it is desired.The compatibility can be supported with the ideas that were discussed in the previous part. 

If, for example, email sending team decides to work with some logging library version 1.2.5, while others use fancy v.1.6.0, then they must go with their choice (v.1.2.5), even if it is a product of their own company. The team is responsible for fixing bugs and makes reasonable decisions, logging package v.1.6.0 might not be properly tested or has undesirable behaviour. 

Creating a microservice with some HTTP RPC API usually does not rise many questions. The different picture is with GUI.

UI monolith that calls some HTTP API endpoints breaks the second law. The idea about a team being able to deploy its service independently from others means that the service should contain everything it needs for properly functionality, including UI controls if it is required:

The website shell should be able to collect independent services from some registry. Notice that autonomy also means that each team should be able to apply different technologies for their services, even different UI technologies (React, Angular, Vue.js, pure HTML, bootstrap, jquery, pure js). Of course it is likely that UI will be done in one technology stack, nevertheless such possibility should exist. The same is true for other technologies (data storage type, runtime and programming language).

A new question appears: how is it possible to control such a mess of technologies? Plus companies need consistent design.

Contradiction
As a team we want to have consistent design across the website.

As a team we want freely modify different parts of the website.

One solution is to separate service and not combine them at all.

Solution (1)

Give each service its own microsite.

Take a look at bbc.co.uk, in addition to the main "news" site there are many sub sites: weather, education, radio, TV and others. Each sub site has its own design, purpose and can be implemented with its own technology stack and its own team.

Solution (2)

Create the UI shell service that will dynamically collect different UI parts from the services.

Today this is scarcely done, however there are some attempts:
1) Scout 24 and Thoughtworks blog post and video
2) Zalando blog post and video
3) Amazon website itself

And what is about design?
Follow strict guide rules for design, share common styles.

An excellent example of such implementation is Building a Visual Language at Aribnb.
Designers specified common guides for sizes, colours and forms. Besides, their teams reuse common blocks across different types of applications. Headers, subheaders and rows have a consistent style in the website, mobile applications and emails.

[Theory] Software craftsmanship Part 1 - Parallel vs united sequential vision

In the series of posts I am going to show important conceptual points for software development.
Part 1 - Parallel vs united sequential vision
Part 2 - Microservices
Part 3 - Deployment pipeline
---------------------------

The whole purpose of a software application is to provide valuable services to its consumers. These valuable services can be constructed from features.

In such a way every backwards-compatible change to the application can be one of the next points:
1) adding a new feature (technical or business request)
2) fixing a bug for the previously added feature

If the modification is not done in a backwards-compatible manner then we can think about the software as a new application. This idea is supported by semantic versioning: MAJOR.MINOR.PATCH

The software from a consumer point of view is seen as a presentation of parallel and independent feature blocks:
On the other side coding corresponds to text, and text updating is a sequential operation: some symbols are replaced, some are added, removed or just moved across the text. In addition to this the code from the small "Feature a" often intersects with the small "Feature b" without having strong separation. Thus coding leads to the united and sequential picture of a software, where every bit is appended to the end of a common block: We can think of it as a process of book writing, with all the chapters and the references between its different parts. The idea of a simultaneous text editing by several authors is a source of lots of problems. Let us specify our first problem

Contradiction (1)
As a writer I would like to change a piece of a text without affecting other editors of the same text.

Contradiction (1 strengthen)
As a writer I need to apply changes to a text that is no longer exist (moved, changed or removed).
A text should be changed and should not be changed.

The simple solution is to apply golden Divide and Conquer method and totally eliminate simultaneous editing problem.

Solution (1)
Divide text blocks in such a manner that every writer has its own independent part. 
Books are often divided by chapters where there is no more than one author per chapter.
Applications can be divided in the same manner with a developer being assigned to support a specific software library. Removing the multi-user editing mode can be also done by usage of small services, for example, when one developer is responsible for a bunch of related cloud functions.



Files renaming, moving files across the project, spacious code refactoring, all these steps lead to later painful code merging. If Solution 1 is not accessible then there are some weak solutions, which do not solve the problem but minimize its pain.

Weak solution (1a)
Divide people in small groups, so the teams can provide autonomous and independent deployable services. This approach is called usage of Microservices. We can minimize the impact of a huge sharable code base by presenting small groups of people, which manage small independent micro projects with small code bases. Also the team becomes primarily responsible for the service business logic and will have deeper understanding of the processes than large teams do.

As long as a code merging is inevitable we also can reduce the pain of conflicts resolving by minimizing applied changes.
Weak solution (1b)
The team can make a rule: to commit only small changes to sharable code base and do it often, at least every day. Thus the developers will have only small merge conflicts, which must be simple to resolve.

This weak solution actually leads us to a new contradiction
Contradiction (2)
As a developer sometimes I work on a large task during several days. 
As a developer I have to make small commits to the common code base at least every day, so the other developers can simply merge my changes.
In other words: changes should be large and be applied rarely, changes should be small and be applied often.

Solution (2)
Wrap large changes in a switchable toggle. Make this toggle 'off' by default. Commit small changes to the common code base for the disabled feature. Simply enable the feature when it is ready.
This is a very common principle in software building, which is called Feature Toggle. Now a developer is able to do small and often changes to a common code base.



Sometimes there is a need to do a breaking change that affects a large group of people. For example, a change in the logging library that is used across the whole application; or take, for instance, a field renaming in a live database. If we check the problem carefully then the contradiction is the same as the contradiction (1):
Contradiction (3)
A part of something should be changed and at the same time should not be changed.

One of the solution is to make long-lasting switch to a new breakable change. I call this principle 3-phase switching:
Solution (3)
1) Create the enhanced version of the block. If there is a sharable state (database, cache, external API) then the new version should also support the old state presentation. 
2) Allow or force clients to switch to the new version of the block during some time.
3) Stay with the new version and remove the old block; or stay with the old solution (rollback).
For example, if a developer makes a breakable change in the logging library like method renaming or drastically change method parameters, then he or she can create a new class (a new library) based on the old one. So there will be two blocks: the old one and the new one. The consumers can slowly switch to the new library.

The concept of 3-phase switching is very powerful and is used widely in different aspects:

• Canary release, which is a process of rolling up a new software to a small group of users and checking whether it works as expected, thus not affecting all customers by a big bang of failure. Rolling up is slowly moving: [10% of customers redirected to the new software, 90% to the old], [20% to new, 80% to the old], ..., [100% to the new software].
• Beta release or a release for private groups, beta testers. 
• A-B testing
• Green/Blue deployment
• Feature toggle (related)
• Software package lifecycle: npm, pypi, nuget, ruby gems, php composer and others.

Let's take another example with a table field renaming in a relational database. The process might look as:

1) Add the new field with a proper name to the table.
2) Create a new endpoint for clients (API) in such a manner that data is written to both: the new and the old field. Wait for clients being switched to the new version.
Due the fact that the new API writes data in compatible way to the old field, clients that are using the old field are not affected at all.

3) When there are no more clients using the old API then simply remove it with the old field. Finally there will be only new renamed field.

Octopus meets Magento on Windows and Linux

One project that I am working with consists of two parts: 

* Backend job runners, written in C#.NET Full Windows Framework and 

* Frontend web site based on Magento CMS, which is written in PHP

In our team there was already the Windows VPS machine for .NET applications. Also we use Continuous Deployment tool Octopus Deploy, which requires Octopus Server to be installed on Windows OS. 

Magento is working by default with LAMP (Linux Apache MySql PHP) infrastructure and nothing is said about Windows. Some guys use WAMP (Windows Apache MySql PHP) infrastructure, but mixing Apache and IIS is not a good idea. Thus we decided to give a try and run Magento in Windows + PHP + IIS alongside existed .NET applications.

The first thing was to convert many Magento .htaccess files (Apache) to web.config (IIS). This task itself is pretty hard and manual. IIS has auto converting feature for .htaccess => IIS rules web.config, but most of the time IIS failed to do it properly.

The next thing is to make a package containing the whole website for Octopus. We use TeamCity for CI and decided to pack, as usual for .NET, everything in a NuGet package. The trick is that NuPack tool works excellent with .NET dlls, but we finally found how to pack even PHP and .htaccess files.

To do so create .nuspec file in the root of the project:

<?xml version="1.0"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
    <!-- Commands: https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Clients/NuGet.CommandLine/Commands/PackCommand.cs -->
    <!-- Schema: https://docs.microsoft.com/en-gb/nuget/schema/nuspec -->
    <metadata>
        <id>MyProject</id>
        <title>Project Title</title>
        <version>1.0.0</version>
        <description>This is a Magento project</description>
        <authors>Your team</authores>
    </metadata>
    <files>
        <file src="**" exclude=".git\**" />
    </files> 

</package> 

The most import thing here is to make the files filter src="**", which includes all kind of types and extensions. The fields "id, version, description, authors" are required.

Then run nuget pack command pointing to our .nuspec file.

nuget pack .nuspec -NoPackageAnalysis -NoDefaultExcludes

 NoPackageAnalysis speeds up the packing, NoDefaultExcludes protects from nuget throwing out special non .NET world files. This can be simply configured in TeamCity NuPack Runner.
The process of packaging takes about 3 minutes as Nuget provides only one level of compression and the project is big (about 40 000 files).
After that you will have your MyProject.nuget file that can be pushed to Octopus (I will skip the details). Another solution is use custom NodeJS package instead of NuGet package.

The next problem was figured out with the help of Octopus. Linux terminal usually swallows the errors from magento setup tool when you run 

php -dmemory_limit=2G bin/magento setup:static-content:deploy

Better to run verbose output '-v' for showing the errors

php -dmemory_limit=2G bin/magento setup:static-content:deploy -v

Gladly Octopus understood the presence of errors even without "-v" argument and refused to continue the deployment. The problem was in the popular Magento theme that we use Sm Market. Bootstrap source code contained import .less rules with wrong file names in Magento/app/design/frontend/Sm/market/web/css/source/bootstrap/_theme.less,
fix it by pointing to the proper file names:

@import "_variables";
@import "_mixins.less";

After a lot of fighting and trying to run Magento in IIS, eventually it was working with (still) a lot of errors and problems due the url rewriting rules and user rights. The time spent on this became too big, so the decision was to move Magento to its Alma mater: Linux machine and Apache. 
However we would like to continue working with awesome Octopus tool. Recently their team added support for Linux OS as tentacles (the server still requires Windows). This was an excellent case for us to give it a try.

Linux

The first step is to follow the guidance Octopus SSH Taget and install mono. Plus mysql, apache, php, etc for Magento.

Then create user "octopus":
sudo useradd -m octopus echo octopus:<the-password-you-want> | sudo chpasswd

We decided not to create a user per app in UAT machine and to run web sites by the octopus user. To do so just add the octopus user to the apache group:

sudo usermod -g apache octopus

The next point is to give octopus the ability to work with apachectl command (CentOS) without entering password. Create octopus section in sudoers:
sudo visudo -f /etc/sudoers.d/octopus 


and put data:

#Allow octopus to control apache 
octopus     ALL=(ALL) NOPASSWD:/usr/sbin/apachectl
Defaults!/usr/sbin/apachectl !requiretty

Thus octopus will be able to run sudo apachectl restart without entering the password.

Apache

Now the time to set up Apache.
Open /etc/httpd/conf/httpd.conf and comment all Directory settings except the global one restrictions.

If you have Include conf.d/*.conf row then you can create  /etc/httpd/conf.d/magento.conf,
otherwise just put the next lines to etc/httpd/conf/httpd.conf

Include /home/octopus/.octopus/Applications/OctopusServer/Development/YourProjectName/*.conf
Include /home/octopus/.octopus/Applications/OctopusServer/UAT/YourProjectName/*.conf

It means that config files from the above folders will be used by Apache. If you have Apache 2.4+ then it is better to use IncludeOptional directive, which does not throw error if the path does not exist.

The configuration files will be created one per project and will update Apache bindings in the next way:

<VirtualHost *:80>
  ServerName www.sitename.com 
  DocumentRoot /home/octopus/.octopus/Applications/OctopusServer/Development/YourProjectName/1.0.20/
  <Directory /home/octopus/.octopus/Applications/OctopusServer/Development/YourProjectName/1.0.20/> 

       Options Indexes FollowSymLinks MultiViews 
       AllowOverride All 
       Order allow,deny 
       Allow from all 
  </Directory>
</VirtualHost>

If you do not like this approach you can use pm2 tool for starting and stoping processes.

Octopus

Set up Octopus steps

We only need some scripts to run in bash.
(Magento installation scripts with Apache config update are placed at the end of this article)

Install and Upgrade Magento step:

parameters are taken from Octopus Variables:

--magento-directory="#{Octopus.Action[Deploy to Apache].Output.Package.InstallationDirectoryPath}" 
--admin-firstname="#{AdminFirstname}" --admin-lastname="#{AdminLastname}" 
--admin-email="#{AdminEmail}" --admin-user="#{AdminUser}" 
--admin-password="#{AdminPassword}" --base-url="#{BaseUrl}" 
--backend-frontname="#{BackendFrontname}" --db-name="#{DbName}" 
--db-user="#{DbUser}" --db-password="#{DbPassword}"

Update Apache Bindings step:

parameters are also taken from Octopus Variables:

--magento-directory="#{Octopus.Action[Deploy to Apache].Output.Package.InstallationDirectoryPath}" 
--server-name="#{WebsiteName}"

And that is it.
The deploy process works like a charm.


MagentoDeploy.sh


MagentoUpdateApacheBindings.sh

Can we stop date formatting mess?

What does the next date exactly present?

Most people from over the world will say that it is the 2nd of November 2017. But it is not true for American and Albanian people.

Airbnb for US culture will show the popup with the 11th of February 2017.

This becomes more entangled when we get Russian, Ukrainian and Kazakstan format DD.MM.YYYY, which is often shortened to DD.MM.YY. For example, the 11th of February 2017 is usually written as 11.02.17.

To make internationalisation more complex some countries (Chine, Lithuania, South Africa) use YYYY-MM-DD or YYYY.MM.DD, which sometimes people like to shorten to YY.MM.DD, and our previous example becomes 17.02.11.

If you think that nothing can be worth, there are countries (Germany, Canada, Sweden, Philippines), which apply different date formats depending on a context.

Requirements

Taking into the account all previous points and preventing likelihood problems the solution of the problem should be:

1) Visible immediately as a static text without any data manipulation (no need to open a popup date editor)

2) A day, a month and a year should be distinguishable from each other

3) The distinguish signs should have the colour of the text and be readable after a monochrome print, e.g. in books and newspapers.

4) The distinguish signs should be simple and built from UTF symbols.

Proposal of a solution

Let us start our visual identification from the shortest amount of time - a day, to the most lasting - a year. The simplest visual effect, that is also very convenient to program or use in draw editors, is underlining. Underlining also works good with all popular charset encodings.

A day gets full width underline = DD;

A month gets a half width underline if the year part is short (YY) = MM; otherwise month has no underline = MM;

A year has no underline = YYYY or YY;

Examples

Our initial confusing date presentation now becomes absolutely clear:

Some people like to trim zeros and write, for example, 7.9.2017. Even this date can be transformed to an obvious presentation: 7.9.2017.

More complex 11.10.12 YY.MM.DD can be transformed to 11.10.12 or to 12.10.11, which is actually the same as  12.10.2012 and 10/12/2012.

And now the bonus for developers and almost anybody who process much data. A common SQL result nowadays says nothing about a date format:

while after our small fix the data is a pleasure to read:

Table of different date formats 

Date	Meaning
2/11/2017	The 11th of February 2017
2/11/2017	The 2nd of November 2017
10/3 or 10/03	The 10th of March
01/17	January 2017
2017-12-10	The 10th of December 2017
05/04/03	The 5th of March 2003
05/04/03	The 3rd of March 2005
2/3/2020	The 3rd of February 2020

Happy coding!

The evolution of web development

Custom HTML Attributes Are Evil

HTML is itself a static content, and many developers have used the specification <!DOCTYPE >, which points out that it is merely a document.
The dynamic part is represented by JavaScript language. And here arose the problem.

Looking how various programming languages are used to build UI reveals that the language itself is a part of UI blocks. For example, Windows Presentation Framework uses XAML with C# controls:

<Grid Background="#555555" Width="400" Height="300">
    <Button Click="OnButtonClick">Press me</Button>

 </Grid>

Here will be created an instance of Grid class and an instance of Button class. Also it is possible to write logic whatever we like inside custom controls as they are just usual classes:

class Grid {
     public Color Background {get; set;}
     // methods and other UI logic
}

The same is true for Java with Android layouts, and many other languages and platforms.

HTML initially does not provide such idea, it does not link <html-element> to a JavaScript instance. To enhance it we can simply declare HTML elements as objects of JavaScript types:
  class div {
     public string style {get;set;}
     // methods and other UI logic

     render: function(){
         // return real html '<div>'+content+'</div>'
     }
  }
  class MyWindowPopup {
     public boolean fullScreen {get;set;}
     public boolean visible {get;set;}
     // methods and other UI logic
  }

Then it becomes possible to build pages and components in a common way:
   <MyWindowPopup visible="true" fullScreen="false">
     <div>Popup text</div>
   </MyWindowPopup>
Moreover we can advance some property types, e.g. use string[] array instead of string for class property <div class="typeof string[]">.

Then a browser or a server (NodeJS) is simply converts JavaScript to HTML and shows layout to a user. This approach took Facebook team with React and its JSX style.
---------------------------------------

Many other teams try to overcome HTML static nature by putting custom attributes, which seems very strange and inconvenient for me. Angular, Vue.js, Aurelia.js and many other JS frameworks use logic with custom marks (*ngIf, v-if, bind.click):

<div *ngIf="currentHero">Hello, {{currentHero.firstName}}</div>
<div *ngFor="let hero of heroes">{{hero.fullName}}</div>

JavaScript Monopoly

Nowadays JavaScript is the king of UI and there is no alternative.
I think every team that works with Web tried to create a converter from their languages to JavaScript. However this approach produces massive bunch of problems due the nature of languages.

Google created GWT where Java is converter to JavaScript and developers can write kind of UI logic in Java. Also Google tried to apply usage of Dart language in modern browsers and again created conversion to JavaScript.
Microsoft uses aspx controls that can be written in .NET compatible language, Razor views, and TagHelpers in Asp Core, which all finally are converted to JavaScript and HTML. You can find same idea for Elm, Python, PHP and other languages.

The problem is that JavaScript and C#, JavaScript and Java, JavaScript and [your favourite language] are very different beasts.

Javascript is a prototype-based language with first-class functions. There is no even idea of an object instance of a class (class declaration leads again to a function). There are just six types in JavaScript: Object, Number, String, Boolean, Null, and Undefined, plus some object helpers like RegEx. There is no even notion of Integer or Decimal type because JavaScript Number only holds double-precision 64-bit format value. Run in a JS console "0.1 + 0.3" and see what you get, while with Decimal types in C# the result will be "0.4".

Thus, developers restrict their language of choice too much and loose the power of JavaScript. In addition to these points most of the time they loose highlighting and intellisense while producing JavaScript and HTML as a string, e.g. in Asp Core TagHelpers:

     public class IconTextTagHelper : TagHelper
    {
        public string  Glyph { get; set; }
        [HtmlAttributeName("class")]
        public string ClassName { get; set; }

        public override void Process(TagHelperContext context, TagHelperOutput output)
        {
           string classnames = String.IsNullOrWhiteSpace(ClassName) ? "icontext" : "icontext " + ClassName;
                output.TagName = "a";
                output.TagMode = TagMode.StartTagAndEndTag;
                output.Attributes.Add("class", classnames);
                
                output.Content.SetHtmlContent(
                // String, string, string
                   $@"<span>
                          <i class=""icon fa fa-{Glyph ?? ""} icontext__icon""></i>
                          <span class=""icontext__text"">{this.Text}</span>
                      </span>");                     
        }
    }
 A way better solution is to work with JavaScript itself, or even better with TypeScript, which provides static type checking.

Breaking JavaScript Monopoly

To overcome the monopoly of JavaScript the IT community are making an attempt to specify WebAssembly, which later can be supported by browsers vendors.
The idea is very simple:
   Browsers will support agnostic language (through ASTs) that will include many features from modern languages. It will be some kind of Intermediate Language that is used in Java and .NET virtual machines nowadays. It will be faster to parse, faster to transfer, faster to run, it will be in binary format instead of text format of current JavaScript.

Sadly web development progress is very slow.
The web community still tries to build some kind of XAML with its custom components, inline styles, namespaces and modules.
It is 2016 and only now we have Flexbox, which is analogue of StackPanel in WPF for arranging elements horizontally or vertically in a single line. And what about Grid,  without a hack and whistles there is no way to do it in modern CSS, only recently W3C CSS Grid specification has passed from a Draft type to a Candidate Recommendation type.

There are so many bright minds in our realm but the overall picture is appalling.

P.S. JavaScript word is used for convenience, while ECMAScript is the proper notion in some places.

SPA

It is common nowadays to have the acronym for SPA as a Single Page Application.

Nevertheless the actual meaning is quite different, the page is not single, the page is actually shelled.

SPA approach has the same idea as in a mobile application or a thin desktop client. A user downloads a thin shell for the app: a header, a footer, a body placeholder and possibly sidebars. Then asynchronously this app updates the content for the placeholders, while footer and header is not reloaded.

From a user experience the app is working with several navigation links and many pages. So calling such shelled app layout as a one page sounds misleading.

SPA = Shelled Page Application

Application workflow strategy and Staging verbosity

One of the famous app flow strategies is DTAP, which is acronym for Development Testing Acceptance Production.

Let us consider each step in detail, but before this let us make step back and add a source control and a continuous integration step for the full picture.

The scheme represent a de-facto standard in current development process.

Source control

• holds project source files and likely provides history changes, as project branches and users control management.

Continous integration 

• a continuous integration tool checks for the source control changes, grabs source files from the source control and builds the projects;
•  runs unit tests;
•  produces artifacts (application packages) that are ready for publishing;
•  publishes the app to Dev server;
•  for the published app runs smoke tests, which check basic and the most important functionality, e.g. products are displaying and a user can log in and pay for a product;
• for the published app runs functional tests, which makes in-depth testing of the whole application pretending to be a human tester, e.g. opening Internet Explore browser, opening page with the contact form, filling contact data and submitting it, checking that email is received or was added to a fake queue.
  (Note: most of the time CI tool cannot itself run Smoke or Functional tests as there is a need to have visual APIs in the operating system, which provides ability for example browsers to be installed or other GUI apps. For these purposes special test apps are built with tools  like SelenuimHQ and are placed on a Dev server (or other suitable place with support of reach GUI), while CI tool can be installed in the cloud or a small linux server without rich GUI support. Then CI tool requests from the Smoke/Functional test app to run these tests.
• finally we can guaranty that the app is almost fully functional on the dev server, otherwise if any of the previous steps are failed the responsible person will be notified about the problem.

Then we have the following steps for the application:

Dev environment

• holds automatically tested application;

Test environment

• if you have testers in your team this place is for them;
  the app is tested by testers in isolation from other steps
• tester makes subtle tests as the app is already tested automatically;
  if there is a bug, then one of the functional test is missing, which should be added by developer in the fix process
• a CI manager can manually request CI tool to propagate the app to Test server(s) after Dev phase
• another approach is to automatically propagate the app from Dev phase to Test phase by CI tool when you have versioned app running simultaneously or several Test servers, so that testers will not be interrupted while they are providing tests

Acceptance (also known as UAT/User Acceptance Test) environment

• holds application that waits for a testing from a stakeholder/ a customer;
• if there is no tester in your team you can propagate (manually or automatically) the app from Dev to UAT phase with CI tool
• A stakeholder makes tests and whether accepts or rejects the changes

Note: environment can be presented by different amount of servers and machines. For example Dev, Test and UAT environment can be hold on a one server, or in other case Test environment can be hold on N-servers.

STAGING

Although DTAP model is obviously clear there is another step called Staging. I've phased a lot of confusion with Staging step among developers as there is no single definition for it.
Some people argue that Staging is another name for UAT step.  Other people go further and combine Test step with UAT step and call this combination as Staging. Any of this is definitely valid while you have one concept across your whole team.
Nevertheless personally I prefer to have the notion of Staging as a pre-production step. What does it mean? Let us look at the image from Steve's blog post about staging in Azure.

Staging is an additional step before publishing an app to a production environment.

Staging

• is propagated from UAT phase after a stakeholder approval
• holds the app ready to go live in production;
• has connection settings (database, email server, and so on) the same as in production 
• CI tool runs Smoke tests to verify critical functionality
• if needed a stakeholder or a developer can manually verify some special points

After all theses steps are done Staging can be simply swapped with a Production environment, for example for a website by changing routes in a hosting tool (IIS, Nginx, Apache). Thus staging becomes production, while production becomes stale staging, which can be removed or used as a backup. This approach is known as Blue Green deployment.

Custom unstructured data types and types' reuse

In 1972 three well-known computer scientists wrote a book Structured Programming, where they mentioned about custom unstructured types:

All structured data must in the last analysis be built up from unstructured components, belonging to a primitive or unstructured types. Some of these unstructured types (for example, reals and integers) may be taken as given by programming language or the hardware of the computer. Although these primitive types are theoretically adequate for all purposes, here are strong practical reasons for encouraging a programmer to define his own unstructured types, both to clarify his intentions about the potential range of values of a variable, and the interpretation of each such value; and to permit subsequent design of an efficient representation.

(...) Such a type is said to be an enumeration, and we suggest a standard notation for the name of the type and associating a name with each of its alternative values.

type suit = (club, diamond, heart, spade);

          (...)

type year = 1900 .. 1960; 

type coordinate = 0 .. 1023;

(...) We therefore introduce the convention that a .. b stands for the inline range of values between a and b inclusive. This is known as subrange of the type to which a and b belong, (...)

Ole-Johan Dahl, Edsger W. Dijkstra, C.A.R. Hoare, Structured Programming, A.P.I.C. Studies in Data Processing, No. 8, 1972, p.97

Some of the languages indeed support subrange types, e.g. Ada and Dephi. Nevertheless, modern popular languages like Java and C# do not support unstructured custom subrange types.

In addition to this there is a huge misconception about a type declaration and its actual meaning. For instance, an Array accepts an Integer as index argument with a range [-2^31, 2^31], so we declare support for -1,-2, ... index values. At the same time only non-negative integers are accepted, i.e. [0, 2^31]. For an honest and clear code it should be an Array[NonnegativeInteger].

Moreover, commonly used structured and unstructured types does not limit to numbers. Nowadays an email is always presented as String:
function SendEmail(String email, String emailBody){ ... }

but we actually mean
function SendEmail(EmailAddress email, String emailBody){ ... }

Making an assertion that an instance of the class/structure EmailAddress is always valid we do not need to check the string every time with IsEmail() helper or an attribute of DataAnnotations. It can have a function EmailAddress TryConvert(String email) for convenient conversion from string.

Other common places for apparent improvements:

• Name checking of a String for a Firstname and a Lastname across whole solution with the same restrictions, probably with no more than 256 special characters and at least 1 character, can be changed to what a programmer really means: class/structure Name.
• class Description with common rules for contact messages, order's comments, feedback messages, etc.
• Measures: degrees, coordinates, weight, etc.
• Date and time Integers up to 2147483648 (2^31) for a year, a month, a day, an hour and a minute look inconvenient for real use and most of the systems after all will throw an exception by taking large numbers for these types.
• Money There is no a banknote with minus $100 value, but there can be a notion of 100$ debt.

With the help of Design by Contracts most of the checks can be done during compilation time.
For interoperability with other systems custom types should be built up from primitive and simple types, like integers and strings, and be convertible from these types.

Computers in USSR | Kazan

Computer science and development evolved in parallel in England, USA and USSR. There are several cities that took major role in the first Soviet computer era: Kiev and Kharkiv in Ukraine, Minsk in Belarus, Moscow, Penza and Kazan in Russia. Due the World War II all vital factories and universities were moved from the west (Kiev, Minsk, Saint-Petersburg, Moscow) to the east, including Kazan city. Therefore, Kazan became one of the major technical and scientific centre in the Soviet Union, and the city was chosen for building the first serial computers in USSR.

In 1953 the Communist Party approved a request to build a computer factory in Kazan to produce electronic devices for the military agency. The factory was ready for large manufacturing in 1959 and the government asked to build two M-20 computers.  The architect was S.A. Lebedev, who already had experience in designing and building computers in Kiev and Moscow, however, they were mere specimens and the newborn task aimed to build the series of computers.

M-20

M-20 is the first serial computer in USSR. It was built from vacuum tubes (valves) and the first two machines occupied the whole level of the factory. The vacuum tubes require colossal energy and emit a lot of heat. You can find the parts of this excellent machine only in the Kazan computer museum, which is based and supported by Margarita Badrutdinova.  She has been working with computers in the factory from its first days.

View of M-20 computer, 1962 

Parts of M-20 computer,

top - keyboard, bottom - 2 bit triggers from vacuum tubes

In the end of 1960 the computers were ready, successfully tested and moved to the customer, which was the military agency. The engineers did not know what kind of calculations and tests they were successfully performed. As history showed the second M-20 machine was used to calculate the path for the space rocket and this same computer was used to launch the first human space journey with Gagarin in the 12th of April, 1961!
M-20 is named after the 20 000 operations per second, during the five years there were built 63 sets of the machine.

Setun

The next machine "Setun" was developed in Moscow by Nikolay Brusentsov and build in Kazan in 1961. The machine got its name by the river "Setun" near Moscow State University. It had three-state logic, which was quite novel at that time and still is a unique serial implementation in the world. Instead of two-state logic "true-false" and bits "1 and 0" Setun had operations with trinary states "true-false-maybe" and stored trits "1 0 -1" in the memory. Some of the benefits of three-state logic are: the program size and data is 1.6 times smaller than two-bits representation, there is no need to take special bit for the number sign, additional branch for the if-else switch.

Setun computer

Parts of Setun computer,

at the bottom in the left side is a small core item - ferret-diode cell,

on the right side is a magnet drum for external memory. 

Almost every part of products was built by the manufacture and often required novelty and tackling difficult tasks. Overall there were produced 47 sets of these trinary machines during 1961-1965 in the Kazan computer factory.

Nairi

Another computer was designed in Yerevan by G. Ovsepyan in 1964. He had a big fight for this machine with the heads of Soviet departments and finally made excellent and robust computer. It was produced in the Kazan factory during 1965-1970.  Nairi was very successful product and very popular in the Soviet Union, the factory engineers made improvements and built overall 500 machines closely working with Ovsepyan's team. Ironically, many Yerevan scientists and the Comunnist party heads stole Ovsepyan's achievements and got awards. He had not been mentioned for decades and some of the cheaters still have political power and are afraid to mention his name.

Nairi computer

Parts of Nairi computer,

in the middle is permanent data storage, 

in the top-right corner is in-line storage

ES-10XX

In 1960s IBM made enormous improvement in computer design by spending 5 billion dollars for one project and created IBM 360. The project was twice as expensive as the first nuclear bomb project "Manhattan". The Soviet Union heads wanted to have a copy of this "toy". And, after the contract between USA and USSR, IBM 360 was licensed to the Soviet Union. The government obliged to create very similar machine, so across the country appeared the series of IBM clones ES-1010, ES-1020, ES-1030, ES-10XX computers.
While giving the lecture in museum Badrutdinova remembered Korolev's words, the father of Soviet rocket program, who liked to quote his tutor Bartini: "Can left behind system overtake advanced system? - No, if it tries to catch up moving behind, but what if it takes a short cut?".  The Soviet ministers did not want to see local and new computer versions, they just wanted to have american clones. This was the fatal error in the strategy and the death of true novelty! The engineers struggled and liked to build improved and new competitive versions, but their power was voiceless. The command to produce ES-1030 computers came to the Kazan computer factory. After some years the engineers with V.F. Gusev finally managed to create advanced and more robust version, model ES-1033. The leaders and consumers were impressed and this computer was successfully exported to other countries. In 1970s and 1980s the manufacture produced overall 2600 of ES-1033 machines.

ES-1033 computer

ES-1045.01 computer 

(input model of the improved version of ES-1045)

IBM-360

For the purity I do not cover all computers and electronic devices that were produced by the Kazan computer factory. It is clear that engineers did very good job and made very good and competitive computers, often patching and extending existed models. The factory holds lots of top Soviet awards for its production. It seems unbelievable, but the quality of computers was so good, that some of them still are in use in Russian military agency. The products was used in all Soviet space centres and exported to India, Check Republic, Poland, Bulgaria, Hungary, Mongolia. Thus, even now it is quite possible to meet one of these old workers in real life applications.

Later the factory was merged with the famous UK company ICL. Now it is part of Fujitsu companies.

Modern awards from Russian government.

On the left side the knight is the present from Kaspersky Antivirus for its partnership.

P.S. I am very thankful to Margarita Badrutdinova for the personal lecture and the book about Kazan computer history. You can reach this small but valuable museum at Sibirskiy trakt Street, 34, Kazan city, Russia.


Sources (in Russian and English):

M. Badrutdinova, The Kazan computer factory, 2004, ISBN5-9222-0091-7

Kazan computer museum http://kazan-computer-museum.blogspot.ru/

M. Badrutdinova and retrotexnika.ru, "Our answer to Steve Jobs and Bill Gates. The development of electronic engineering in USSR" https://www.youtube.com/watch?v=_9cr4dQPoa8

Setun emulator http://trinary.ru/projects/setunws/

Nairi: triumph and drama http://lebed.com/2003/art3598.htm

Universal systems ES-series https://www.youtube.com/watch?v=Yy2unObU8x8

Virtual Russian computer museum http://www.computer-museum.ru/

Virtual Ukrainian computer museum http://www.icfcst.kiev.ua/MUSEUM/museum-map_r.html

ICL http://www.icl.ru/