Sunday, February 26, 2017

Octopus meets Magento on Windows and Linux

One project that I am working with consists of two parts: 
* Backend job runners, written in C#.NET Full Windows Framework and 
* Frontend web site based on Magento CMS, which is written in PHP

In our team there was already the Windows VPS machine for .NET applications. Also we use Continuous Deployment tool Octopus Deploy, which requires Octopus Server to be installed on Windows OS. 
Magento is working by default with LAMP (Linux Apache MySql PHP) infrastructure and nothing is said about Windows. Some guys use WAMP (Windows Apache MySql PHP) infrastructure, but mixing Apache and IIS is not a good idea. Thus we decided to give a try and run Magento in Windows + PHP + IIS alongside existed .NET applications.

The first thing was to convert many Magento .htaccess files (Apache) to web.config (IIS). This task itself is pretty hard and manual. IIS has auto converting feature for .htaccess => IIS rules web.config, but most of the time IIS failed to do it properly.

The next thing is to make a package containing the whole website for Octopus. We use TeamCity for CI and decided to pack, as usual for .NET, everything in a NuGet package. The trick is that NuPack tool works excellent with .NET dlls, but we finally found how to pack even PHP and .htaccess files.

To do so create .nuspec file in the root of the project:
<?xml version="1.0"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
    <!-- Commands: https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Clients/NuGet.CommandLine/Commands/PackCommand.cs -->
    <!-- Schema: https://docs.microsoft.com/en-gb/nuget/schema/nuspec -->
    <metadata>
        <id>MyProject</id>
        <title>Project Title</title>
        <version>1.0.0</version>
        <description>This is a Magento project</description>
        <authors>Your team</authores>
    </metadata>
    <files>
        <file src="**" exclude=".git\**" />
    </files> 
</package> 

The most import thing here is to make the files filter src="**", which includes all kind of types and extensions. The fields "id, version, description, authors" are required.

Then run nuget pack command pointing to our .nuspec file.
nuget pack .nuspec -NoPackageAnalysis -NoDefaultExcludes

 NoPackageAnalysis speeds up the packing, NoDefaultExcludes protects from nuget throwing out special non .NET world files. This can be simply configured in TeamCity NuPack Runner.
The process of packaging takes about 3 minutes as Nuget provides only one level of compression and the project is big (about 40 000 files).

After that you will have your MyProject.nuget file that can be pushed to Octopus (I will skip the details). Another solution is use custom NodeJS package instead of NuGet package.

The next problem was figured out with the help of Octopus. Linux terminal usually swallows the errors from magento setup tool when you run 
php -dmemory_limit=2G bin/magento setup:static-content:deploy

Better to run verbose output '-v' for showing the errors
php -dmemory_limit=2G bin/magento setup:static-content:deploy -v

Gladly Octopus understood the presence of errors even without "-v" argument and refused to continue the deployment. The problem was in the popular Magento theme that we use Sm Market. Bootstrap source code contained import .less rules with wrong file names in Magento/app/design/frontend/Sm/market/web/css/source/bootstrap/_theme.less,
fix it by pointing to the proper file names:
@import "_variables";
@import "_mixins.less";



After a lot of fighting and trying to run Magento in IIS, eventually it was working with (still) a lot of errors and problems due the url rewriting rules and user rights. The time spent on this became too big, so the decision was to move Magento to its Alma mater: Linux machine and Apache. 
However we would like to continue working with awesome Octopus tool. Recently their team added support for Linux OS as tentacles (the server still requires Windows). This was an excellent case for us to give it a try.

Linux

The first step is to follow the guidance Octopus SSH Taget and install mono. Plus mysql, apache, php, etc for Magento.

Then create user "octopus":
sudo useradd -m octopus echo octopus:<the-password-you-want> | sudo chpasswd

We decided not to create a user per app in UAT machine and to run web sites by the octopus user. To do so just add the octopus user to the apache group:
sudo usermod -g apache octopus

The next point is to give octopus the ability to work with apachectl command (CentOS) without entering password. Create octopus section in sudoers:
sudo visudo -f /etc/sudoers.d/octopus 


and put data:
#Allow octopus to control apache 
octopus     ALL=(ALL) NOPASSWD:/usr/sbin/apachectl
Defaults!/usr/sbin/apachectl !requiretty

Thus octopus will be able to run sudo apachectl restart without entering the password.


Apache

Now the time to set up Apache.
Open /etc/httpd/conf/httpd.conf and comment all Directory settings except the global one restrictions.

If you have Include conf.d/*.conf row then you can create  /etc/httpd/conf.d/magento.conf,
otherwise just put the next lines to etc/httpd/conf/httpd.conf
Include /home/octopus/.octopus/Applications/OctopusServer/Development/YourProjectName/*.conf
Include /home/octopus/.octopus/Applications/OctopusServer/UAT/YourProjectName/*.conf

It means that config files from the above folders will be used by Apache. If you have Apache 2.4+ then it is better to use IncludeOptional directive, which does not throw error if the path does not exist.

The configuration files will be created one per project and will update Apache bindings in the next way:
<VirtualHost *:80>
  ServerName www.sitename.com 
  DocumentRoot /home/octopus/.octopus/Applications/OctopusServer/Development/YourProjectName/1.0.20/
  <Directory /home/octopus/.octopus/Applications/OctopusServer/Development/YourProjectName/1.0.20/> 
       Options Indexes FollowSymLinks MultiViews 
       AllowOverride All 
       Order allow,deny 
       Allow from all 
  </Directory>
</VirtualHost>

If you do not like this approach you can use pm2 tool for starting and stoping processes.

Octopus

Set up Octopus steps
We only need some scripts to run in bash.
(Magento installation scripts with Apache config update are placed at the end of this article)


Install and Upgrade Magento step:


parameters are taken from Octopus Variables:
--magento-directory="#{Octopus.Action[Deploy to Apache].Output.Package.InstallationDirectoryPath}" 
--admin-firstname="#{AdminFirstname}" --admin-lastname="#{AdminLastname}" 
--admin-email="#{AdminEmail}" --admin-user="#{AdminUser}" 
--admin-password="#{AdminPassword}" --base-url="#{BaseUrl}" 
--backend-frontname="#{BackendFrontname}" --db-name="#{DbName}" 
--db-user="#{DbUser}" --db-password="#{DbPassword}"

Update Apache Bindings step:

parameters are also taken from Octopus Variables:
--magento-directory="#{Octopus.Action[Deploy to Apache].Output.Package.InstallationDirectoryPath}" 
--server-name="#{WebsiteName}"


And that is it.
The deploy process works like a charm.


MagentoDeploy.sh


MagentoUpdateApacheBindings.sh




Saturday, February 11, 2017

Can we stop date formatting mess?

What does the next date exactly present?


Most people from over the world will say that it is the 2nd of November 2017. But it is not true for American and Albanian people.

Airbnb for US culture will show the popup with the 11th of February 2017.




















This becomes more entangled when we get Russian, Ukrainian and Kazakstan format DD.MM.YYYY, which is often shortened to DD.MM.YY. For example, the 11th of February 2017 is usually written as 11.02.17.

To make internationalisation more complex some countries (Chine, Lithuania, South Africa) use YYYY-MM-DD or YYYY.MM.DD, which sometimes people like to shorten to YY.MM.DD, and our previous example becomes 17.02.11.

If you think that nothing can be worth, there are countries (Germany, Canada, Sweden, Philippines), which apply different date formats depending on a context.

Requirements

Taking into the account all previous points and preventing likelihood problems the solution of the problem should be:
1) Visible immediately as a static text without any data manipulation (no need to open a popup date editor)
2) A day, a month and a year should be distinguishable from each other
3) The distinguish signs should have the colour of the text and be readable after a monochrome print, e.g. in books and newspapers.
4) The distinguish signs should be simple and built from UTF symbols.

Proposal of a solution

Let us start our visual identification from the shortest amount of time - a day, to the most lasting - a year. The simplest visual effect, that is also very convenient to program or use in draw editors, is underlining. Underlining also works good with all popular charset encodings.

A day gets full width underline = DD;
A month gets a half width underline if the year part is short (YY) = MM; otherwise month has no underline = MM;
A year has no underline = YYYY or YY;

Examples

Our initial confusing date presentation now becomes absolutely clear:


Some people like to trim zeros and write, for example, 7.9.2017. Even this date can be transformed to an obvious presentation: 7.9.2017.

More complex 11.10.12 YY.MM.DD can be transformed to 11.10.12 or to 12.10.11, which is actually the same as  12.10.2012 and 10/12/2012.

And now the bonus for developers and almost anybody who process much data. A common SQL result nowadays says nothing about a date format:

while after our small fix the data is a pleasure to read:

Table of different date formats 

Date Meaning
2/11/2017 The 11th of February 2017
2/11/2017 The 2nd of November 2017
10/3 or 10/03 The 10th of March
01/17 January 2017
2017-12-10 The 10th of December 2017
05/04/03 The 5th of March 2003
05/04/03 The 3rd of March 2005
2/3/2020 The 3rd of February 2020


Happy coding!