Automate backup restoration

present slides https://slides.woluweb.be

video (FR) YouTube : rJbYrQQlRP4

JUG Wallonie is the Belgian French-speaking Joomla User Group

Meeting Saturday 21 October 2017 from 9:30 to 12:00
@ Mont-Saint-Guibert (RueDuWeb) | Belgium

https://www.jugwallonie.be Facebook : joomla.belgique
logo JUGW

The quote

« An untested backup is as good as no backup at all »

Quote of                    
the day          
the week
          the month
                    the year
                              your life !

Warning

1. Think local, act global

This presentation has been made with care
for our local Joomla User Group (JUG)

It is presented live in front of real human beings

But it is also streamed and will also remain available 24/7 on YouTube so that everybody can enjoy it virtually ;)

2. At your own risks

Look at this presentation at your own risks

This presentation contains no allergens but the (physical/virtual) presence at a JUG can induce a dependence.

If you feel the symptoms after this presentation, follow instructions :

3. Not sponsored

In this presentation I just share my enthousiasm about a nice solution so that everyone can benefit from my own experience

But note that I am not sponsored in any way by the developer of the mentioned Restoration Tool (all I have is a free sticker )

Like many I just happen to be a happy subscriber of his Backup Tool…
… and his Restoration Tool is available at no extra cost

In the previous episode…

1. Automatic backups & Amazon S3

In a previous presentation I explained in detail “how to secure your automatic backups by transferring them to the cloud on Amazon S3 with Akeeba Backup”

EN https://www.slideshare.net/woluweb/how-to-become-a-joomla-tester
FR https://www.slideshare.net/woluweb/comment-devenir-un-testeur-joomla

2. The previous presentation

One script to automate them all

In this presentation, I will explain in detail the next step, namely how to use a script that will automatically

  1. restore the backups of all your sites (for ex. on your local server)
  2. upgrade Joomla (experimental)
  3. update all extensions (experimental)

1. Introduction

1.1. When is it useful ??

Different cases :

  1. test whether backups are indeed OK
  2. test before updating extensions
  3. test before upgrading Joomla
    so now you can easily contribute testing Beta & RC ! ;)
  4. mass site creation
  5. maintaining demo sites

1.2. Only for Joomla ??

The automated restoration solution works for all the major CMS. The automated upgrade/update is nevertheless specific to Joomla, unless you know of a script that can do that job.

1.3. Which tools ??

  1. Akeeba Backup (free or pro) to take the backups
  2. Akeeba UNiTE (no extra cost if you have any subscription)
  3. A central script + 1 XML file for each website to handle
  4. A local server (Win/Mac/Linux. UNiTE is agnostic)

1.4. Akeeba UNiTE ??

To quote the site :

“A command-line application created for power users. Automatically restore one or multiple sites. Useful for mass site creation or maintaining demo sites. Exclusively available to AKEEBABACKUP, BACKUPWP, SOLOPHP, ESSENTIALS and JOOMLADELUXE subscribers.”

logo Akeebalogo UNiTE

https://www.akeebabackup.com/products/akeeba-unite.html https://www.akeebabackup.com/documentation/unite/using.htm

2. Brief presentation

2.1. Before

Let’s suppose you manage plenty of websites.

As you know, you should definitely check the integrity & completeness of your backups on a regular basis.

Besides this, before upgrading Joomla and updating all extensions, you would like to be sure that everything will be OK.

Last, now that it becomes so easy to restore/upgrade/update all your sites, you will be very happy to test the future Beta and Release Candidate releases of Joomla and to contribute !Before

2.2. During

Let’s see how to automate the whole process very easily.

One double-click on a script and all your sites are restored/upgraded/updated.

See the “exciting” video of the script doing your work on the next slide (1m32s for 2 sites in this example).During

2.3. After

All you need to do after is to check on your local server if all websites are fine…After

The scripts

1. Download and install Akeeba UNiTE

First of all, please RTFM :) Everything is explained in detail in the following official documentation https://www.akeebabackup.com/documentation/unite/using.htm

Download Akeeba UNiTE and unzip it on your local server. For this presentation, I have uncompressed it in a folder named

akeeba_unite
```My local server being Xampp for Windows, the root is ```C:\xampp\htdocs``` and so here is the complete path in my case :

```php
C:\xampp\htdocs\akeeba_unite\

For every site, create an XML file that you put in the inbox subfolder of Akeeba Unite. In my case :

C:\xampp\htdocs\akeeba_unite\inbox\

2. The XML file for a local .jpa backup file

Example of XML file (local)

<?xml version="1.0" encoding="UTF-8"?>
<unite scripting="02_angie">
  <siteInfo>
    <package>/demo_backups/1demo.com.jpa</package>
    <deletePackage>0</deletePackage>
    <localLog>test.log</localLog>
    <emailSysop>0</emailSysop>
    <name>Restored with Akeeba Unite</name>
    <email>site@example.com</email>
    <absolutepath>C:\xampp\htdocs\akeeba_unite_output\1demo</absolutepath>

    <adminID>79</adminID>
    <adminUser>admin</adminUser>
    <adminEmail>marc@demo.com</adminEmail>
    <adminPassword><![CDATA[admin]]></adminPassword>

    <force_ssl>0</force_ssl>
    <removephpini>1</removephpini>
    <removehtpasswd>1</removehtpasswd>
    <replacehtaccess>1</replacehtaccess>

  </siteInfo>

  <databaseInfo>
    <database name="site">
      <changecollation>0</changecollation>
      <dbdriver>mysqli</dbdriver>
      <dbhost>localhost</dbhost>
      <dbuser>root</dbuser>
      <dbpass></dbpass>
      <dbname>1demo</dbname>
      <dbprefix>test_</dbprefix>

      <existing>drop</existing>
      <foreignkey>1</foreignkey>
      <noautovalue>1</noautovalue>
      <replace>0</replace>
      <utf8db>0</utf8db>
      <utf8tables>0</utf8tables>
      <utf8mb4>0</utf8mb4>
    </database>
  </databaseInfo>

</unite>

2.2. Comments : <package> tag

In my case, all .jpa backup files are stored in a subfolder called demo_backups :

C:\xampp\htdocs\akeeba_unite\inbox\demo_backups

Therefore, for a filename being 1demo.com.jpa, the <package> tag is as follows :

<package>/demo_backups/1demo.com.jpa</package>

2.3. Comments : <absolutepath> tag

I want the restoration to be put in a folder called akeeba_unite_output\1demo

Therefore the tag is as follows :

<absolutepath>C:\xampp\htdocs\akeeba_unite_output\1demo</absolutepath>

These 'admin’ Tags are specific to Joomla
but you have similar Tags for all the other CMS

, , , … : see https://www.akeebabackup.com/documentation/unite/using.html

<adminID>79</adminID>
<adminUser>admin</adminUser>
<adminEmail>marc@demo.com</adminEmail>
<adminPassword><![CDATA[admin]]></adminPassword>

adminID : you HAVE to put the right ID of the Super User bc UNiTE has no other way to know

adminUser : you can change the Name of that user… but that won’t change its Username (used to login)

adminEmail : if you feel like, you can change it

adminPassword : very handy, you can change it so that you can quickly login on your Local Server

2.5. Comments : ANGIE parameters 1/2

<force_ssl>0</force_ssl>
<removephpini>1</removephpini>
<removehtpasswd>1</removehtpasswd>
<replacehtaccess>1</replacehtaccess>

Akeeba UNiTE was already very efficient when it comes to install on a local server the backups of all sites.

But when I discovered it this summer, I still had to do a little series of manual interventions :

2.6. Comments : ANGIE parameters 2/2

Then I contacted Nicholas (the author of Akeeba)… and a few weeks later he came with a new version of Akeeba UNiTE, offering those 4 new options (these being already available when you restore manually btw)

Txs Nicholas :)

For more information, see

https://www.akeebabackup.com/download/akeeba-unite/3–1–0.html > Show release notes

https://www.akeebabackup.com/support/desktop-utilities/Ticket/28141

2.7. Comments : <dbname>

Just a little tip : for each site, you have to choose its database name. If you give the same “prefix” (like s3_ in the following example), then all those datatabases will be gathered in PhpMyAdmin, which makes it easier to manage (for example if you want to delete them all)

<dbname>s3_website1</dbname>

3. The XML file for a .jpa backup file from AmazonS3

Example of XML file (AS3)

<?xml version="1.0" encoding="UTF-8"?>
<unite scripting="02_angie">

    <s3>
        <accesskey>ABCDEF</accesskey>
        <secretkey>123456</secretkey>
        <signature>v4</signature>
        <bucket>mybucket</bucket>
        <region>eu-central-1</region>
        <ssl>1</ssl>
        <filename>path/to/site-backup.jpa</filename>
    </s3>

  <siteInfo>
    <package from="s3"></package>
    <deletePackage>0</deletePackage>
    <localLog>test.log</localLog>
    <emailSysop>0</emailSysop>
    <name>Restored with Akeeba Unite</name>
    <email>site@example.com</email>
    <absolutepath>C:\xampp\htdocs\akeeba_unite_output\3demo-amazons3</absolutepath>

    <adminID>79</adminID>
    <adminUser>admin</adminUser>
    <adminEmail>marc@demo.com</adminEmail>
    <adminPassword><![CDATA[admin]]></adminPassword>

    <force_ssl>0</force_ssl>
    <removephpini>1</removephpini>
    <removehtpasswd>1</removehtpasswd>
    <replacehtaccess>1</replacehtaccess>

  </siteInfo>

  <databaseInfo>
    <database name="site">
      <changecollation>0</changecollation>
      <dbdriver>mysqli</dbdriver>
      <dbhost>localhost</dbhost>
      <dbuser>root</dbuser>
      <dbpass></dbpass>
      <dbname>3demo</dbname>
      <dbprefix>test_</dbprefix>

      <existing>drop</existing>
      <foreignkey>1</foreignkey>
      <noautovalue>1</noautovalue>
      <replace>0</replace>
      <utf8db>0</utf8db>
      <utf8tables>0</utf8tables>
      <utf8mb4>0</utf8mb4>
    </database>
  </databaseInfo>

</unite>

3.2. Amazon S3 v4 API integration

When I discovered Akeeba UNiTE this summer, the restoration directly from Amazon S3 was already present, but only with signature v2. Therefore, the tool would not be able to access my backups, which are stored in EU (Frankfurt) and use v4.

Following an exchange with Nicholas, again he upgraded UNiTE and amended the Documentation to integrate Amazon S3 v4 !

Txs Nicholas :)

In particular, a new parameter is now needed, ie the Region

http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

For more information, see

https://www.akeebabackup.com/documentation/unite/using.html#xml-s3

4. 'The’ Script to automate them all

4.1. The principle

The principle is very simple : simply execute the following command from the Akeeba UNiTE folder on your local server

restore-site.php mywebsite.xml

Paradoxically, on a live server running Unix at my hosting companies I knew how to do it…

… but on my local server I had no clue :)

4.2. On a local server on Linux/Mac

If you are a Linux user, no doubt you don’t need my/any help to run the script

If you are a Mac user, macOS comes with PHP pre-installed.
So you just need to do php ./restore-site.php mywebsite.xml

4.3. On a local server on Windows from the Command Line

  1. Search and launch CMD (the Command Line)

  2. Go to the PHP directory of your local server. In my case

    cd c:\xampp\php
  3. Type the following command, mywebsite.xml being in the inbox folder
    php C:\xampp\htdocs\akeeba_unite\restore-site.php mywebsite.xml

4.4. On a local server on Windows with a Batch file

Of course, when you have many website to restore, you will want to create a Batch file.

4.5. Three parts in the Batch file

The full Batch file is on the next slide, but here are the key commands :

  1. Restore the website

    php restore-site.php 1demo.com.xml
  2. Copy the update script into the CLI folder of the website

    copy "C:\xampp\htdocs\akeeba_unite\inbox\update.php" "C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php"
  3. Upgrade Joomla and update extensions
    php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --core
    php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --extensions

Batch file (Windows example)

REM This Script allows to (i) automate the installation of backups (ii) upgrade Joomla (iii) update all extensions
pause

set STARTTIME=%TIME%

REM Needed for Windows : this adds the folder containing php.exe to the path (Xampp being here the Local Server)
PATH=%PATH%;C:\xampp\php

REM Change Directory to the folder containing your script
CD C:\xampp\htdocs\akeeba_unite\

REM Restore all websites
php restore-site.php 1demo.com.xml
php restore-site.php 2demo.com.xml
REM php restore-site.php 3demo.com-amazons3.xml

REM Copy the update.php script in the CLI folder of each website
copy "C:\xampp\htdocs\akeeba_unite\inbox\update.php" "C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php"
copy "C:\xampp\htdocs\akeeba_unite\inbox\update.php" "C:\xampp\htdocs\akeeba_unite_output\2demo\cli\update.php"

REM Upgrade Joomla
php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --core
php C:\xampp\htdocs\akeeba_unite_output\2demo\cli\update.php --core

REM Update all extensions
php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --extensions
php C:\xampp\htdocs\akeeba_unite_output\2demo\cli\update.php --extensions

@echo off
echo Start Time : %STARTTIME%
echo End Time   : %time%
pause
exit

4.7. After running the script

If you restore many websites, after the script has run you might want to copy-paste the contents of your Command Line window to your favourite Notepad.

That way, you can quickly check whether all restorations (and updates/upgrades) went fine.

NB : the script show the Start Time and End Time. This is of course optional. In my case, each restore takes on average less than 1 minute.

4.8. Run a customised query after restoration

You can create a SQL file to suit you own needs and point to it in your XML file in UNiTE

https://www.akeebabackup.com/documentation/unite/using.html#xml-extrasql

That’s BTW how you can automate everything with UNiTE, for example

Fine-tuning & troubleshooting

1. update.php is 'experimental’

For more information about update.php, see
https://github.com/joomla-projects/cli-update

But please be aware that update.php should not be used on production sites, as it has drawbacks :

See for example https://github.com/joomla-projects/cli-update/issues/1

Also, read here the feedback by Nicholas : I had written the same kind of script. Eh, I wouldn’t recommend updating Joomla! from CLI. The last bit which cleans up old files is still a hit and miss and we’ve only been able to make it work through the web interface. We being Michael Babker, George Willson and yours truly. The update is based on Akeeba Restore which is also the heart of Kickstart, UNiTE and the integrated restoration feature in my backup software. The trick is that we have a post-extraction script which loads a thin layer of Joomla! to execute the post-update tasks. This needs to load the NEW version of Joomla! you have just installed. When running from CLI you end up with some of the loaded classes being from the PREVIOUSLY installed version of Joomla! which can make things go boom.

2. httpS

Your website forces SSL (https) but your local server does not handle https ?

4 alternative solutions :

  1. set $force_ssl to 0 in your configuration.php file
  2. just force your browser to show that “insecure website” (you’re on YOUR local server anyway)
  3. have fun and install an SSL certificate on your local server
    http://www.blog.florian-bogey.fr/installer-un-certificat-ssl-sous-wamp.html
  4. use a local server with like MAMP and its built-in SSL support (the best option for Windows and macOS)

3. configuration.php

If the configuration.php is not part of the backup, the automatic restoration will not work (although a manual restoration with Akeeba would work !).

So be sure that configuration.php is part of your backup…

4. configuration.php > memcache

If you use Memcache on your live site, the locally restored website will not work.

Simply edit configuration.php and change the $cache_handler as follows :

$cache_handler = 'file'

5. configuration.php > customisation

More generally about anything related to configuration.php :

If you are restoring the same site on your local / demo / dev / whatever server all the time you can create a small ZIP file with a prefabricated configuration.php file and have UNiTE extract it to the site’s root.

https://www.akeebabackup.com/documentation/unite/using.html#xml-extrafiles

That’s what Nicholas himself does with his continuous restoration of my live site’s backups to my local server :)

6. Redirected to “www.localhost”

When you check your sites, you get redirected from localhost to www.localhost ?

You probably left the original .htaccess file forcing www.
Just delete it and replace it by the native .htaccess file

Even better : use the option in the XML file allowing to automate it !

Another cause can be some “conflict” with your antivirus software.

Example from a forum :
“My Avast ! Free Antivirus redirects localhost:8080 to www.localhost.com:8080. I reset the web redirection setting in Avast!”

7. SiteGround Cache Plugin

If you use SiteGround’s Cache Plugin, you might get an error message on the front-end of your website.

Simply disable that plugin from the backend in order to test your site effectively.

You can also automate this by creating a SQL file that disable misbehaving plugins and point to it in your XML file in UNiTE.

8. Restore phase freezes

Your process freezes in the middle of the restore phase ?

Just delete the stealth .htaccess file and finish the installation manually (localhost/yourwebsite)

9. No images

You see no images on your local website ? You probably left the original .htaccess file. Just delete it and replace it by the native .htaccess file

Even better : use the option in the XML file allowing to automate it !

10. View tables

In the case of two websites “sharing some tables” (like the Users table) with a View, the restoration would not work for the website having the View instead of the Table.

11. Name of site

The XML file allows to change the Name of the site.

Well, for multilingual sites, if you have setup a different Name for some language, of course this is left as it is.

So if you really need to have that changed as well, you can also play with Akeeba UNiTE in order to run some SQL query automatically.

12. Backup would not be restored

Well, there could be many reasons to that one. But what I have experienced from a couple of sites is that there were some files with special character at the root of the original website, like

blabla_still_useful?.txt

(even though that question mark had never caused me any issue during manual backup/restore !)

Going further

1. How to handle the backup files

1.1. Option 1 : directly from the Cloud

Actually you don’t even have to download your backups in the first place.

In your XML file, you can directly specify your credentials.

This can be done with the following Tag for Amazon S3 : https://www.akeebabackup.com/documentation/unite/using.html#xml-s3
Or with the following Tag if you connect directly to your website : https://www.akeebabackup.com/documentation/unite/using.html#xml-remote

The disadvantage : you have to edit each XML file to adapt for the name of the backup file because (unless you configure it otherwise) the latter typically contains a timestamp.
Example : site-www.mysite.com-20170926-075404

1.2. Option 2 : download first all backup files from the Cloud

To download all your backups from Amazon S3, use for example Cyberduck

NB : contrary to other softwares like FileZilla, Cyberduck can indeed login to AS3

(Or download them from each website, but that will take much more time…)

1.3. Option 3 : backups already available locally

Well, if your backups are already available for example on the local server where you want to restore them all, no need to download them.

1.3.1. Be efficient/lazy : manual solution

In Options 2 and 3, your backups are on your local server. But as already mentionned for Option 1, the name of the backup files typically contain a timestamp.

One could of course simply adapt each XML file accordingly everytime.

But I guess a better solution is to first rename the backup files manually.

1.3.2. Be efficient/lazy : automatic solution

An even more efficient solution is to rename the backup files automatically.

Using Windows, this can be done for example with a tool like Bulk Rename Utility : http://www.bulkrenameutility.co.uk/Download.php

In my case, I remove the first 9 characters (site-www.) and the last 16 characters (timestamp of the backup) :
Bulk Rename Utility

1.4. Option 4.

As suggested by Nicholas, you can even automate the whole process (download from AS3/rename/restore) with shell scripts.

With other words, a kind of mix of all previous Options :

FYI that bit works perfectly on a Raspberry Pi 2, a really underpowered ARM thingie with barely 512M RAM.

2. Task Scheduler

I’m not sure I would need that, but on Windows you could use the Task Scheduler in order to launch the restoration of all backups automatically at regular intervals.

But this would of course first require to also automate the Download of the backups.

3. MailCatcher

If you want to test your websites further, you might want to install a MailCatcher. That way, you can also simulate the sending of a newsletter, the transactional emails etc.

See for example http://nilhcem.com/FakeSMTP

4. Other backup & archive tools

4.1. Arkiv

See https://github.com/Amaury/Arkiv :

Arkiv is an easy-to-use backup and archive tool.

Arkiv is designed to backup local files and MySQL databases, and archive them on Amazon S3 and Amazon Glacier. Backup files are removed (locally and from Amazon S3) after defined delays.

Arkiv could backup your data on a daily or an hourly basis (you can choose which day and/or which hours it will be launched). It is written in pure shell, so it can be used on any Unix/Linux machine.

4.2. Zipi

Control, repair or convert your zip, jpa, rar files

https://zipi-tools.io/

Presentation in French :
https://inetis.ch/item/zipi-outil-verification-continue-de-vos-backups

4.3. Installatron Remote

Someone has kindly told me about another solution to backup/restore/upgrade websites.

I haven’t tested it, but feel free to see what it can do for yourself :

http://installatron.com/remote

5. Other DevOps tools

I discovered the existence of update.php in an excellent presentation made by Andy Gaskell at #jd17uk

See
https://www.youtube.com/watch?v=w5ORPXt6W_o
for even more interesting DevOps tools :)

Next episode

The previous episode was about automating backups in the cloud

EN https://www.slideshare.net/woluweb/joomla-akeeba-backup-and-amazon-s3
FR https://www.slideshare.net/woluweb/akeeba-backup-et-amazon-s3

The present episode is about automating restoration (and updates/upgrades) on your local server

In the next episode, why would you not do the final step and learn how to become a Joomla Tester ?

EN https://www.slideshare.net/woluweb/how-to-become-a-joomla-tester
FR https://www.slideshare.net/woluweb/comment-devenir-un-testeur-joomla

Spread the Joomla Love

Spread the Joomla Love

Thank you

This presentation : https://slides.woluweb.be

Watch the video (FR) : YouTube : rJbYrQQlRP4

Any suggestion about it ?
Please feel free to contact me. I’ll be happy to keep improving it :)

Marc Dechèvre     @woluweb     +32 474 37 13 12

https://www.woluweb.be/contact

https://www.slideshare.net/woluweb
https://www.facebook.com/woluweb