Automate backup restoration
present slides https://slides.woluweb.be
video (FR) YouTube : rJbYrQQlRP4
JUG Wallonie is the Belgian French-speaking Joomla User Group
Meeting Saturday 21 October 2017 from 9:30 to 12:00
@ Mont-Saint-Guibert (RueDuWeb) | Belgium
« An untested backup is as good as no backup at all »
your life !
1. Think local, act global
This presentation has been made with care
for our local Joomla User Group (JUG)
It is presented live in front of real human beings
But it is also streamed and will also remain available 24/7 on YouTube so that everybody can enjoy it virtually ;)
2. At your own risks
Look at this presentation at your own risks
This presentation contains no allergens but the (physical/virtual) presence at a JUG can induce a dependence.
If you feel the symptoms after this presentation, follow instructions :
- go to more JUG meetings, JoomlaDays, Conferences etc
- whatever your level, starter or pro, share your knowledge
- contribute to the forum, write articles, make presentations, …
- start volunteering https://volunteers.joomla.org
- spread the Joomla Love
3. Not sponsored
In this presentation I just share my enthousiasm about a nice solution so that everyone can benefit from my own experience
But note that I am not sponsored in any way by the developer of the mentioned Restoration Tool (all I have is a free sticker )
Like many I just happen to be a happy subscriber of his Backup Tool…
… and his Restoration Tool is available at no extra cost
In the previous episode…
1. Automatic backups & Amazon S3
In a previous presentation I explained in detail “how to secure your automatic backups by transferring them to the cloud on Amazon S3 with Akeeba Backup”
2. The previous presentation
One script to automate them all
In this presentation, I will explain in detail the next step, namely how to use a script that will automatically
- restore the backups of all your sites (for ex. on your local server)
- upgrade Joomla (experimental)
- update all extensions (experimental)
1.1. When is it useful ??
Different cases :
- test whether backups are indeed OK
- test before updating extensions
- test before upgrading Joomla
so now you can easily contribute testing Beta & RC ! ;)
- mass site creation
- maintaining demo sites
1.2. Only for Joomla ??
The automated restoration solution works for all the major CMS. The automated upgrade/update is nevertheless specific to Joomla, unless you know of a script that can do that job.
- Drupal 7/8
- Magento 1.x/2.x
- phpBB 3
1.3. Which tools ??
- Akeeba Backup (free or pro) to take the backups
- Akeeba UNiTE (no extra cost if you have any subscription)
- A central script + 1 XML file for each website to handle
- A local server (Win/Mac/Linux. UNiTE is agnostic)
1.4. Akeeba UNiTE ??
To quote the site :
“A command-line application created for power users. Automatically restore one or multiple sites. Useful for mass site creation or maintaining demo sites. Exclusively available to AKEEBABACKUP, BACKUPWP, SOLOPHP, ESSENTIALS and JOOMLADELUXE subscribers.”
2. Brief presentation
Let’s suppose you manage plenty of websites.
As you know, you should definitely check the integrity & completeness of your backups on a regular basis.
Besides this, before upgrading Joomla and updating all extensions, you would like to be sure that everything will be OK.
Last, now that it becomes so easy to restore/upgrade/update all your sites, you will be very happy to test the future Beta and Release Candidate releases of Joomla and to contribute !
Let’s see how to automate the whole process very easily.
One double-click on a script and all your sites are restored/upgraded/updated.
See the “exciting” video of the script doing your work on the next slide (1m32s for 2 sites in this example).
All you need to do after is to check on your local server if all websites are fine…
1. Download and install Akeeba UNiTE
First of all, please RTFM :) Everything is explained in detail in the following official documentation https://www.akeebabackup.com/documentation/unite/using.htm
Download Akeeba UNiTE and unzip it on your local server. For this presentation, I have uncompressed it in a folder named
akeeba_unite ```My local server being Xampp for Windows, the root is ```C:\xampp\htdocs``` and so here is the complete path in my case : ```php C:\xampp\htdocs\akeeba_unite\
For every site, create an XML file that you put in the inbox subfolder of Akeeba Unite. In my case :
2. The XML file for a local .jpa backup file
Example of XML file (local)
<?xml version="1.0" encoding="UTF-8"?> <unite scripting="02_angie"> <siteInfo> <package>/demo_backups/1demo.com.jpa</package> <deletePackage>0</deletePackage> <localLog>test.log</localLog> <emailSysop>0</emailSysop> <name>Restored with Akeeba Unite</name> <email>email@example.com</email> <absolutepath>C:\xampp\htdocs\akeeba_unite_output\1demo</absolutepath> <adminID>79</adminID> <adminUser>admin</adminUser> <adminEmail>firstname.lastname@example.org</adminEmail> <adminPassword><![CDATA[admin]]></adminPassword> <force_ssl>0</force_ssl> <removephpini>1</removephpini> <removehtpasswd>1</removehtpasswd> <replacehtaccess>1</replacehtaccess> </siteInfo> <databaseInfo> <database name="site"> <changecollation>0</changecollation> <dbdriver>mysqli</dbdriver> <dbhost>localhost</dbhost> <dbuser>root</dbuser> <dbpass></dbpass> <dbname>1demo</dbname> <dbprefix>test_</dbprefix> <existing>drop</existing> <foreignkey>1</foreignkey> <noautovalue>1</noautovalue> <replace>0</replace> <utf8db>0</utf8db> <utf8tables>0</utf8tables> <utf8mb4>0</utf8mb4> </database> </databaseInfo> </unite>
2.2. Comments : <package> tag
In my case, all .jpa backup files are stored in a subfolder called demo_backups :
Therefore, for a filename being 1demo.com.jpa, the
<package> tag is as follows :
2.3. Comments : <absolutepath> tag
I want the restoration to be put in a folder called
Therefore the tag is as follows :
2.4. Comments : tags related to the Super User
These 'admin’ Tags are specific to Joomla
but you have similar Tags for all the other CMS
<adminID>79</adminID> <adminUser>admin</adminUser> <adminEmail>email@example.com</adminEmail> <adminPassword><![CDATA[admin]]></adminPassword>
adminID : you HAVE to put the right ID of the Super User bc UNiTE has no other way to know
adminUser : you can change the Name of that user… but that won’t change its Username (used to login)
adminEmail : if you feel like, you can change it
adminPassword : very handy, you can change it so that you can quickly login on your Local Server
2.5. Comments : ANGIE parameters 1/2
<force_ssl>0</force_ssl> <removephpini>1</removephpini> <removehtpasswd>1</removehtpasswd> <replacehtaccess>1</replacehtaccess>
Akeeba UNiTE was already very efficient when it comes to install on a local server the backups of all sites.
But when I discovered it this summer, I still had to do a little series of manual interventions :
- all my sites have SSL enforced… but I have no SSL on my local server
- some of my sites have php.ini at the root, which would have side-effects on my local server
- all my sites have their administration protected by an htpsswd. I had to delete those manually on the local server
- all my sites have long htaccess (optimisation+security), which had side-effects on the local server (like adding www)
2.6. Comments : ANGIE parameters 2/2
Then I contacted Nicholas (the author of Akeeba)… and a few weeks later he came with a new version of Akeeba UNiTE, offering those 4 new options (these being already available when you restore manually btw)
Txs Nicholas :)
For more information, see
https://www.akeebabackup.com/download/akeeba-unite/3–1–0.html > Show release notes
2.7. Comments : <dbname>
Just a little tip : for each site, you have to choose its database name. If you give the same “prefix” (like
s3_ in the following example), then all those datatabases will be gathered in PhpMyAdmin, which makes it easier to manage (for example if you want to delete them all)
3. The XML file for a .jpa backup file from AmazonS3
Example of XML file (AS3)
<?xml version="1.0" encoding="UTF-8"?> <unite scripting="02_angie"> <s3> <accesskey>ABCDEF</accesskey> <secretkey>123456</secretkey> <signature>v4</signature> <bucket>mybucket</bucket> <region>eu-central-1</region> <ssl>1</ssl> <filename>path/to/site-backup.jpa</filename> </s3> <siteInfo> <package from="s3"></package> <deletePackage>0</deletePackage> <localLog>test.log</localLog> <emailSysop>0</emailSysop> <name>Restored with Akeeba Unite</name> <email>firstname.lastname@example.org</email> <absolutepath>C:\xampp\htdocs\akeeba_unite_output\3demo-amazons3</absolutepath> <adminID>79</adminID> <adminUser>admin</adminUser> <adminEmail>email@example.com</adminEmail> <adminPassword><![CDATA[admin]]></adminPassword> <force_ssl>0</force_ssl> <removephpini>1</removephpini> <removehtpasswd>1</removehtpasswd> <replacehtaccess>1</replacehtaccess> </siteInfo> <databaseInfo> <database name="site"> <changecollation>0</changecollation> <dbdriver>mysqli</dbdriver> <dbhost>localhost</dbhost> <dbuser>root</dbuser> <dbpass></dbpass> <dbname>3demo</dbname> <dbprefix>test_</dbprefix> <existing>drop</existing> <foreignkey>1</foreignkey> <noautovalue>1</noautovalue> <replace>0</replace> <utf8db>0</utf8db> <utf8tables>0</utf8tables> <utf8mb4>0</utf8mb4> </database> </databaseInfo> </unite>
3.2. Amazon S3 v4 API integration
When I discovered Akeeba UNiTE this summer, the restoration directly from Amazon S3 was already present, but only with signature v2. Therefore, the tool would not be able to access my backups, which are stored in EU (Frankfurt) and use v4.
Following an exchange with Nicholas, again he upgraded UNiTE and amended the Documentation to integrate Amazon S3 v4 !
Txs Nicholas :)
In particular, a new parameter is now needed, ie the Region
For more information, see
4. 'The’ Script to automate them all
4.1. The principle
The principle is very simple : simply execute the following command from the Akeeba UNiTE folder on your local server
Paradoxically, on a live server running Unix at my hosting companies I knew how to do it…
… but on my local server I had no clue :)
4.2. On a local server on Linux/Mac
If you are a Linux user, no doubt you don’t need my/any help to run the script
If you are a Mac user, macOS comes with PHP pre-installed.
So you just need to do php ./restore-site.php mywebsite.xml
4.3. On a local server on Windows from the Command Line
Search and launch CMD (the Command Line)
Go to the PHP directory of your local server. In my case
- Type the following command,
mywebsite.xmlbeing in the
php C:\xampp\htdocs\akeeba_unite\restore-site.php mywebsite.xml
4.4. On a local server on Windows with a Batch file
Of course, when you have many website to restore, you will want to create a Batch file.
.batfile, for example in
- copy-paste the batch file shown hereafter
- and adapt it of course according to the names of your folders and XML files
4.5. Three parts in the Batch file
The full Batch file is on the next slide, but here are the key commands :
Restore the website
php restore-site.php 1demo.com.xml
Copy the update script into the CLI folder of the website
copy "C:\xampp\htdocs\akeeba_unite\inbox\update.php" "C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php"
- Upgrade Joomla and update extensions
php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --core php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --extensions
Batch file (Windows example)
REM This Script allows to (i) automate the installation of backups (ii) upgrade Joomla (iii) update all extensions pause set STARTTIME=%TIME% REM Needed for Windows : this adds the folder containing php.exe to the path (Xampp being here the Local Server) PATH=%PATH%;C:\xampp\php REM Change Directory to the folder containing your script CD C:\xampp\htdocs\akeeba_unite\ REM Restore all websites php restore-site.php 1demo.com.xml php restore-site.php 2demo.com.xml REM php restore-site.php 3demo.com-amazons3.xml REM Copy the update.php script in the CLI folder of each website copy "C:\xampp\htdocs\akeeba_unite\inbox\update.php" "C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php" copy "C:\xampp\htdocs\akeeba_unite\inbox\update.php" "C:\xampp\htdocs\akeeba_unite_output\2demo\cli\update.php" REM Upgrade Joomla php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --core php C:\xampp\htdocs\akeeba_unite_output\2demo\cli\update.php --core REM Update all extensions php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --extensions php C:\xampp\htdocs\akeeba_unite_output\2demo\cli\update.php --extensions @echo off echo Start Time : %STARTTIME% echo End Time : %time% pause exit
4.7. After running the script
If you restore many websites, after the script has run you might want to copy-paste the contents of your Command Line window to your favourite Notepad.
That way, you can quickly check whether all restorations (and updates/upgrades) went fine.
NB : the script show the Start Time and End Time. This is of course optional. In my case, each restore takes on average less than 1 minute.
4.8. Run a customised query after restoration
You can create a SQL file to suit you own needs and point to it in your XML file in UNiTE
That’s BTW how you can automate everything with UNiTE, for example
- disable misbehaving plugins (see example in Troubleshooting)
- disable all email / live services plugins
- anonymize the users (names and emails) when restoring a backup of the site locally
Fine-tuning & troubleshooting
1. update.php is 'experimental’
For more information about update.php, see
But please be aware that update.php should not be used on production sites, as it has drawbacks :
See for example https://github.com/joomla-projects/cli-update/issues/1
Also, read here the feedback by Nicholas : I had written the same kind of script. Eh, I wouldn’t recommend updating Joomla! from CLI. The last bit which cleans up old files is still a hit and miss and we’ve only been able to make it work through the web interface. We being Michael Babker, George Willson and yours truly. The update is based on Akeeba Restore which is also the heart of Kickstart, UNiTE and the integrated restoration feature in my backup software. The trick is that we have a post-extraction script which loads a thin layer of Joomla! to execute the post-update tasks. This needs to load the NEW version of Joomla! you have just installed. When running from CLI you end up with some of the loaded classes being from the PREVIOUSLY installed version of Joomla! which can make things go boom.
Your website forces SSL (https) but your local server does not handle https ?
4 alternative solutions :
- set $force_ssl to 0 in your configuration.php file
- just force your browser to show that “insecure website” (you’re on YOUR local server anyway)
- have fun and install an SSL certificate on your local server
- use a local server with like MAMP and its built-in SSL support (the best option for Windows and macOS)
If the configuration.php is not part of the backup, the automatic restoration will not work (although a manual restoration with Akeeba would work !).
So be sure that configuration.php is part of your backup…
4. configuration.php > memcache
If you use Memcache on your live site, the locally restored website will not work.
Simply edit configuration.php and change the
$cache_handler as follows :
$cache_handler = 'file'
5. configuration.php > customisation
More generally about anything related to configuration.php :
If you are restoring the same site on your local / demo / dev / whatever server all the time you can create a small ZIP file with a prefabricated configuration.php file and have UNiTE extract it to the site’s root.
That’s what Nicholas himself does with his continuous restoration of my live site’s backups to my local server :)
6. Redirected to “www.localhost”
When you check your sites, you get redirected from
You probably left the original .htaccess file forcing
Just delete it and replace it by the native .htaccess file
Even better : use the option in the XML file allowing to automate it !
Another cause can be some “conflict” with your antivirus software.
Example from a forum :
“My Avast ! Free Antivirus redirects localhost:8080 to www.localhost.com:8080. I reset the web redirection setting in Avast!”
7. SiteGround Cache Plugin
If you use SiteGround’s Cache Plugin, you might get an error message on the front-end of your website.
Simply disable that plugin from the backend in order to test your site effectively.
You can also automate this by creating a SQL file that disable misbehaving plugins and point to it in your XML file in UNiTE.
8. Restore phase freezes
Your process freezes in the middle of the restore phase ?
Just delete the stealth .htaccess file and finish the installation manually (
9. No images
You see no images on your local website ? You probably left the original .htaccess file. Just delete it and replace it by the native .htaccess file
Even better : use the option in the XML file allowing to automate it !
10. View tables
In the case of two websites “sharing some tables” (like the Users table) with a View, the restoration would not work for the website having the View instead of the Table.
11. Name of site
The XML file allows to change the Name of the site.
Well, for multilingual sites, if you have setup a different Name for some language, of course this is left as it is.
So if you really need to have that changed as well, you can also play with Akeeba UNiTE in order to run some SQL query automatically.
12. Backup would not be restored
Well, there could be many reasons to that one. But what I have experienced from a couple of sites is that there were some files with special character at the root of the original website, like
(even though that question mark had never caused me any issue during manual backup/restore !)
1. How to handle the backup files
1.1. Option 1 : directly from the Cloud
Actually you don’t even have to download your backups in the first place.
In your XML file, you can directly specify your credentials.
This can be done with the following Tag for Amazon S3 : https://www.akeebabackup.com/documentation/unite/using.html#xml-s3
Or with the following Tag if you connect directly to your website : https://www.akeebabackup.com/documentation/unite/using.html#xml-remote
The disadvantage : you have to edit each XML file to adapt for the name of the backup file because (unless you configure it otherwise) the latter typically contains a timestamp.
1.2. Option 2 : download first all backup files from the Cloud
To download all your backups from Amazon S3, use for example Cyberduck
NB : contrary to other softwares like FileZilla, Cyberduck can indeed login to AS3
(Or download them from each website, but that will take much more time…)
1.3. Option 3 : backups already available locally
Well, if your backups are already available for example on the local server where you want to restore them all, no need to download them.
1.3.1. Be efficient/lazy : manual solution
In Options 2 and 3, your backups are on your local server. But as already mentionned for Option 1, the name of the backup files typically contain a timestamp.
One could of course simply adapt each XML file accordingly everytime.
But I guess a better solution is to first rename the backup files manually.
1.3.2. Be efficient/lazy : automatic solution
An even more efficient solution is to rename the backup files automatically.
Using Windows, this can be done for example with a tool like Bulk Rename Utility : http://www.bulkrenameutility.co.uk/Download.php
In my case, I remove the first 9 characters (site-www.) and the last 16 characters (timestamp of the backup) :
1.4. Option 4.
As suggested by Nicholas, you can even automate the whole process (download from AS3/rename/restore) with shell scripts.
With other words, a kind of mix of all previous Options :
use s3cmd to list the backups in the bucket and awk to extract the name of the latest backup
- then s3cmd again to download it and rename it to site.jpa
- then launch UNiTE
FYI that bit works perfectly on a Raspberry Pi 2, a really underpowered ARM thingie with barely 512M RAM.
2. Task Scheduler
I’m not sure I would need that, but on Windows you could use the Task Scheduler in order to launch the restoration of all backups automatically at regular intervals.
But this would of course first require to also automate the Download of the backups.
If you want to test your websites further, you might want to install a MailCatcher. That way, you can also simulate the sending of a newsletter, the transactional emails etc.
See for example http://nilhcem.com/FakeSMTP
4. Other backup & archive tools
Arkiv is an easy-to-use backup and archive tool.
Arkiv is designed to backup local files and MySQL databases, and archive them on Amazon S3 and Amazon Glacier. Backup files are removed (locally and from Amazon S3) after defined delays.
Arkiv could backup your data on a daily or an hourly basis (you can choose which day and/or which hours it will be launched). It is written in pure shell, so it can be used on any Unix/Linux machine.
Control, repair or convert your zip, jpa, rar files
Presentation in French :
4.3. Installatron Remote
Someone has kindly told me about another solution to backup/restore/upgrade websites.
I haven’t tested it, but feel free to see what it can do for yourself :
5. Other DevOps tools
I discovered the existence of update.php in an excellent presentation made by Andy Gaskell at #jd17uk
for even more interesting DevOps tools :)
The previous episode was about automating backups in the cloud
The present episode is about automating restoration (and updates/upgrades) on your local server
In the next episode, why would you not do the final step and learn how to become a Joomla Tester ?
Spread the Joomla Love
This presentation : https://slides.woluweb.be
Watch the video (FR) : YouTube : rJbYrQQlRP4
Any suggestion about it ?
Please feel free to contact me. I’ll be happy to keep improving it :)
Marc Dechèvre @woluweb +32 474 37 13 12