--- title: 'Automate backup restoration' created_time: '2018-06-04 15:33:51' language: fr subtitle: '' --- # Automate backup restoration **Compatible with**
Joomla :fa-joomla: | WordPress | Drupal | Prestashop | Magento | Moodle | phpBB present slides https://slides.woluweb.be video (FR) [YouTube : rJbYrQQlRP4](https://www.youtube.com/watch?v=rJbYrQQlRP4) ![Marc](%URL%.images/marc_dechevre.jpg "150x150") ## The quote **« An untested backup is as good as no backup at all »** Quote of
the day
the week
the month
the year
your life ! ## Warning ### Think local, act global This presentation has initially been made
for our local Joomla User Group (JUG Wallonie)
https://www.jugwallonie.be [Facebook : joomla.belgique](https://www.facebook.com/groups/joomla.belgique)
But it was also streamed in order to remain available 24/7 on YouTube so that everybody can enjoy it virtually ;) Since then, it has been updated and also presented a.o. at different JoomlaDays : https://www.joomladagen.nl https://www.joomladay.fr ### At your own risks **Look at this presentation at your own risks** This presentation contains no allergens but the (physical/virtual) presence at Joomla events can induce a dependence. If you feel the symptoms after this presentation, follow instructions : * go to more JUG meetings, JoomlaDays, Conferences etc * whatever your level, starter or pro, share your knowledge * contribute to the forum, write articles, make presentations, ... * start volunteering https://volunteers.joomla.org * spread the Joomla Love ### Not sponsored In this presentation I just share my enthousiasm about a nice solution so that everyone can benefit from my own experience But note that **I am not sponsored in any way** by the developer of the mentioned Restoration Tool (all I have is a free sticker ) Like many I just happen to be a happy subscriber of his Backup Tool...
... and his Restoration Tool is available at no extra cost ## In the previous episode... ### Automatic backups & Amazon S3 In a **previous presentation** I explained in detail "how to secure your automatic backups by transferring them to the cloud on Amazon S3 with Akeeba Backup" EN https://www.slideshare.net/woluweb/how-to-become-a-joomla-tester

FR https://www.slideshare.net/woluweb/comment-devenir-un-testeur-joomla
### The previous presentation ## One script to automate them all In **this presentation**, I will explain in detail the next step, namely how to use a script that will automatically 1. **restore** the backups of all your sites (for ex. on your local server) 2. **upgrade** Joomla (experimental) 3. **update** all extensions (experimental) ### When is it useful ? Different cases : 1. test whether backups are indeed OK 2. test before updating extensions/plugins (or installing new ones) 3. test before upgrading your CMS
so now you can easily contribute testing Beta & RC ! ;) 4. mass site creation 5. maintaining demo sites ### Only for Joomla ? The automated **restoration** solution works **for all the major CMS**. The automated upgrade/update is nevertheless specific to Joomla, unless you know of a script that can do that job. * Joomla! :fa-joomla: * WordPress * Drupal * Prestashop * Magento * Moodle * phpBB ### Which tools ? 1. Akeeba Backup (free or pro) to take the backups 2. Akeeba UNiTE (no extra cost if you have any subscription) 3. A central script + 1 XML file for each website to handle 4. A local server (Win/Mac/Linux. UNiTE is agnostic) ### Akeeba Backup ? Akeeba Backup for Joomla :fa-joomla: Download https://www.akeebabackup.com/products/akeeba-backup.html Documentation https://www.akeebabackup.com/documentation/akeeba-backup-documentation.html Akeeba Backup for WordPress Download https://www.akeebabackup.com/products/akeeba-backup-wordpress.html Documentation https://www.akeebabackup.com/documentation/akeeba-solo.html Akeeba "SOLO" for all other CMS or PHP applications Download https://www.akeebabackup.com/products/akeeba-solo.html Documentation https://www.akeebabackup.com/documentation/akeeba-solo.html ### Akeeba UNiTE ? To quote the site : "A command-line application created for power users. Automatically restore one or multiple sites. Useful for mass site creation or maintaining demo sites. Exclusively available to AKEEBABACKUP, BACKUPWP, SOLOPHP, ESSENTIALS and JOOMLADELUXE subscribers." ![akeeba](%URL%.images/logo-akeeba.svg "250x75") ![unite](%URL%.images/logo-unite.png "75x75") https://www.akeebabackup.com/products/akeeba-unite.html https://www.akeebabackup.com/documentation/unite/ ## A few screenshots ### Before Let's suppose you manage plenty of websites. As you know, you should definitely check the **integrity & completeness** of your backups on a regular basis. Besides this, before **upgrading Joomla and updating all extensions**, you would like to be sure that everything will be OK. Last, now that it becomes so easy to restore/upgrade/update all your sites, you will be very happy to test the future **Beta and Release Candidate** releases of Joomla and to contribute ! --- ![Before](%URL%.images/before.png) ### During Let's see how to automate the whole process very easily. One double-click on a script and all your sites are restored/upgraded/updated. See the "exciting" video of the script doing **your work** on the next slide (1m32s for 2 sites in this example). --- ![During](%URL%.images/script.gif) ### After All you need to do after is to check on your local server if all websites are fine... --- ![After](%URL%.images/after.png) ## Download and install Akeeba UNiTE First of all, please RTFM :) Everything is explained in detail in the following official documentation https://www.akeebabackup.com/documentation/unite/ Download Akeeba UNiTE and unzip it on your local server. For this presentation, I have uncompressed it in a folder named ```php akeeba_unite ``` --- My local server being Xampp for Windows, the root is ```C:\xampp\htdocs``` and so here is the complete path in my case : ```php C:\xampp\htdocs\akeeba_unite\ ``` For every site, create an XML file that you typically put in the same folder as Akeeba UNiTE. You could create a subfolder of course.
Note that before version 4.x of Akeeba UNiTe, the XML files were typically in a ```inbox``` subfolder.
## The XML file for a local .jpa backup file ### Example of XML file : local ```php /demo_backups/1demo.jpa 0 test.log 0 Restored with Akeeba Unite site@example.com C:\xampp\htdocs\akeeba_unite_output\1demo 79 admin marc@demo.com 0 1 1 1 0 mysqli localhost root 1demo test_ drop 1 1 0 0 0 0 ``` ### Comments : `````` tag In my case, all .jpa backup files are stored in a subfolder called ```demo_backups``` : ```bat C:\xampp\htdocs\akeeba_unite\demo_backups ``` Therefore, for a filename being 1demo.jpa, the `````` tag is as follows : ```bat /demo_backups/1demo.jpa ``` ### Comments : `````` tag I want the restoration to be put in a folder called ```akeeba_unite_output\1demo``` Therefore the `````` tag is as follows : ```bat C:\xampp\htdocs\akeeba_unite_output\1demo ``` ### Comments : tags related to the Super User :fa-joomla: == These 'admin' Tags are specific to Joomla
but you have similar Tags for all the other CMS== , , , ... : see https://www.akeebabackup.com/documentation/unite/ ```bat 79 admin marc@demo.com ``` **adminID** : you HAVE to put the right ID of the Super User bc UNiTE has no other way to know **adminUser** : you can change the Name of that user... but that won't change its Username (used to login) **adminEmail** : if you feel like, you can change it **adminPassword** : very handy to quickly login on your Local Server ### Comments : ANGIE parameters 1/2 ```bat 0 1 1 1 ``` Akeeba UNiTE was already very efficient when it comes to install on a local server the backups of all sites. But when I discovered it, I still had to do a little series of manual interventions : * all my sites have SSL enforced... but I have no SSL on my local server * some of my sites have php.ini at the root, which would have side-effects on my local server * all my sites have their administration protected by an htpsswd. I had to delete those manually on the local server * all my sites have long htaccess (optimisation+security), which had side-effects on the local server (like adding www) ### Comments : ANGIE parameters 2/2 Then I contacted Nicholas (the author of Akeeba)... and a few weeks later he came with a new version of Akeeba UNiTE, offering those 4 new options (these being already available when you restore manually btw) **Txs Nicholas :)** For more information, see the release notes of version 3.1.0 https://www.akeebabackup.com/support/desktop-utilities/Ticket/28141 ### Comments : `````` Just a little tip : for each site, you have to choose its database name. If you give the same "prefix" (like ```s3_``` in the following example), then all those datatabases will be gathered in PhpMyAdmin, which makes it easier to manage (for example if you want to delete them all) ```php s3_website1 ``` ## The XML file for a .jpa backup file from AmazonS3 ### Example of XML file : AS3 ```php ABCDEF 123456 v4 mybucket eu-central-1 1 path/to/site-backup.jpa 0 test.log 0 Restored with Akeeba Unite site@example.com C:\xampp\htdocs\akeeba_unite_output\3demo-amazons3 79 admin marc@demo.com 0 1 1 1 0 mysqli localhost root 3demo test_ drop 1 1 0 0 0 0 ``` ### Amazon S3 v4 API integration When I discovered Akeeba UNiTE, the restoration directly from Amazon S3 was already present, but only with signature v2. Therefore, the tool would not be able to access my backups, which are stored in EU (Frankfurt) and use v4. Following an exchange with Nicholas, again he upgraded UNiTE and amended the Documentation to integrate Amazon S3 v4 ! **Txs Nicholas :)** In particular, a new parameter is now needed, ie the Region http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region For more information, see https://www.akeebabackup.com/documentation/unite/using.html#xml-s3 ## 'The' Script to automate them all ### The principle for restoration The principle for restoration is very simple : simply execute the following command from the Akeeba UNiTE folder on your local server ```php unite.phar mywebsite.xml ``` Note : before version 4.x released in 2018, the command was ```php restore-site.php mywebsite.xml ``` #### About version 4.x of Akeeba UNiTE Advantage of the new **phar** format introduced in version 4.x of Akeeba UNiTE: everything is in **1** file :) Note : before version 4, the XML files were by default in a subfolder called '''inbox''' But anyway you are of course free to place your xml files whereever you want. Example from the Manual : ```php /usr/bin/php-cli /path/to/unite.phar /home/myuser/backups/mysite.xml ``` #### Command line on a local server on Linux/Mac If you are a Linux user, no doubt you don't need my/any help to run the script If you are a Mac user, macOS comes with PHP pre-installed.
So you just need to do php ./unite.phar mywebsite.xml #### Command Line on a local server on Windows Paradoxically, on a live server running Unix at my hosting companies I knew how to use the Command Line... but on my Windows local server I had no clue :) So here the answer : 1. Search and launch CMD (the Command Line) 2. Go to the PHP directory of your local server. In my case ```php cd c:\xampp\php ``` 3. Type the following command, ```mywebsite.xml``` being in the same folder as ```unite.phar``` ```php php C:\xampp\htdocs\akeeba_unite\unite.phar mywebsite.xml ``` #### On a local server on Windows with a Batch file Of course, when you have many website to restore, you want to create a Batch file in order to process **all** your websites in one double-click - create a ```.bat``` file, for example in ```C:\``` - copy-paste the batch file shown hereafter - and adapt it of course according to the names of your folders and XML files ### The principle for upgrade/update There is a script out there allowing to upgrade Joomla and update its extensions from the Command Line. It is a proof of concept so - do not use it on Production site (see next slide) - but for what we are doing (testing purposes), it is just fine Download simply ```update.php``` from
https://github.com/joomla-projects/cli-update and put that file in the same folder as '''unite.phar''' NB: afterwards we will copy this file on each restored website in order to be executed, see hereafter #### update.php is 'experimental' But please be aware that ```update.php``` should not be used on Production sites, as it has drawbacks : See for example https://github.com/joomla-projects/cli-update/issues/1

Also, read here the feedback by Nicholas : I had written the same kind of script. Eh, I wouldn't recommend updating Joomla! from CLI. The last bit which cleans up old files is still a hit and miss and we've only been able to make it work through the web interface. We being Michael Babker, George Wilson and yours truly. The update is based on Akeeba Restore which is also the heart of Kickstart, UNiTE and the integrated restoration feature in my backup software. The trick is that we have a post-extraction script which loads a thin layer of Joomla! to execute the post-update tasks. This needs to load the NEW version of Joomla! you have just installed. When running from CLI you end up with some of the loaded classes being from the PREVIOUSLY installed version of Joomla! which can make things go boom.
NB : see the future of CLI update here: https://github.com/joomla-projects/gsoc18_cli_update ### Three parts in the Batch file The full Batch file is detailed in 2 versions on the next 2 slides (Windows example), but here are the key commands/steps : 1. Restore the website ```php php unite.phar 1demo.xml ``` 2. Copy the update script into the CLI folder of the website ```php copy "C:\xampp\htdocs\akeeba_unite\update.php" "C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php" ``` 3. Update extensions and upgrade Joomla ```php php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --extensions php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --core ``` ### Batch file ```php REM This Script (i) automates installation of backups (ii) upgrades Joomla (iii) updates all extensions set STARTTIME=%TIME% REM Needed for Windows: adds the folder containing php.exe to the path (Xampp being here the Local Server) PATH=%PATH%;C:\xampp\php REM Change Directory to the folder containing your script CD C:\xampp\htdocs\akeeba_unite\ REM Restore all websites php unite.phar 1demo.xml php unite.phar 2demo.xml REM Copy the update.php script in the CLI folder of each website copy "C:\xampp\htdocs\akeeba_unite\update.php" "C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php" copy "C:\xampp\htdocs\akeeba_unite\update.php" "C:\xampp\htdocs\akeeba_unite_output\2demo\cli\update.php" REM Update all extensions php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --extensions php C:\xampp\htdocs\akeeba_unite_output\2demo\cli\update.php --extensions REM Upgrade Joomla php C:\xampp\htdocs\akeeba_unite_output\1demo\cli\update.php --core php C:\xampp\htdocs\akeeba_unite_output\2demo\cli\update.php --core echo Start Time : %STARTTIME% echo End Time : %time% pause ``` ### Batch file with a loop - if many websites ```php REM This Script (i) automates installation of backups (ii) upgrades Joomla (iii) updates all extensions set STARTTIME=%TIME% REM Needed for Windows: adds the folder containing php.exe to the path (Xampp being here the Local Server) PATH=%PATH%;C:\xampp\php REM Change Directory to the folder containing your script CD C:\xampp\htdocs\akeeba_unite\ for %%x in (1demo 2demo 3demo 4demo) do ( REM Restore backup php unite.phar %%x.xml REM Copy the update.php script in the CLI folder of each website copy "C:\xampp\htdocs\akeeba_unite\update.php" "C:\xampp\htdocs\akeeba_unite_output\%%x\cli\update.php" REM Update all extensions php C:\xampp\htdocs\amazons3\%%x\cli\update.php --extensions REM Upgrade Joomla php C:\xampp\htdocs\amazons3\%%x\cli\update.php --core ) echo Start Time : %STARTTIME% echo End Time : %time% pause ``` ### After running the script If you restore many websites, after the script has run you might want to copy-paste the contents of your Command Line window to your favourite Notepad. That way, you can quickly check whether all restorations (and updates/upgrades) went fine. NB : the script show the Start Time and End Time. This is of course optional. In my case, each restore takes on average less than 1 minute. ### Run a customised query after restoration Note : you can even create a SQL file to suit you own needs and point to it in your XML file in UNiTE https://www.akeebabackup.com/documentation/unite/using.html#xml-extrasql That's BTW how you can automate everything with UNiTE, for example * disable misbehaving plugins (see examples in Troubleshooting) * disable all email / live services plugins * anonymize the users (names and emails) when restoring a backup of the site locally ## Fine-tuning & troubleshooting ### httpS Your website forces SSL (https) but your local server does not handle https ? 5 alternative solutions : 1. manual solution : set $force_ssl to 0 in your configuration.php file 2. just force your browser to show that "insecure website" (you're on YOUR local server anyway) 3. have fun and install an SSL certificate on your local server
http://www.blog.florian-bogey.fr/installer-un-certificat-ssl-sous-wamp.html 4. use a local server with like MAMP and its built-in SSL support (the best option for Windows and macOS) 5. ... or enjoy the ```0``` parameter that you can put in your XML file
### configuration.php If the configuration.php is not part of the backup, the automatic restoration **will not work** (although a manual restoration with Akeeba would work !). So be sure that configuration.php is part of your backup... ### configuration.php > memcache If you use Memcache on your live site, the locally restored website will not work. Simply edit configuration.php and change the ```$cache_handler``` as follows : ```php $cache_handler = 'file' ``` ### configuration.php > customisation More generally about anything related to configuration.php : If you are restoring the same site on your local / demo / dev / whatever server all the time you can create a small ZIP file with a prefabricated configuration.php file and have UNiTE extract it to the site's root. https://www.akeebabackup.com/documentation/unite/using.html#xml-extrafiles That's what Nicholas himself does with his continuous restoration of my live site's backups to my local server :) ### Redirected to "www.localhost" When you check your sites, you get redirected from ```localhost``` to ```www.localhost``` ? You probably left the original .htaccess file forcing ```www```.
Just delete it and replace it by the native .htaccess file Even better : use the option in the XML file allowing to automate it ! Another cause can be some "conflict" with your antivirus software. Example from a forum :
"My Avast ! Free Antivirus redirects localhost:8080 to www.localhost.com:8080. I reset the web redirection setting in Avast!"
### SiteGround Cache Plugin If you use SiteGround's Cache Plugin, you might get an error message on the front-end of your website. Simply disable that plugin from the backend in order to test your site effectively. You can also automate this by creating a SQL file that disable misbehaving plugins and point to it in your XML file in UNiTE. ### Restore phase freezes Your process freezes in the middle of the restore phase ? Just delete the stealth .htaccess file and finish the installation manually (```localhost/yourwebsite```) ### No images You see no images on your local website ? You probably left the original .htaccess file. Just delete it and replace it by the native .htaccess file Even better : use the option in the XML file allowing to automate it ! ### View tables In the case of two websites "sharing some tables" (like the Users table) with a View, the restoration would not work for the website having the View instead of the Table. ### Name of site The XML file allows to change the Name of the site. Well, for multilingual sites, if you have setup a different Name for some language, of course this is left as it is. So if you really need to have that changed as well, you can also play with Akeeba UNiTE in order to run some SQL query automatically. ### Backup would not be restored Well, there could be many reasons to that one. But what I have experienced from a couple of sites is that there were some files with special character at the root of the original website, like ```blabla_still_useful?.txt``` (even though that question mark had never caused me any issue during manual backup/restore !) ### Issue with Acymailing and local restoration This issue has in fact nothing to do in itself with Akeeba UNiTE, but when you restore a website having an Acymailing module on the front-end, typically the page will throw a popup telling that "localhost is not allowed". In order to fix that, go to ```Acymailing > Configuration > Subscription > Allowed domains for the redirection via the module``` and add ```|localhost``` to the field ## Going further ### How to handle the backup files #### Option 1 : directly from the Cloud Actually you don't even have to download your backups in the first place. In your XML file, you can directly specify your credentials. This can be done with the following Tag for Amazon S3 : https://www.akeebabackup.com/documentation/unite/using.html#xml-s3
Or with the following Tag if you connect directly to your website : https://www.akeebabackup.com/documentation/unite/using.html#xml-remote
The disadvantage : you have to edit each XML file to adapt for the name of the backup file because (unless you configure it otherwise) the latter typically contains a timestamp.
Example : ```site-www.mysite.com-20170926-075404``` #### Option 2 : download first all backup files from the Cloud To download all your backups from Amazon S3, use for example Cyberduck NB : contrary to other softwares like FileZilla, Cyberduck can indeed login to AS3 (Or download them from each website, but that will take much more time...) #### Option 3 : backups already available locally Well, if your backups are already available for example on the local server where you want to restore them all, no need to download them. ##### Be efficient/lazy : manual solution In **Options 2 and 3**, your backups are on your local server. But as already mentionned for Option 1, the name of the backup files typically contain a timestamp. One could of course simply adapt each XML file accordingly everytime. But I guess a better solution is to first rename the backup files manually. ##### Be efficient/lazy : automatic solution An even more efficient solution is to rename the backup files automatically. Using Windows, this can be done for example with a tool like Bulk Rename Utility : http://www.bulkrenameutility.co.uk/Download.php In my case, I remove the first 9 characters (site-www.) and the last 16 characters (timestamp of the backup) :
![Bulk Rename Utility](%URL%.images/bulk_rename_utility.png) #### Option 4. As suggested by Nicholas, you can even **automate the whole process** (download from AS3/rename/restore) with shell scripts.
With other words, a kind of mix of all previous Options : * use s3cmd to list the backups in the bucket and awk to extract the name of the latest backup
https://github.com/s3tools/s3cmd * then s3cmd again to download it and rename it to site.jpa * then launch UNiTE FYI that bit works perfectly on a Raspberry Pi 2, a really underpowered ARM thingie with barely 512M RAM. ### Task Scheduler I'm not sure I would need that, but on Windows you could use the Task Scheduler in order to launch the restoration of all backups automatically at regular intervals. But this would of course first require to also automate the Download of the backups. ### MailCatcher If you want to test your websites further, you might want to install a MailCatcher. That way, you can also simulate the sending of a newsletter, the transactional emails etc. See for example http://nilhcem.com/FakeSMTP ### Other backup & archive tools #### Arkiv See https://github.com/Amaury/Arkiv : Arkiv is an easy-to-use backup and archive tool. Arkiv is designed to backup local files and MySQL databases, and archive them on Amazon S3 and Amazon Glacier. Backup files are removed (locally and from Amazon S3) after defined delays. Arkiv could backup your data on a daily or an hourly basis (you can choose which day and/or which hours it will be launched). It is written in pure shell, so it can be used on any Unix/Linux machine. #### Zipi Control, repair or convert your zip, jpa, rar files https://zipi-tools.io/ Presentation in French :
https://inetis.ch/item/zipi-outil-verification-continue-de-vos-backups #### Installatron Remote Someone has kindly told me about another solution to backup/restore/upgrade websites. I haven't tested it, but feel free to see what it can do for yourself : http://installatron.com/remote ### Other DevOps tools I discovered the existence of update.php in an excellent presentation made by **Andy Gaskell at #jd17uk** See
https://www.youtube.com/watch?v=w5ORPXt6W_o
for even more interesting DevOps tools :) ## Next episode **The previous episode** was about automating backups in the cloud EN https://www.slideshare.net/woluweb/joomla-akeeba-backup-and-amazon-s3
FR https://www.slideshare.net/woluweb/akeeba-backup-et-amazon-s3
**The present episode** is about automating restoration (and updates/upgrades) on your local server **In the next episode**, why would you not do the final step and learn how to become a Joomla Tester ? EN https://www.slideshare.net/woluweb/how-to-become-a-joomla-tester
FR https://www.slideshare.net/woluweb/comment-devenir-un-testeur-joomla
--- ## Spread the Joomla Love ![Spread the Joomla Love](%URL%.images/spread_the_joomla_love.jpg) ## Joomla! GSoC 18 - CLI Update **The project** (published on 2 June 2018) The aim of this project is to add a good CLI Application to the Joomla! CMS, that can perform the updates without going to the administrative panel. Administrators can just go via SSH to update somethings within the CMS. https://community.joomla.org/gsoc-2018/joomla-gsoc-18-cli-update.html The idea for it to be core feature in Joomla! 4. See the code here : https://github.com/joomla-projects/gsoc18_cli_update ## Thank you https://slides.woluweb.be Watch the video (FR) : [YouTube : rJbYrQQlRP4](https://www.youtube.com/watch?v=rJbYrQQlRP4) Any suggestion about this presentation ?
Please feel free to contact me. I'll be happy to keep improving it :) Marc Dechèvre | @woluweb
+32 474 37 13 12 | +32 2 772 58 69 https://www.woluweb.be/contact https://www.slideshare.net/woluweb