FromDual TechFeed (en)

Table of Content

• Introduction
• Prerequisites
• Full Backup
  • Create Full Backup
  • Prepare Full Backup
  • Restore Full Backup
  • Prepare slave from full backup
  • Prepare GTID slave from full backup
  • Advantages / Disadvantages
  • Important Hints
• Incremental Backup
  • Create Incremental Backup
  • Prepare Incremental Backup
  • Restore Incremental Backup
  • Advantages / Disadvantages
  • Important Hints
  • Differential Backup
• Partial Backup
  • Create Partial Backup
  • Prepare Partial Backup
  • Restore Partial Backup
  • Advantages / Disadvantages
  • Important Hints

Introduction

No one can deny that one of the most important and daily tasks for DBAs is performing backup and restore operations, we're not required to perform backup and restore operations only when we want to add new replication slave, when we want to implement disaster recovery procedures or when we want to prepare testing or staging server for the running production system, but even if we're going to make any changes to the database schema in order to enhance the database performance, it's recommended to have fresh backup copy before making any live changes, so if backup and restore operations cannot be handled smoothly, we're going to face many troubles in our daily work. If we're going to talk about backup and restore operations, Xtrabackup tool will be strongly appeared.

Xtrabackup tool is a free open source tool developed by Percona to perform physical backup and restore operations which is much faster than performing logical backup and restore using the MySQL utilities (mysqldump and mysql), and many other advantages.

Xtrabackup tool has many options and features which are very useful, but in this article, I'll go through only on how to use this tool to perform simple full, incremental and partial backups and restores, advantages and disadvantages of each method and some important tips.

For more information about Xtrabackup tool, you can browse the manual document from here.

Prerequisites

• MySQL server installed.
• Download the xtrabackup tool.
• Install it as explained in the manual document.

Full Backup

If you want to have a full backup from your entire database system with the shortest/fastest backup and restore time, this method will be very useful for you. As compared to the full logical database backup using mysqldump and mysql utilities (very long time to backup and more than the doubled time to restore), taking a full physical backup using Xtrabackup tool will make your life much easier.

Below is the needed steps to make a full physical database backup using XtraBackup tool:

Create Backup

A simple Xtrabackup command to backup the full databases should be something like:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp /backup/dir/path/full-backup . . . innobackupex: completed OK!

A timestamped folder (for ex. "2013-11-06_00-00-00") would be created to contain all backup files if we didn't use the option "--no-timestamp" in the above command (I didn't use the timestamped folders here to just simplify the names for the reader, otherwise, it's very useful when using automated backup scripts).

Xtrabackup tool now created the backup files under the folder "full-backup" plus some extra files like "xtrabackup-checkpoints" file which contains some information (useful in the incremental backups) like:

• backup_type = full-backuped : which indicates the backup type "full backup".
• from_lsn = 0 : which indicates the log sequence number where the backup process started from (0 means from the beginning).
• to_lsn = 3768762 : which indicates the log sequence number where the backup process ended at.

Another important file is "xtrabackup_binlog_info" which is very useful in replication setups:

[root@ ~]# cat xtrabackup_binlog_info mysql-bin.000027 191

WHERE:

• mysql-bin.000027: is the binary log file name of the master when the backup created.
• 191: is the binary log position of the backup.

Prepare Full Backup

The backed up files are not ready at the moment to be restored, we must prepare the backup files first as follows:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log /backup/dir/path/full-backup . . . innobackupex: completed OK!

Now, the full backup is ready to be restored ...

Restore Full Backup

To get the full backup restored, the MySQL instance should be stopped first and then one of the following two procedures should be done:

• Using the copy back option: [root@ ~]# innobackupex --user=db_user –-password='db_password' --copy-back /backup/dir/path/full-backup . . . innobackupex: completed OK!
  

  Xtrabackup tool - in this method - will copy all files under the backup folder (full-backup) to the MySQL datadir which must be indicated in the my.cnf file, otherwise, it wouldn't know where the datadir should be placed.

• Using the operating system copy or move commands:
  

  If you don't want to keep the backup files on your local system (you have another copy in an external server), the move command will be very fast to get your backup restored:

  [root@ ~]# mv /backup/dir/path/full-backup /var/lib/mysql
  

As the user who moves/copies the files into MySQL datadir is not "mysql" user, you should make sure that mysql user has the right permissions on its datadir and also the path "/var/lib/mysql" should be replaced with the MySQL datadir if it's set to a different path.

[root@ ~]# chown -R mysql:mysql /var/lib/mysql

After moving/copying the backup files into MySQL datadir, you are free to start the MySQL instance again.

Prepare slave from full backup

Preparing a slave using Xtrabackup is pretty easy and a straight forward process:

• Restore the full backup as explained above.
• Check the binary logs information of the backup:
  [root@ ~]# cat xtrabackup_binlog_info mysql-bin.000027 191
  
• Execute the CHANGE MASTER TO command using the above info and start the slave:
  SQL> CHANGE MASTER TO -> MASTER_HOST='master_ip', -> MASTER_PORT=master_port, -> MASTER_USER='slave_user_name', -> MASTER_PASSWORD='slave_password', -> MASTER_LOG_FILE='mysql-bin.000027', ## taken from xtrabackup_binlog_info -> MASTER_LOG_POS=191; ## taken from xtrabackup_binlog_info SQL> START SLAVE;
  

For more information on how to set up MySQL Replication, check out this nice manual link.

Prepare GTID slave from full backup

GTID is supported in Xtrabackup starting from version 2.1.0. To restore a GTID slave server, the GTID_MODE should be enabled on the master server before creating its backup, otherwise, no GTID values will be included in the "xtrabackup_binlog_info" file under the backup directory. However, the following steps should be done:

• Restore the full backup normally as explained above.
• Check the GTID value of the backup:
  [root@ ~]# cat xtrabackup_binlog_info mysql-bin.000027 191 b9b4712a-df64-11e3-b391-60672090eb04:1-12
  
• Set the variable GTID_PURGED with the GTID value of the backup:
  SQL> SET GLOBAL GTID_PURGED="b9b4712a-df64-11e3-b391-60672090eb04:1-12";
  
• Execute the auto position CHANGE MASTER TO command and start the slave:
  SQL> CHANGE MASTER TO -> MASTER_HOST='master_ip', -> MASTER_PORT=master_port, -> MASTER_USER='slave_user_name', -> MASTER_PASSWORD='slave_password', -> MASTER_AUTO_POSITION = 1; SQL> START SLAVE;
  

For more information on how to set up Transaction-based Replication in MySQL, check out this link.

Advantages / Disadvantages

• Advantages:
  • Fast, simple and easy way to get your full database backed up and restored.
  • All Xtrabackup tool features (like streaming: move the backed up files directly to a remote server) are supported in the full backup method.
  • Simple way to introduce a new slave to the master.
• Disadvantages:
  • We have to replace the entire MySQL datadir with the new one (In other words, the datadir folder should be empty/removed before the restore process).
  • We can't extract one single database or single table from the whole backup (Unless it's MyISAM table), which means that you have to take it all or leave it all.

Important Hints

• The message innobackupex: completed OK! should be printed at the end of every xtrabackup command , otherwise, it would be failed to make a successful command (backup, prepare or restore).
• The ib_logfile files size should be the same in both source and destination servers, if not, you have to either remove them from the backup folder (which will be restored) before starting the MySQL instance and MySQL will create new ones for you OR change those files size in the destination server's configuration file to match the same size in the backup before starting the MySQL instance
• The MySQL user used in the Xtrabackup tool, should have at least the following privileges (RELOAD, LOCK TABLES and REPLICATION CLIENT).
• To prepare a new slave from another slave, just add the two options (“--slave-info" & --safe-slave-backup”) to the backup command and use the information in the file "xtrabackup_slave_info" under the backup folder to issue the "CHANGE MASTER TO" command in the new slave after finishing the restore.
• To Accelerate the preparation process of your backup, just add the option "--use-memory" in the prepare command in order to allocate more used memory (Xtrabackup will use the specified memory as an internal innodb_buffer_pool_size for the prepare process), for ex: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --use-memory=512M /backup/dir/path/full-backup . . . innobackupex: completed OK!
• The preparation process consists of two steps, replaying the committed transactions and rolling back the uncommitted transactions, using the --apply-log option only in the preparation command will do both steps for you.
• The backup folder "/backup/dir/path/full-backup" SHOULD NOT be created before executing the backup command, because Xtrabackup will create that folder for you, and it will fail to continue processing if that folder was already exist.

Incremental Backup

When you have a very large database system, you will need large enough storage to store your database backups in, and if you want to perform a daily backup then your mission will be more difficult. In such cases, the incremental database backup method will be very useful. It allows you to have only the database changes (delta) - after the physical full backup – with the minimum storage space required in a fast way, and hence, you can perform the daily backup operations to your database system without the need to having large storage available.

The following steps describe a simple way to perform your physical incremental database backup using XtraBackup tool:

Create Incremental Backup

To perform an incremental backup, we should first perform a full backup - the same like we did in the previous section - to be the base backup of the upcoming incremental backups.

Creating the full backup (Base Backup):

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp /backup/dir/path/full-backup . . . innobackupex: completed OK!

The "xtrabackup-checkpoints" file contents will be something like:

• backup_type = full-backuped : which indicates the backup type "full backup".
• from_lsn = 0 : which indicates the log sequence number where the backup process started from (0 means from the beginning).
• to_lsn = 3768762 : which indicates the log sequence number where the backup process ended at.

Creating the first incremental backup:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp --incremental /backup/dir/path/inc1 --incremental-basedir=/backup/dir/path/full-backup . . . innobackupex: completed OK!

We informed the Xtrabackup tool to perform an incremental backup by adding the command "--incremental", and by specifying the full-backup path as the basedir, we informed it from which backup it should start tracking the database changes.

The "xtrabackup-checkpoints" file contents will be something like:

• backup_type = incremental : which indicates the backup type "incremental backup".
• from_lsn = 3768762 : which indicates the log sequence number where the backup process started from (the same LSN as the previous full backup ended at).
• to_lsn = 4908762 : which indicates the log sequence number where the backup process ended at.

Creating the second incremental backup:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp --incremental /backup/dir/path/inc2 --incremental-basedir=/backup/dir/path/inc1 . . . innobackupex: completed OK!

We informed the Xtrabackup tool to perform an incremental backup by adding the command "--incremental", and by specifying the 1st incremental backup path as the basedir, we informed it to start tracking the database changes since the last incremental (not the full backup).

The "xtrabackup-checkpoints" file contents will be something like:

• backup_type = incremental : which indicates the backup type "incremental backup".
• from_lsn = 4908762 : which indicates the log sequence number where the backup process started from (the same LSN as the 1st incremental backup ended at).
• to_lsn = 6508762 : which indicates the log sequence number where the backup process ended at.

Note: We can create as many incremental backups as we want by using the same procedure above.

Prepare Incremental Backup

As mentioned earlier in the article, the preparation process consists of two steps (replaying the committed transactions and rolling back the uncommitted transactions) and using the --apply-log option only will do both of them (like we did in the full backup) but in the incremental backups, we MUST do them separately as follows:

• Replay the committed transactions on the base backup (by adding the option "--redo-only"): [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --redo-only /backup/dir/path/full-backup . . . innobackupex: completed OK!
• Replay the committed transactions on the 1st incremental backup: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --redo-only /backup/dir/path/full-backup --incremental-dir=/backup/dir/path/inc1 . . . innobackupex: completed OK!
  

  Note: we specified the full backup folder here, because replaying the committed transactions steps, appends all changes from the incremental backup to the full backup.

• Replay the committed transactions on the 2nd incremental backup: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --redo-only /backup/dir/path/full-backup --incremental-dir=/backup/dir/path/inc2 . . . innobackupex: completed OK!
  

  Note: here, we didn't use the 1st incremental backup folder, because all changes in the 1st incremental was already appended to the full backup in the previous step.

• Finally, roll back all uncommitted transactions: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log /backup/dir/path/full-backup . . . innobackupex: completed OK!
  

  Note: as the full backup folder contains all data now (full + 1st & 2nd incremental), there's no need to do this step on the incremental backup folders.

Now, the incremental backup is ready to be restored ...

Restore Incremental Backup

The full backup folder will be the only folder to be restored (there's no need to the incremental backup folders) as it contains all data after appending the changes from all incremental backup. We can restore it the same way we did in the full backup restore.

Advantages / Disadvantages

• Advantages:
  • Less storage resources needed.
  • Faster than the full backup.
• Disadvantages:

  In addition to the disadvantages of the full backup, there are other ones:

  • Complicate and hard process to implement as compared to the full backup.
  • The incremental backup strategy, based on Log Sequence Number which affects only XtraDB and InnoDB storage engines while the others (like MyISAM) will be backed up completely (changed + unchanged data) in each incremental backup process.
  • If we have many incremental backups, appending all of them might consume time and might be confusing as well.
  • If one of the incremental backups become corrupted or not available for any reason, we will not be able to add all incremental backups after that to the full backup.

Important Hints

• The backup preparation sequence steps above, MUST be followed using the same order.
• If the "--redo-only" option was not be used in any of the preparation steps (except the final step), all up coming incremental backups will be useless as we won't be able to add them to the base backup anymore.
• Replaying the committed transactions steps bring all incremental data and append it to the full backup, so that, the rolling back of the uncommitted transactions step should be execute only on the full backup (as it contains already the whole data).).
• In the incremental backups, Xtrabackup generates two files for every table ".delta" & ".meta"(for ex. test.ibd.delta & test.ibd.meta), the delta file size reflects the changes which was applied on that table since the last incremental backup.
• The preparation time of the individual incremental backup will depend on how much data changed there since the last incremental.
• The preparation time for the full backup - in most cases - is really small as compared to the incremental ones because full backups apply the redo logs only while the incremental backups apply the deltas plus the redo logs. So if the delta files are big, the preparation process will take longer.
• Full backups is recommended against Incremental backups if there are many changes applied on the DB, while the incremental backups are recommended when there are few changes applied on the DB.

Differential backup

We can use the incremental backup strategy in order to perform differential backups, but we should consider the following points:

• We always specify the full backup folder as the base backup (in the incremental we specify the previous incremental folder as a base backup)
• All incremental backups between differential and full backups MUST BE ignored when preparing the backup files because the differential backup contains already all changes since the last full backup.
• In the backup preparation process, we should consider the last differential backup as the first incremental backup and all incremental backups after that could be applied normally.

Note: Having differnetial backups in the middle of incremental backups will be useful for many reasons, such as:

• Differential backups reduce the backup preparation steps/time needed because differential backp will replace all its previous incremental backups.
• Differential backups reduce the chances of loosing the incremental backups if we have corrupted incremental backup in the middle, because in this case, differential backup will act as a backup of the previous incremental backups.

Partial Backup

Unlike MyISAM, having physical database backup for a single database or table is not possible if the table engine type is InnoDB. But by using the partial database backup method in the XtraBackup tool, it will be possible to have physical InnoDB tables backup the same like MyISAM ones (but with some restrictions).

The following steps describing how to perform partial database backup using XtraBackup tool:

Create Partial Backup

A simple Xtrabackup command to backup some databases (or tables) should be something like:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp --databases=”db1 db2 db3.tbl1” /backup/dir/path/partial-backup . . . innobackupex: completed OK!
Prepare Partial Backup

The same like the other backup methods, the backed up files are not ready until we get them prepared by adding the "--export" option as follows:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --export /backup/dir/path/partial-backup . . . innobackupex: completed OK!

Some errors regarding those not included InnoDB tables from the backup may be appeared, but that's fine. Also there will be a notification of creating the ".exp" file for each table which will be used in the import (restore) process.

Now, the partial backup is ready to be restored ...

Restore Partial Backup

The restore process of the partial backup is quite different than the full and incremental backups.

To restore a partial backup, the following steps should be made:

• Unlike the other methods (Full and Incremental backups), MySQL instance on the destination server shouldn't be stopped because we will have to execute some SQL commands.
• On the destination server, we should create new tables (as many as we have in the partial backup or as we will restore) with the same structure like the one in the partial backup and then discard its table space: mysql> CREATE TABLE db.tbl1 (...)ENGINE=INNODB; mysql> ALTER TABLE db.tbl1 DISCARD TABLESPACE;
• Copy “.ibd” and “.exp” files for each table into the corresponding DB directory then assign the right permissions to mysql user: [root@ ~]# cp /backup/dir/path/partial-backup/db/tbl1.ibd /var/lib/mysql/db [root@ ~]# cp /backup/dir/path/partial-backup/db/tbl1.exp /var/lib/mysql/db [root@ ~]# chown -R mysql:mysql /var/lib/mysql/db
• Now we should tell MySQL to use the new table spaces: mysql> ALTER TABLE db.tbl1 IMPORT TABLESPACE;

Advantages / Disadvantages

• Advantages:
  • Although it's a complicated process, but it allows us to backup and restore individual InnoDB tables the same like MyISAM.
  • Useful when having huge InnoDB tables and we want to backup/restore them only.
• Disadvantages:
  • The streaming feature is not available in the partial backup.
  • Restoring/importing individual tables or databases from a partial backup is not applicable unless the destination server is Percona Server.
  • In addition to restoring the files(copy back), three SQL statements should be executed for each table (table creation + two ALTER statements) in order to get them ready for use, which means that we might do a very boring job (or we have to create a special script) to get the partial backup restored if it contains many tables.

Important Hints

• Although we didn't remove the MySQL datadir before the restore process (like full and incremental backups) as well as having the MySQL instance running, but we can restore the partial backup using the same way (remove the datadir contents and copy/move the backup files to the datadir), but we should take into our consideration that we'll have only the backed up databases/tables and all other databases/tables (which are not included in the partial backup) will be missed.
• “innodb_file_per_table” server option must be enabled (in both source and destination servers).
• "innodb_expand_import" option must be enabled in the destination server which is available only in Percona server (and that explain why we can restore partial backup on Percona server only).
• Beside the "--databases" option, other two alternative options to perform the same needs could be used but we must provide each table with the fully qualified naming format:
  • --include='db.tbl'.
  • --tables-file=/path/to/file.txt ==> in that file, we can add multiple tables one per line in the fully qualified naming format.

Now, you can use the Xtrabackup tool to perform full, incremental and partial database backups, you can decide which method(s) of them are suitable for you according to the advantages and disadvantages of each one, and by considering the important hints for each method you can perform your backup efficiently.

I hope you found this article useful for you and to be familiar with such wonderful tool.

Taxonomy upgrade extras: GTIDxtrabackupBackup

Table of Content

• Introduction
• Prerequisites
• Full Backup
  • Create Full Backup
  • Prepare Full Backup
  • Restore Full Backup
  • Prepare slave from full backup
  • Prepare GTID slave from full backup
  • Advantages / Disadvantages
  • Important Hints
• Incremental Backup
  • Create Incremental Backup
  • Prepare Incremental Backup
  • Restore Incremental Backup
  • Advantages / Disadvantages
  • Important Hints
  • Differential Backup
• Partial Backup
  • Create Partial Backup
  • Prepare Partial Backup
  • Restore Partial Backup
  • Advantages / Disadvantages
  • Important Hints

Introduction

No one can deny that one of the most important and daily tasks for DBAs is performing backup and restore operations, we're not required to perform backup and restore operations only when we want to add new replication slave, when we want to implement disaster recovery procedures or when we want to prepare testing or staging server for the running production system, but even if we're going to make any changes to the database schema in order to enhance the database performance, it's recommended to have fresh backup copy before making any live changes, so if backup and restore operations cannot be handled smoothly, we're going to face many troubles in our daily work. If we're going to talk about backup and restore operations, Xtrabackup tool will be strongly appeared.

Xtrabackup tool is a free open source tool developed by Percona to perform physical backup and restore operations which is much faster than performing logical backup and restore using the MySQL utilities (mysqldump and mysql), and many other advantages.

Xtrabackup tool has many options and features which are very useful, but in this article, I'll go through only on how to use this tool to perform simple full, incremental and partial backups and restores, advantages and disadvantages of each method and some important tips.

For more information about Xtrabackup tool, you can browse the manual document from here.

Prerequisites

• MySQL server installed.
• Download the xtrabackup tool.
• Install it as explained in the manual document.

Full Backup

If you want to have a full backup from your entire database system with the shortest/fastest backup and restore time, this method will be very useful for you. As compared to the full logical database backup using mysqldump and mysql utilities (very long time to backup and more than the doubled time to restore), taking a full physical backup using Xtrabackup tool will make your life much easier.

Below is the needed steps to make a full physical database backup using XtraBackup tool:

Create Backup

A simple Xtrabackup command to backup the full databases should be something like:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp /backup/dir/path/full-backup . . . innobackupex: completed OK!

A timestamped folder (for ex. "2013-11-06_00-00-00") would be created to contain all backup files if we didn't use the option "--no-timestamp" in the above command (I didn't use the timestamped folders here to just simplify the names for the reader, otherwise, it's very useful when using automated backup scripts).

Xtrabackup tool now created the backup files under the folder "full-backup" plus some extra files like "xtrabackup-checkpoints" file which contains some information (useful in the incremental backups) like:

• backup_type = full-backuped : which indicates the backup type "full backup".
• from_lsn = 0 : which indicates the log sequence number where the backup process started from (0 means from the beginning).
• to_lsn = 3768762 : which indicates the log sequence number where the backup process ended at.

Another important file is "xtrabackup_binlog_info" which is very useful in replication setups:

[root@ ~]# cat xtrabackup_binlog_info mysql-bin.000027 191

WHERE:

• mysql-bin.000027: is the binary log file name of the master when the backup created.
• 191: is the binary log position of the backup.

Prepare Full Backup

The backed up files are not ready at the moment to be restored, we must prepare the backup files first as follows:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log /backup/dir/path/full-backup . . . innobackupex: completed OK!

Now, the full backup is ready to be restored ...

Restore Full Backup

To get the full backup restored, the MySQL instance should be stopped first and then one of the following two procedures should be done:

• Using the copy back option: [root@ ~]# innobackupex --user=db_user –-password='db_password' --copy-back /backup/dir/path/full-backup . . . innobackupex: completed OK!
  

  Xtrabackup tool - in this method - will copy all files under the backup folder (full-backup) to the MySQL datadir which must be indicated in the my.cnf file, otherwise, it wouldn't know where the datadir should be placed.

• Using the operating system copy or move commands:
  

  If you don't want to keep the backup files on your local system (you have another copy in an external server), the move command will be very fast to get your backup restored:

  [root@ ~]# mv /backup/dir/path/full-backup /var/lib/mysql
  

As the user who moves/copies the files into MySQL datadir is not "mysql" user, you should make sure that mysql user has the right permissions on its datadir and also the path "/var/lib/mysql" should be replaced with the MySQL datadir if it's set to a different path.

[root@ ~]# chown -R mysql:mysql /var/lib/mysql

After moving/copying the backup files into MySQL datadir, you are free to start the MySQL instance again.

Prepare slave from full backup

Preparing a slave using Xtrabackup is pretty easy and a straight forward process:

• Restore the full backup as explained above.
• Check the binary logs information of the backup:
  [root@ ~]# cat xtrabackup_binlog_info mysql-bin.000027 191
  
• Execute the CHANGE MASTER TO command using the above info and start the slave:
  SQL> CHANGE MASTER TO -> MASTER_HOST='master_ip', -> MASTER_PORT=master_port, -> MASTER_USER='slave_user_name', -> MASTER_PASSWORD='slave_password', -> MASTER_LOG_FILE='mysql-bin.000027', ## taken from xtrabackup_binlog_info -> MASTER_LOG_POS=191; ## taken from xtrabackup_binlog_info SQL> START SLAVE;
  

For more information on how to set up MySQL Replication, check out this nice manual link.

Prepare GTID slave from full backup

GTID is supported in Xtrabackup starting from version 2.1.0. To restore a GTID slave server, the GTID_MODE should be enabled on the master server before creating its backup, otherwise, no GTID values will be included in the "xtrabackup_binlog_info" file under the backup directory. However, the following steps should be done:

• Restore the full backup normally as explained above.
• Check the GTID value of the backup:
  [root@ ~]# cat xtrabackup_binlog_info mysql-bin.000027 191 b9b4712a-df64-11e3-b391-60672090eb04:1-12
  
• Set the variable GTID_PURGED with the GTID value of the backup:
  SQL> SET GLOBAL GTID_PURGED="b9b4712a-df64-11e3-b391-60672090eb04:1-12";
  
• Execute the auto position CHANGE MASTER TO command and start the slave:
  SQL> CHANGE MASTER TO -> MASTER_HOST='master_ip', -> MASTER_PORT=master_port, -> MASTER_USER='slave_user_name', -> MASTER_PASSWORD='slave_password', -> MASTER_AUTO_POSITION = 1; SQL> START SLAVE;
  

For more information on how to set up Transaction-based Replication in MySQL, check out this link.

Advantages / Disadvantages

• Advantages:
  • Fast, simple and easy way to get your full database backed up and restored.
  • All Xtrabackup tool features (like streaming: move the backed up files directly to a remote server) are supported in the full backup method.
  • Simple way to introduce a new slave to the master.
• Disadvantages:
  • We have to replace the entire MySQL datadir with the new one (In other words, the datadir folder should be empty/removed before the restore process).
  • We can't extract one single database or single table from the whole backup (Unless it's MyISAM table), which means that you have to take it all or leave it all.

Important Hints

• The message innobackupex: completed OK! should be printed at the end of every xtrabackup command , otherwise, it would be failed to make a successful command (backup, prepare or restore).
• The ib_logfile files size should be the same in both source and destination servers, if not, you have to either remove them from the backup folder (which will be restored) before starting the MySQL instance and MySQL will create new ones for you OR change those files size in the destination server's configuration file to match the same size in the backup before starting the MySQL instance
• The MySQL user used in the Xtrabackup tool, should have at least the following privileges (RELOAD, LOCK TABLES and REPLICATION CLIENT).
• To prepare a new slave from another slave, just add the two options (“--slave-info" & --safe-slave-backup”) to the backup command and use the information in the file "xtrabackup_slave_info" under the backup folder to issue the "CHANGE MASTER TO" command in the new slave after finishing the restore.
• To Accelerate the preparation process of your backup, just add the option "--use-memory" in the prepare command in order to allocate more used memory (Xtrabackup will use the specified memory as an internal innodb_buffer_pool_size for the prepare process), for ex: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --use-memory=512M /backup/dir/path/full-backup . . . innobackupex: completed OK!
• The preparation process consists of two steps, replaying the committed transactions and rolling back the uncommitted transactions, using the --apply-log option only in the preparation command will do both steps for you.
• The backup folder "/backup/dir/path/full-backup" SHOULD NOT be created before executing the backup command, because Xtrabackup will create that folder for you, and it will fail to continue processing if that folder was already exist.

Incremental Backup

When you have a very large database system, you will need large enough storage to store your database backups in, and if you want to perform a daily backup then your mission will be more difficult. In such cases, the incremental database backup method will be very useful. It allows you to have only the database changes (delta) - after the physical full backup – with the minimum storage space required in a fast way, and hence, you can perform the daily backup operations to your database system without the need to having large storage available.

The following steps describe a simple way to perform your physical incremental database backup using XtraBackup tool:

Create Incremental Backup

To perform an incremental backup, we should first perform a full backup - the same like we did in the previous section - to be the base backup of the upcoming incremental backups.

Creating the full backup (Base Backup):

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp /backup/dir/path/full-backup . . . innobackupex: completed OK!

The "xtrabackup-checkpoints" file contents will be something like:

• backup_type = full-backuped : which indicates the backup type "full backup".
• from_lsn = 0 : which indicates the log sequence number where the backup process started from (0 means from the beginning).
• to_lsn = 3768762 : which indicates the log sequence number where the backup process ended at.

Creating the first incremental backup:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp --incremental /backup/dir/path/inc1 --incremental-basedir=/backup/dir/path/full-backup . . . innobackupex: completed OK!

We informed the Xtrabackup tool to perform an incremental backup by adding the command "--incremental", and by specifying the full-backup path as the basedir, we informed it from which backup it should start tracking the database changes.

The "xtrabackup-checkpoints" file contents will be something like:

• backup_type = incremental : which indicates the backup type "incremental backup".
• from_lsn = 3768762 : which indicates the log sequence number where the backup process started from (the same LSN as the previous full backup ended at).
• to_lsn = 4908762 : which indicates the log sequence number where the backup process ended at.

Creating the second incremental backup:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp --incremental /backup/dir/path/inc2 --incremental-basedir=/backup/dir/path/inc1 . . . innobackupex: completed OK!

We informed the Xtrabackup tool to perform an incremental backup by adding the command "--incremental", and by specifying the 1st incremental backup path as the basedir, we informed it to start tracking the database changes since the last incremental (not the full backup).

The "xtrabackup-checkpoints" file contents will be something like:

• backup_type = incremental : which indicates the backup type "incremental backup".
• from_lsn = 4908762 : which indicates the log sequence number where the backup process started from (the same LSN as the 1st incremental backup ended at).
• to_lsn = 6508762 : which indicates the log sequence number where the backup process ended at.

Note: We can create as many incremental backups as we want by using the same procedure above.

Prepare Incremental Backup

As mentioned earlier in the article, the preparation process consists of two steps (replaying the committed transactions and rolling back the uncommitted transactions) and using the --apply-log option only will do both of them (like we did in the full backup) but in the incremental backups, we MUST do them separately as follows:

• Replay the committed transactions on the base backup (by adding the option "--redo-only"): [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --redo-only /backup/dir/path/full-backup . . . innobackupex: completed OK!
• Replay the committed transactions on the 1st incremental backup: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --redo-only /backup/dir/path/full-backup --incremental-dir=/backup/dir/path/inc1 . . . innobackupex: completed OK!
  

  Note: we specified the full backup folder here, because replaying the committed transactions steps, appends all changes from the incremental backup to the full backup.

• Replay the committed transactions on the 2nd incremental backup: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --redo-only /backup/dir/path/full-backup --incremental-dir=/backup/dir/path/inc2 . . . innobackupex: completed OK!
  

  Note: here, we didn't use the 1st incremental backup folder, because all changes in the 1st incremental was already appended to the full backup in the previous step.

• Finally, roll back all uncommitted transactions: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log /backup/dir/path/full-backup . . . innobackupex: completed OK!
  

  Note: as the full backup folder contains all data now (full + 1st & 2nd incremental), there's no need to do this step on the incremental backup folders.

Now, the incremental backup is ready to be restored ...

Restore Incremental Backup

The full backup folder will be the only folder to be restored (there's no need to the incremental backup folders) as it contains all data after appending the changes from all incremental backup. We can restore it the same way we did in the full backup restore.

Advantages / Disadvantages

• Advantages:
  • Less storage resources needed.
  • Faster than the full backup.
• Disadvantages:

  In addition to the disadvantages of the full backup, there are other ones:

  • Complicate and hard process to implement as compared to the full backup.
  • The incremental backup strategy, based on Log Sequence Number which affects only XtraDB and InnoDB storage engines while the others (like MyISAM) will be backed up completely (changed + unchanged data) in each incremental backup process.
  • If we have many incremental backups, appending all of them might consume time and might be confusing as well.
  • If one of the incremental backups become corrupted or not available for any reason, we will not be able to add all incremental backups after that to the full backup.

Important Hints

• The backup preparation sequence steps above, MUST be followed using the same order.
• If the "--redo-only" option was not be used in any of the preparation steps (except the final step), all up coming incremental backups will be useless as we won't be able to add them to the base backup anymore.
• Replaying the committed transactions steps bring all incremental data and append it to the full backup, so that, the rolling back of the uncommitted transactions step should be execute only on the full backup (as it contains already the whole data).).
• In the incremental backups, Xtrabackup generates two files for every table ".delta" & ".meta"(for ex. test.ibd.delta & test.ibd.meta), the delta file size reflects the changes which was applied on that table since the last incremental backup.
• The preparation time of the individual incremental backup will depend on how much data changed there since the last incremental.
• The preparation time for the full backup - in most cases - is really small as compared to the incremental ones because full backups apply the redo logs only while the incremental backups apply the deltas plus the redo logs. So if the delta files are big, the preparation process will take longer.
• Full backups is recommended against Incremental backups if there are many changes applied on the DB, while the incremental backups are recommended when there are few changes applied on the DB.

Differential backup

We can use the incremental backup strategy in order to perform differential backups, but we should consider the following points:

• We always specify the full backup folder as the base backup (in the incremental we specify the previous incremental folder as a base backup)
• All incremental backups between differential and full backups MUST BE ignored when preparing the backup files because the differential backup contains already all changes since the last full backup.
• In the backup preparation process, we should consider the last differential backup as the first incremental backup and all incremental backups after that could be applied normally.

Note: Having differnetial backups in the middle of incremental backups will be useful for many reasons, such as:

• Differential backups reduce the backup preparation steps/time needed because differential backp will replace all its previous incremental backups.
• Differential backups reduce the chances of loosing the incremental backups if we have corrupted incremental backup in the middle, because in this case, differential backup will act as a backup of the previous incremental backups.

Partial Backup

Unlike MyISAM, having physical database backup for a single database or table is not possible if the table engine type is InnoDB. But by using the partial database backup method in the XtraBackup tool, it will be possible to have physical InnoDB tables backup the same like MyISAM ones (but with some restrictions).

The following steps describing how to perform partial database backup using XtraBackup tool:

Create Partial Backup

A simple Xtrabackup command to backup some databases (or tables) should be something like:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp --databases=”db1 db2 db3.tbl1” /backup/dir/path/partial-backup . . . innobackupex: completed OK!
Prepare Partial Backup

The same like the other backup methods, the backed up files are not ready until we get them prepared by adding the "--export" option as follows:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --export /backup/dir/path/partial-backup . . . innobackupex: completed OK!

Some errors regarding those not included InnoDB tables from the backup may be appeared, but that's fine. Also there will be a notification of creating the ".exp" file for each table which will be used in the import (restore) process.

Now, the partial backup is ready to be restored ...

Restore Partial Backup

The restore process of the partial backup is quite different than the full and incremental backups.

To restore a partial backup, the following steps should be made:

• Unlike the other methods (Full and Incremental backups), MySQL instance on the destination server shouldn't be stopped because we will have to execute some SQL commands.
• On the destination server, we should create new tables (as many as we have in the partial backup or as we will restore) with the same structure like the one in the partial backup and then discard its table space: mysql> CREATE TABLE db.tbl1 (...)ENGINE=INNODB; mysql> ALTER TABLE db.tbl1 DISCARD TABLESPACE;
• Copy “.ibd” and “.exp” files for each table into the corresponding DB directory then assign the right permissions to mysql user: [root@ ~]# cp /backup/dir/path/partial-backup/db/tbl1.ibd /var/lib/mysql/db [root@ ~]# cp /backup/dir/path/partial-backup/db/tbl1.exp /var/lib/mysql/db [root@ ~]# chown -R mysql:mysql /var/lib/mysql/db
• Now we should tell MySQL to use the new table spaces: mysql> ALTER TABLE db.tbl1 IMPORT TABLESPACE;

Advantages / Disadvantages

• Advantages:
  • Although it's a complicated process, but it allows us to backup and restore individual InnoDB tables the same like MyISAM.
  • Useful when having huge InnoDB tables and we want to backup/restore them only.
• Disadvantages:
  • The streaming feature is not available in the partial backup.
  • Restoring/importing individual tables or databases from a partial backup is not applicable unless the destination server is Percona Server.
  • In addition to restoring the files(copy back), three SQL statements should be executed for each table (table creation + two ALTER statements) in order to get them ready for use, which means that we might do a very boring job (or we have to create a special script) to get the partial backup restored if it contains many tables.

Important Hints

• Although we didn't remove the MySQL datadir before the restore process (like full and incremental backups) as well as having the MySQL instance running, but we can restore the partial backup using the same way (remove the datadir contents and copy/move the backup files to the datadir), but we should take into our consideration that we'll have only the backed up databases/tables and all other databases/tables (which are not included in the partial backup) will be missed.
• “innodb_file_per_table” server option must be enabled (in both source and destination servers).
• "innodb_expand_import" option must be enabled in the destination server which is available only in Percona server (and that explain why we can restore partial backup on Percona server only).
• Beside the "--databases" option, other two alternative options to perform the same needs could be used but we must provide each table with the fully qualified naming format:
  • --include='db.tbl'.
  • --tables-file=/path/to/file.txt ==> in that file, we can add multiple tables one per line in the fully qualified naming format.

Now, you can use the Xtrabackup tool to perform full, incremental and partial database backups, you can decide which method(s) of them are suitable for you according to the advantages and disadvantages of each one, and by considering the important hints for each method you can perform your backup efficiently.

I hope you found this article useful for you and to be familiar with such wonderful tool.

Taxonomy upgrade extras: GTIDxtrabackupBackup

Table of Content

• Introduction
• Prerequisites
• Full Backup
  • Create Full Backup
  • Prepare Full Backup
  • Restore Full Backup
  • Prepare slave from full backup
  • Prepare GTID slave from full backup
  • Advantages / Disadvantages
  • Important Hints
• Incremental Backup
  • Create Incremental Backup
  • Prepare Incremental Backup
  • Restore Incremental Backup
  • Advantages / Disadvantages
  • Important Hints
  • Differential Backup
• Partial Backup
  • Create Partial Backup
  • Prepare Partial Backup
  • Restore Partial Backup
  • Advantages / Disadvantages
  • Important Hints

Introduction

No one can deny that one of the most important and daily tasks for DBAs is performing backup and restore operations, we're not required to perform backup and restore operations only when we want to add new replication slave, when we want to implement disaster recovery procedures or when we want to prepare testing or staging server for the running production system, but even if we're going to make any changes to the database schema in order to enhance the database performance, it's recommended to have fresh backup copy before making any live changes, so if backup and restore operations cannot be handled smoothly, we're going to face many troubles in our daily work. If we're going to talk about backup and restore operations, Xtrabackup tool will be strongly appeared.

Xtrabackup tool is a free open source tool developed by Percona to perform physical backup and restore operations which is much faster than performing logical backup and restore using the MySQL utilities (mysqldump and mysql), and many other advantages.

Xtrabackup tool has many options and features which are very useful, but in this article, I'll go through only on how to use this tool to perform simple full, incremental and partial backups and restores, advantages and disadvantages of each method and some important tips.

For more information about Xtrabackup tool, you can browse the manual document from here.

Prerequisites

• MySQL server installed.
• Download the xtrabackup tool.
• Install it as explained in the manual document.

Full Backup

If you want to have a full backup from your entire database system with the shortest/fastest backup and restore time, this method will be very useful for you. As compared to the full logical database backup using mysqldump and mysql utilities (very long time to backup and more than the doubled time to restore), taking a full physical backup using Xtrabackup tool will make your life much easier.

Below is the needed steps to make a full physical database backup using XtraBackup tool:

Create Backup

A simple Xtrabackup command to backup the full databases should be something like:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp /backup/dir/path/full-backup . . . innobackupex: completed OK!

A timestamped folder (for ex. "2013-11-06_00-00-00") would be created to contain all backup files if we didn't use the option "--no-timestamp" in the above command (I didn't use the timestamped folders here to just simplify the names for the reader, otherwise, it's very useful when using automated backup scripts).

Xtrabackup tool now created the backup files under the folder "full-backup" plus some extra files like "xtrabackup-checkpoints" file which contains some information (useful in the incremental backups) like:

• backup_type = full-backuped : which indicates the backup type "full backup".
• from_lsn = 0 : which indicates the log sequence number where the backup process started from (0 means from the beginning).
• to_lsn = 3768762 : which indicates the log sequence number where the backup process ended at.

Another important file is "xtrabackup_binlog_info" which is very useful in replication setups:

[root@ ~]# cat xtrabackup_binlog_info mysql-bin.000027 191

WHERE:

• mysql-bin.000027: is the binary log file name of the master when the backup created.
• 191: is the binary log position of the backup.

Prepare Full Backup

The backed up files are not ready at the moment to be restored, we must prepare the backup files first as follows:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log /backup/dir/path/full-backup . . . innobackupex: completed OK!

Now, the full backup is ready to be restored ...

Restore Full Backup

To get the full backup restored, the MySQL instance should be stopped first and then one of the following two procedures should be done:

• Using the copy back option: [root@ ~]# innobackupex --user=db_user –-password='db_password' --copy-back /backup/dir/path/full-backup . . . innobackupex: completed OK!
  

  Xtrabackup tool - in this method - will copy all files under the backup folder (full-backup) to the MySQL datadir which must be indicated in the my.cnf file, otherwise, it wouldn't know where the datadir should be placed.

• Using the operating system copy or move commands:
  

  If you don't want to keep the backup files on your local system (you have another copy in an external server), the move command will be very fast to get your backup restored:

  [root@ ~]# mv /backup/dir/path/full-backup /var/lib/mysql
  

As the user who moves/copies the files into MySQL datadir is not "mysql" user, you should make sure that mysql user has the right permissions on its datadir and also the path "/var/lib/mysql" should be replaced with the MySQL datadir if it's set to a different path.

[root@ ~]# chown -R mysql:mysql /var/lib/mysql

After moving/copying the backup files into MySQL datadir, you are free to start the MySQL instance again.

Prepare slave from full backup

Preparing a slave using Xtrabackup is pretty easy and a straight forward process:

• Restore the full backup as explained above.
• Check the binary logs information of the backup:
  [root@ ~]# cat xtrabackup_binlog_info mysql-bin.000027 191
  
• Execute the CHANGE MASTER TO command using the above info and start the slave:
  SQL> CHANGE MASTER TO -> MASTER_HOST='master_ip', -> MASTER_PORT=master_port, -> MASTER_USER='slave_user_name', -> MASTER_PASSWORD='slave_password', -> MASTER_LOG_FILE='mysql-bin.000027', ## taken from xtrabackup_binlog_info -> MASTER_LOG_POS=191; ## taken from xtrabackup_binlog_info SQL> START SLAVE;
  

For more information on how to set up MySQL Replication, check out this nice manual link.

Prepare GTID slave from full backup

GTID is supported in Xtrabackup starting from version 2.1.0. To restore a GTID slave server, the GTID_MODE should be enabled on the master server before creating its backup, otherwise, no GTID values will be included in the "xtrabackup_binlog_info" file under the backup directory. However, the following steps should be done:

• Restore the full backup normally as explained above.
• Check the GTID value of the backup:
  [root@ ~]# cat xtrabackup_binlog_info mysql-bin.000027 191 b9b4712a-df64-11e3-b391-60672090eb04:1-12
  
• Set the variable GTID_PURGED with the GTID value of the backup:
  SQL> SET GLOBAL GTID_PURGED="b9b4712a-df64-11e3-b391-60672090eb04:1-12";
  
• Execute the auto position CHANGE MASTER TO command and start the slave:
  SQL> CHANGE MASTER TO -> MASTER_HOST='master_ip', -> MASTER_PORT=master_port, -> MASTER_USER='slave_user_name', -> MASTER_PASSWORD='slave_password', -> MASTER_AUTO_POSITION = 1; SQL> START SLAVE;
  

For more information on how to set up Transaction-based Replication in MySQL, check out this link.

Advantages / Disadvantages

• Advantages:
  • Fast, simple and easy way to get your full database backed up and restored.
  • All Xtrabackup tool features (like streaming: move the backed up files directly to a remote server) are supported in the full backup method.
  • Simple way to introduce a new slave to the master.
• Disadvantages:
  • We have to replace the entire MySQL datadir with the new one (In other words, the datadir folder should be empty/removed before the restore process).
  • We can't extract one single database or single table from the whole backup (Unless it's MyISAM table), which means that you have to take it all or leave it all.

Important Hints

• The message innobackupex: completed OK! should be printed at the end of every xtrabackup command , otherwise, it would be failed to make a successful command (backup, prepare or restore).
• The ib_logfile files size should be the same in both source and destination servers, if not, you have to either remove them from the backup folder (which will be restored) before starting the MySQL instance and MySQL will create new ones for you OR change those files size in the destination server's configuration file to match the same size in the backup before starting the MySQL instance
• The MySQL user used in the Xtrabackup tool, should have at least the following privileges (RELOAD, LOCK TABLES and REPLICATION CLIENT).
• To prepare a new slave from another slave, just add the two options (“--slave-info" & --safe-slave-backup”) to the backup command and use the information in the file "xtrabackup_slave_info" under the backup folder to issue the "CHANGE MASTER TO" command in the new slave after finishing the restore.
• To Accelerate the preparation process of your backup, just add the option "--use-memory" in the prepare command in order to allocate more used memory (Xtrabackup will use the specified memory as an internal innodb_buffer_pool_size for the prepare process), for ex: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --use-memory=512M /backup/dir/path/full-backup . . . innobackupex: completed OK!
• The preparation process consists of two steps, replaying the committed transactions and rolling back the uncommitted transactions, using the --apply-log option only in the preparation command will do both steps for you.
• The backup folder "/backup/dir/path/full-backup" SHOULD NOT be created before executing the backup command, because Xtrabackup will create that folder for you, and it will fail to continue processing if that folder was already exist.

Incremental Backup

When you have a very large database system, you will need large enough storage to store your database backups in, and if you want to perform a daily backup then your mission will be more difficult. In such cases, the incremental database backup method will be very useful. It allows you to have only the database changes (delta) - after the physical full backup – with the minimum storage space required in a fast way, and hence, you can perform the daily backup operations to your database system without the need to having large storage available.

The following steps describe a simple way to perform your physical incremental database backup using XtraBackup tool:

Create Incremental Backup

To perform an incremental backup, we should first perform a full backup - the same like we did in the previous section - to be the base backup of the upcoming incremental backups.

Creating the full backup (Base Backup):

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp /backup/dir/path/full-backup . . . innobackupex: completed OK!

The "xtrabackup-checkpoints" file contents will be something like:

• backup_type = full-backuped : which indicates the backup type "full backup".
• from_lsn = 0 : which indicates the log sequence number where the backup process started from (0 means from the beginning).
• to_lsn = 3768762 : which indicates the log sequence number where the backup process ended at.

Creating the first incremental backup:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp --incremental /backup/dir/path/inc1 --incremental-basedir=/backup/dir/path/full-backup . . . innobackupex: completed OK!

We informed the Xtrabackup tool to perform an incremental backup by adding the command "--incremental", and by specifying the full-backup path as the basedir, we informed it from which backup it should start tracking the database changes.

The "xtrabackup-checkpoints" file contents will be something like:

• backup_type = incremental : which indicates the backup type "incremental backup".
• from_lsn = 3768762 : which indicates the log sequence number where the backup process started from (the same LSN as the previous full backup ended at).
• to_lsn = 4908762 : which indicates the log sequence number where the backup process ended at.

Creating the second incremental backup:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp --incremental /backup/dir/path/inc2 --incremental-basedir=/backup/dir/path/inc1 . . . innobackupex: completed OK!

We informed the Xtrabackup tool to perform an incremental backup by adding the command "--incremental", and by specifying the 1st incremental backup path as the basedir, we informed it to start tracking the database changes since the last incremental (not the full backup).

The "xtrabackup-checkpoints" file contents will be something like:

• backup_type = incremental : which indicates the backup type "incremental backup".
• from_lsn = 4908762 : which indicates the log sequence number where the backup process started from (the same LSN as the 1st incremental backup ended at).
• to_lsn = 6508762 : which indicates the log sequence number where the backup process ended at.

Note: We can create as many incremental backups as we want by using the same procedure above.

Prepare Incremental Backup

As mentioned earlier in the article, the preparation process consists of two steps (replaying the committed transactions and rolling back the uncommitted transactions) and using the --apply-log option only will do both of them (like we did in the full backup) but in the incremental backups, we MUST do them separately as follows:

• Replay the committed transactions on the base backup (by adding the option "--redo-only"): [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --redo-only /backup/dir/path/full-backup . . . innobackupex: completed OK!
• Replay the committed transactions on the 1st incremental backup: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --redo-only /backup/dir/path/full-backup --incremental-dir=/backup/dir/path/inc1 . . . innobackupex: completed OK!
  

  Note: we specified the full backup folder here, because replaying the committed transactions steps, appends all changes from the incremental backup to the full backup.

• Replay the committed transactions on the 2nd incremental backup: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --redo-only /backup/dir/path/full-backup --incremental-dir=/backup/dir/path/inc2 . . . innobackupex: completed OK!
  

  Note: here, we didn't use the 1st incremental backup folder, because all changes in the 1st incremental was already appended to the full backup in the previous step.

• Finally, roll back all uncommitted transactions: [root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log /backup/dir/path/full-backup . . . innobackupex: completed OK!
  

  Note: as the full backup folder contains all data now (full + 1st & 2nd incremental), there's no need to do this step on the incremental backup folders.

Now, the incremental backup is ready to be restored ...

Restore Incremental Backup

The full backup folder will be the only folder to be restored (there's no need to the incremental backup folders) as it contains all data after appending the changes from all incremental backup. We can restore it the same way we did in the full backup restore.

Advantages / Disadvantages

• Advantages:
  • Less storage resources needed.
  • Faster than the full backup.
• Disadvantages:

  In addition to the disadvantages of the full backup, there are other ones:

  • Complicate and hard process to implement as compared to the full backup.
  • The incremental backup strategy, based on Log Sequence Number which affects only XtraDB and InnoDB storage engines while the others (like MyISAM) will be backed up completely (changed + unchanged data) in each incremental backup process.
  • If we have many incremental backups, appending all of them might consume time and might be confusing as well.
  • If one of the incremental backups become corrupted or not available for any reason, we will not be able to add all incremental backups after that to the full backup.

Important Hints

• The backup preparation sequence steps above, MUST be followed using the same order.
• If the "--redo-only" option was not be used in any of the preparation steps (except the final step), all up coming incremental backups will be useless as we won't be able to add them to the base backup anymore.
• Replaying the committed transactions steps bring all incremental data and append it to the full backup, so that, the rolling back of the uncommitted transactions step should be execute only on the full backup (as it contains already the whole data).).
• In the incremental backups, Xtrabackup generates two files for every table ".delta" & ".meta"(for ex. test.ibd.delta & test.ibd.meta), the delta file size reflects the changes which was applied on that table since the last incremental backup.
• The preparation time of the individual incremental backup will depend on how much data changed there since the last incremental.
• The preparation time for the full backup - in most cases - is really small as compared to the incremental ones because full backups apply the redo logs only while the incremental backups apply the deltas plus the redo logs. So if the delta files are big, the preparation process will take longer.
• Full backups is recommended against Incremental backups if there are many changes applied on the DB, while the incremental backups are recommended when there are few changes applied on the DB.

Differential backup

We can use the incremental backup strategy in order to perform differential backups, but we should consider the following points:

• We always specify the full backup folder as the base backup (in the incremental we specify the previous incremental folder as a base backup)
• All incremental backups between differential and full backups MUST BE ignored when preparing the backup files because the differential backup contains already all changes since the last full backup.
• In the backup preparation process, we should consider the last differential backup as the first incremental backup and all incremental backups after that could be applied normally.

Note: Having differnetial backups in the middle of incremental backups will be useful for many reasons, such as:

• Differential backups reduce the backup preparation steps/time needed because differential backp will replace all its previous incremental backups.
• Differential backups reduce the chances of loosing the incremental backups if we have corrupted incremental backup in the middle, because in this case, differential backup will act as a backup of the previous incremental backups.

Partial Backup

Unlike MyISAM, having physical database backup for a single database or table is not possible if the table engine type is InnoDB. But by using the partial database backup method in the XtraBackup tool, it will be possible to have physical InnoDB tables backup the same like MyISAM ones (but with some restrictions).

The following steps describing how to perform partial database backup using XtraBackup tool:

Create Partial Backup

A simple Xtrabackup command to backup some databases (or tables) should be something like:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --no-timestamp --databases=”db1 db2 db3.tbl1” /backup/dir/path/partial-backup . . . innobackupex: completed OK!
Prepare Partial Backup

The same like the other backup methods, the backed up files are not ready until we get them prepared by adding the "--export" option as follows:

[root@ ~]# innobackupex --user=db_user –-password='db_password' --apply-log --export /backup/dir/path/partial-backup . . . innobackupex: completed OK!

Some errors regarding those not included InnoDB tables from the backup may be appeared, but that's fine. Also there will be a notification of creating the ".exp" file for each table which will be used in the import (restore) process.

Now, the partial backup is ready to be restored ...

Restore Partial Backup

The restore process of the partial backup is quite different than the full and incremental backups.

To restore a partial backup, the following steps should be made:

• Unlike the other methods (Full and Incremental backups), MySQL instance on the destination server shouldn't be stopped because we will have to execute some SQL commands.
• On the destination server, we should create new tables (as many as we have in the partial backup or as we will restore) with the same structure like the one in the partial backup and then discard its table space: mysql> CREATE TABLE db.tbl1 (...)ENGINE=INNODB; mysql> ALTER TABLE db.tbl1 DISCARD TABLESPACE;
• Copy “.ibd” and “.exp” files for each table into the corresponding DB directory then assign the right permissions to mysql user: [root@ ~]# cp /backup/dir/path/partial-backup/db/tbl1.ibd /var/lib/mysql/db [root@ ~]# cp /backup/dir/path/partial-backup/db/tbl1.exp /var/lib/mysql/db [root@ ~]# chown -R mysql:mysql /var/lib/mysql/db
• Now we should tell MySQL to use the new table spaces: mysql> ALTER TABLE db.tbl1 IMPORT TABLESPACE;

Advantages / Disadvantages

• Advantages:
  • Although it's a complicated process, but it allows us to backup and restore individual InnoDB tables the same like MyISAM.
  • Useful when having huge InnoDB tables and we want to backup/restore them only.
• Disadvantages:
  • The streaming feature is not available in the partial backup.
  • Restoring/importing individual tables or databases from a partial backup is not applicable unless the destination server is Percona Server.
  • In addition to restoring the files(copy back), three SQL statements should be executed for each table (table creation + two ALTER statements) in order to get them ready for use, which means that we might do a very boring job (or we have to create a special script) to get the partial backup restored if it contains many tables.

Important Hints

• Although we didn't remove the MySQL datadir before the restore process (like full and incremental backups) as well as having the MySQL instance running, but we can restore the partial backup using the same way (remove the datadir contents and copy/move the backup files to the datadir), but we should take into our consideration that we'll have only the backed up databases/tables and all other databases/tables (which are not included in the partial backup) will be missed.
• “innodb_file_per_table” server option must be enabled (in both source and destination servers).
• "innodb_expand_import" option must be enabled in the destination server which is available only in Percona server (and that explain why we can restore partial backup on Percona server only).
• Beside the "--databases" option, other two alternative options to perform the same needs could be used but we must provide each table with the fully qualified naming format:
  • --include='db.tbl'.
  • --tables-file=/path/to/file.txt ==> in that file, we can add multiple tables one per line in the fully qualified naming format.

Now, you can use the Xtrabackup tool to perform full, incremental and partial database backups, you can decide which method(s) of them are suitable for you according to the advantages and disadvantages of each one, and by considering the important hints for each method you can perform your backup efficiently.

I hope you found this article useful for you and to be familiar with such wonderful tool.

Table of Content

• Introduction
• Prerequisites
  • System information
  • Installed packages
  • Required packages
• Upgrade the first node
  • Upgrade the installed binaries
  • Upgrade mysql schema
  • Prepare the node to join the cluster
  • Joining the cluster
• Rolling upgrade the other nodes
• Get rid of old release option

Introduction

Codership announced from weeks ago introducing the Galera Cluster new release 3.0 having many bug fixes, performance enhancements plus the main purpose which is working with MySQL 5.6. In this article, I'll go through the upgrade steps from Galera 2.x to the new release 3.0, but at the time of writing this article - as mentioned in the Codership release notes - THIS IS A BETA QUALITY RELEASE FOR TESTING PURPOSES. NOT RECOMMENDED FOR PRODUCTION YET.

Important note: a new Galera version (3.1) will be available soon for production and it will be INCOMPATIBLE with this beta version (3.0) but still compatible with 2.x, so again DO NOT go for production using 3.0 and postpone the production upgrade process until 3.1 become available.

Prerequisites System information

The following are the cluster system information:

• Operating System: CentOS release 6.4 (64 bit)
• Cluster system consists of 3 cluster nodes (192.168.1.251 "gcservera",192.168.1.252 "gcserverb" & 192.168.1.253 "gcserverc")

Installed packages

The following are the packages installed on the three cluster nodes:

• MySQL version: mysql-5.5.33_wsrep_23.7.6 (RPM)
• Galera provider version:galera-23.2.7 (RPM)

mysql> show global variables like'%version%'; +-------------------------+--------------------------------------------------+ | Variable_name | Value | +-------------------------+--------------------------------------------------+ | innodb_version | 5.5.33 | | protocol_version | 10 | | slave_type_conversions | | | version | 5.5.33-log | | version_comment | MySQL Community Server (GPL), wsrep_23.7.6.r3919 | | version_compile_machine | x86_64 | | version_compile_os | Linux | +-------------------------+--------------------------------------------------+ 7 rows in set (0.03 sec) Required packages

The following are the needed packages to be installed:

• MySQL 5.6 + Galera plugin: Could be downloaded from here.
• Galera provider 3.0: Could be downloaded from here

Upgrade the first node Upgrade the installed binaries

To upgrade the installed binaries, you have to stop the mysqld first.

Important: If you are using a load balancer in your cluster system, you should bring the node in question out from the load balancer before stopping the mysqld.

[root@gcservera ~]# /etc/init.d/mysql stop

And then, upgrade with the binaries:

[root@gcservera ~]# rpm -qa|grep MySQL MySQL-server-5.5.33_wsrep_23.7.6-1.rhel6.x86_64 MySQL-client-5.5.34-1.el6.x86_64 [root@gcservera ~]# rpm -e MySQL-server-5.5.33_wsrep_23.7.6-1.rhel6.x86_64 [root@gcservera ~]# rpm -ivh /downloads/MySQL-server-5.6.13_wsrep_24.0-1.rhel6.x86_64.rpm Preparing... ########################################### [100%] 1:MySQL-server ########################################### [100%] [root@gcservera ~]# rpm -U /downloads/galera-24.3.0-1.rhel6.x86_64.rpm Upgrade mysql schema

It's recommended to perform the mysql schema upgrade before joining the cluster, so that, we need to start the node first as a standalone instance by disabling the provider option:

#my.cnf [mysqld] . . #wsrep_provider=/usr/lib64/galera/libgalera_smm.so

Then start the instance and perform the upgrade using the mysql_upgrade utility:

[root@gcservera ~]# /etc/init.d/mysql start Starting MySQL........... SUCCESS! [root@gcservera ~]# mysql_upgrade Prepare the node to join the cluster

All running nodes (second and third nodes) are using the old galera version (2.7) at the moment, so that a backward compatibility option (wsrep_provider_options="socket.checksum=1") MUST be added in the node's configuration in order to join the cluser, otherwise, it will fail to join it back.

Also don't forget to enable again the provider option:

#my.cnf [mysqld] . . wsrep_provider=/usr/lib64/galera/libgalera_smm.so wsrep_provider_options="socket.checksum=1" Joining the cluster

Now we are ready to join the cluster by restarting the node

[root@gcservera ~]# /etc/init.d/mysql restart Stopping MySQL........... SUCCESS! Starting MySQL........... SUCCESS!

We can check the new version as follows:

mysql> show global variables like'%version%'; +-------------------------+------------------------------------------------+ | Variable_name | Value | +-------------------------+------------------------------------------------+ | innodb_version | 5.6.13 | | protocol_version | 10 | | slave_type_conversions | | | version | 5.6.13-log | | version_comment | MySQL Community Server (GPL), wsrep_24.0.r3937 | | version_compile_machine | x86_64 | | version_compile_os | Linux | +-------------------------+------------------------------------------------+ 7 rows in set (0.05 sec)

And the cluster status as well:

mysql> show global status like'wsrep%'; +----------------------------+--------------------------------------+ | Variable_name | Value | +----------------------------+--------------------------------------+ | wsrep_local_state_uuid | ac53dc1e-3aff-11e3-b970-eb7044f6dc77 | . . | wsrep_local_state_comment | Synced | . . | wsrep_cluster_size | 3 | | wsrep_cluster_state_uuid | ac53dc1e-3aff-11e3-b970-eb7044f6dc77 | | wsrep_cluster_status | Primary | | wsrep_connected | ON | | wsrep_local_index | 2 | | wsrep_provider_name | Galera | | wsrep_provider_vendor | Codership Oy | | wsrep_provider_version | 24.3.0(r159) | | wsrep_ready | ON | +----------------------------+--------------------------------------+ 43 rows in set (0.13 sec)

Important: if you are using a load balancer in the cluster, it's now the time to add this node back again to it.

Rolling upgrade the other nodes

You can start doing the rolling upgrade to the other nodes the same like the first one but after making sure that all nodes state (wsrep_local_state_comment) is Synced. If you stopped a node while it's in the Donor state, then the donor and the joiner nodes might be crashed, so make sure of that first.

Get rid of the old release option

After making the upgrade on all the cluster nodes, the backward compatibility option (wsrep_provider_options="socket.checksum=1") is not needed anymore, so removing it from the configuration files and doing a rolling restart on all nodes will do the mission.

Have fun with the new MySQL and Galera releases :)

Taxonomy upgrade extras: upgradegalera

Table of Content

• Introduction
• Prerequisites
  • System information
  • Installed packages
  • Required packages
• Upgrade the first node
  • Upgrade the installed binaries
  • Upgrade mysql schema
  • Prepare the node to join the cluster
  • Joining the cluster
• Rolling upgrade the other nodes
• Get rid of old release option

Introduction

Codership announced from weeks ago introducing the Galera Cluster new release 3.0 having many bug fixes, performance enhancements plus the main purpose which is working with MySQL 5.6. In this article, I'll go through the upgrade steps from Galera 2.x to the new release 3.0, but at the time of writing this article - as mentioned in the Codership release notes - THIS IS A BETA QUALITY RELEASE FOR TESTING PURPOSES. NOT RECOMMENDED FOR PRODUCTION YET.

Important note: a new Galera version (3.1) will be available soon for production and it will be INCOMPATIBLE with this beta version (3.0) but still compatible with 2.x, so again DO NOT go for production using 3.0 and postpone the production upgrade process until 3.1 become available.

Prerequisites System information

The following are the cluster system information:

• Operating System: CentOS release 6.4 (64 bit)
• Cluster system consists of 3 cluster nodes (192.168.1.251 "gcservera",192.168.1.252 "gcserverb" & 192.168.1.253 "gcserverc")

Installed packages

The following are the packages installed on the three cluster nodes:

• MySQL version: mysql-5.5.33_wsrep_23.7.6 (RPM)
• Galera provider version:galera-23.2.7 (RPM)

mysql> show global variables like'%version%'; +-------------------------+--------------------------------------------------+ | Variable_name | Value | +-------------------------+--------------------------------------------------+ | innodb_version | 5.5.33 | | protocol_version | 10 | | slave_type_conversions | | | version | 5.5.33-log | | version_comment | MySQL Community Server (GPL), wsrep_23.7.6.r3919 | | version_compile_machine | x86_64 | | version_compile_os | Linux | +-------------------------+--------------------------------------------------+ 7 rows in set (0.03 sec) Required packages

The following are the needed packages to be installed:

• MySQL 5.6 + Galera plugin: Could be downloaded from here.
• Galera provider 3.0: Could be downloaded from here

Upgrade the first node Upgrade the installed binaries

To upgrade the installed binaries, you have to stop the mysqld first.

Important: If you are using a load balancer in your cluster system, you should bring the node in question out from the load balancer before stopping the mysqld.

[root@gcservera ~]# /etc/init.d/mysql stop

And then, upgrade with the binaries:

[root@gcservera ~]# rpm -qa|grep MySQL MySQL-server-5.5.33_wsrep_23.7.6-1.rhel6.x86_64 MySQL-client-5.5.34-1.el6.x86_64 [root@gcservera ~]# rpm -e MySQL-server-5.5.33_wsrep_23.7.6-1.rhel6.x86_64 [root@gcservera ~]# rpm -ivh /downloads/MySQL-server-5.6.13_wsrep_24.0-1.rhel6.x86_64.rpm Preparing... ########################################### [100%] 1:MySQL-server ########################################### [100%] [root@gcservera ~]# rpm -U /downloads/galera-24.3.0-1.rhel6.x86_64.rpm Upgrade mysql schema

It's recommended to perform the mysql schema upgrade before joining the cluster, so that, we need to start the node first as a standalone instance by disabling the provider option:

#my.cnf [mysqld] . . #wsrep_provider=/usr/lib64/galera/libgalera_smm.so

Then start the instance and perform the upgrade using the mysql_upgrade utility:

[root@gcservera ~]# /etc/init.d/mysql start Starting MySQL........... SUCCESS! [root@gcservera ~]# mysql_upgrade Prepare the node to join the cluster

All running nodes (second and third nodes) are using the old galera version (2.7) at the moment, so that a backward compatibility option (wsrep_provider_options="socket.checksum=1") MUST be added in the node's configuration in order to join the cluser, otherwise, it will fail to join it back.

Also don't forget to enable again the provider option:

#my.cnf [mysqld] . . wsrep_provider=/usr/lib64/galera/libgalera_smm.so wsrep_provider_options="socket.checksum=1" Joining the cluster

Now we are ready to join the cluster by restarting the node

[root@gcservera ~]# /etc/init.d/mysql restart Stopping MySQL........... SUCCESS! Starting MySQL........... SUCCESS!

We can check the new version as follows:

mysql> show global variables like'%version%'; +-------------------------+------------------------------------------------+ | Variable_name | Value | +-------------------------+------------------------------------------------+ | innodb_version | 5.6.13 | | protocol_version | 10 | | slave_type_conversions | | | version | 5.6.13-log | | version_comment | MySQL Community Server (GPL), wsrep_24.0.r3937 | | version_compile_machine | x86_64 | | version_compile_os | Linux | +-------------------------+------------------------------------------------+ 7 rows in set (0.05 sec)

And the cluster status as well:

mysql> show global status like'wsrep%'; +----------------------------+--------------------------------------+ | Variable_name | Value | +----------------------------+--------------------------------------+ | wsrep_local_state_uuid | ac53dc1e-3aff-11e3-b970-eb7044f6dc77 | . . | wsrep_local_state_comment | Synced | . . | wsrep_cluster_size | 3 | | wsrep_cluster_state_uuid | ac53dc1e-3aff-11e3-b970-eb7044f6dc77 | | wsrep_cluster_status | Primary | | wsrep_connected | ON | | wsrep_local_index | 2 | | wsrep_provider_name | Galera | | wsrep_provider_vendor | Codership Oy | | wsrep_provider_version | 24.3.0(r159) | | wsrep_ready | ON | +----------------------------+--------------------------------------+ 43 rows in set (0.13 sec)

Important: if you are using a load balancer in the cluster, it's now the time to add this node back again to it.

Rolling upgrade the other nodes

You can start doing the rolling upgrade to the other nodes the same like the first one but after making sure that all nodes state (wsrep_local_state_comment) is Synced. If you stopped a node while it's in the Donor state, then the donor and the joiner nodes might be crashed, so make sure of that first.

Get rid of the old release option

After making the upgrade on all the cluster nodes, the backward compatibility option (wsrep_provider_options="socket.checksum=1") is not needed anymore, so removing it from the configuration files and doing a rolling restart on all nodes will do the mission.

Have fun with the new MySQL and Galera releases :)

Table of Content

• Introduction
• Prerequisites
  • System information
  • Installed packages
  • Required packages
• Upgrade the first node
  • Upgrade the installed binaries
  • Upgrade mysql schema
  • Prepare the node to join the cluster
  • Joining the cluster
• Rolling upgrade the other nodes
• Get rid of old release option

Introduction

Codership announced from weeks ago introducing the Galera Cluster new release 3.0 having many bug fixes, performance enhancements plus the main purpose which is working with MySQL 5.6. In this article, I'll go through the upgrade steps from Galera 2.x to the new release 3.0, but at the time of writing this article - as mentioned in the Codership release notes - THIS IS A BETA QUALITY RELEASE FOR TESTING PURPOSES. NOT RECOMMENDED FOR PRODUCTION YET.

Important note: a new Galera version (3.1) will be available soon for production and it will be INCOMPATIBLE with this beta version (3.0) but still compatible with 2.x, so again DO NOT go for production using 3.0 and postpone the production upgrade process until 3.1 become available.

Prerequisites System information

The following are the cluster system information:

• Operating System: CentOS release 6.4 (64 bit)
• Cluster system consists of 3 cluster nodes (192.168.1.251 "gcservera",192.168.1.252 "gcserverb" & 192.168.1.253 "gcserverc")

Installed packages

The following are the packages installed on the three cluster nodes:

• MySQL version: mysql-5.5.33_wsrep_23.7.6 (RPM)
• Galera provider version:galera-23.2.7 (RPM)

mysql> show global variables like'%version%'; +-------------------------+--------------------------------------------------+ | Variable_name | Value | +-------------------------+--------------------------------------------------+ | innodb_version | 5.5.33 | | protocol_version | 10 | | slave_type_conversions | | | version | 5.5.33-log | | version_comment | MySQL Community Server (GPL), wsrep_23.7.6.r3919 | | version_compile_machine | x86_64 | | version_compile_os | Linux | +-------------------------+--------------------------------------------------+ 7 rows in set (0.03 sec) Required packages

The following are the needed packages to be installed:

• MySQL 5.6 + Galera plugin: Could be downloaded from here.
• Galera provider 3.0: Could be downloaded from here

Upgrade the first node Upgrade the installed binaries

To upgrade the installed binaries, you have to stop the mysqld first.

Important: If you are using a load balancer in your cluster system, you should bring the node in question out from the load balancer before stopping the mysqld.

[root@gcservera ~]# /etc/init.d/mysql stop

And then, upgrade with the binaries:

[root@gcservera ~]# rpm -qa|grep MySQL MySQL-server-5.5.33_wsrep_23.7.6-1.rhel6.x86_64 MySQL-client-5.5.34-1.el6.x86_64 [root@gcservera ~]# rpm -e MySQL-server-5.5.33_wsrep_23.7.6-1.rhel6.x86_64 [root@gcservera ~]# rpm -ivh /downloads/MySQL-server-5.6.13_wsrep_24.0-1.rhel6.x86_64.rpm Preparing... ########################################### [100%] 1:MySQL-server ########################################### [100%] [root@gcservera ~]# rpm -U /downloads/galera-24.3.0-1.rhel6.x86_64.rpm Upgrade mysql schema

It's recommended to perform the mysql schema upgrade before joining the cluster, so that, we need to start the node first as a standalone instance by disabling the provider option:

#my.cnf [mysqld] . . #wsrep_provider=/usr/lib64/galera/libgalera_smm.so

Then start the instance and perform the upgrade using the mysql_upgrade utility:

[root@gcservera ~]# /etc/init.d/mysql start Starting MySQL........... SUCCESS! [root@gcservera ~]# mysql_upgrade Prepare the node to join the cluster

All running nodes (second and third nodes) are using the old galera version (2.7) at the moment, so that a backward compatibility option (wsrep_provider_options="socket.checksum=1") MUST be added in the node's configuration in order to join the cluser, otherwise, it will fail to join it back.

Also don't forget to enable again the provider option:

#my.cnf [mysqld] . . wsrep_provider=/usr/lib64/galera/libgalera_smm.so wsrep_provider_options="socket.checksum=1" Joining the cluster

Now we are ready to join the cluster by restarting the node

[root@gcservera ~]# /etc/init.d/mysql restart Stopping MySQL........... SUCCESS! Starting MySQL........... SUCCESS!

We can check the new version as follows:

mysql> show global variables like'%version%'; +-------------------------+------------------------------------------------+ | Variable_name | Value | +-------------------------+------------------------------------------------+ | innodb_version | 5.6.13 | | protocol_version | 10 | | slave_type_conversions | | | version | 5.6.13-log | | version_comment | MySQL Community Server (GPL), wsrep_24.0.r3937 | | version_compile_machine | x86_64 | | version_compile_os | Linux | +-------------------------+------------------------------------------------+ 7 rows in set (0.05 sec)

And the cluster status as well:

mysql> show global status like'wsrep%'; +----------------------------+--------------------------------------+ | Variable_name | Value | +----------------------------+--------------------------------------+ | wsrep_local_state_uuid | ac53dc1e-3aff-11e3-b970-eb7044f6dc77 | . . | wsrep_local_state_comment | Synced | . . | wsrep_cluster_size | 3 | | wsrep_cluster_state_uuid | ac53dc1e-3aff-11e3-b970-eb7044f6dc77 | | wsrep_cluster_status | Primary | | wsrep_connected | ON | | wsrep_local_index | 2 | | wsrep_provider_name | Galera | | wsrep_provider_vendor | Codership Oy | | wsrep_provider_version | 24.3.0(r159) | | wsrep_ready | ON | +----------------------------+--------------------------------------+ 43 rows in set (0.13 sec)

Important: if you are using a load balancer in the cluster, it's now the time to add this node back again to it.

Rolling upgrade the other nodes

You can start doing the rolling upgrade to the other nodes the same like the first one but after making sure that all nodes state (wsrep_local_state_comment) is Synced. If you stopped a node while it's in the Donor state, then the donor and the joiner nodes might be crashed, so make sure of that first.

Get rid of the old release option

After making the upgrade on all the cluster nodes, the backward compatibility option (wsrep_provider_options="socket.checksum=1") is not needed anymore, so removing it from the configuration files and doing a rolling restart on all nodes will do the mission.

Have fun with the new MySQL and Galera releases :)

We had a Galera Cluster support case recently. The customer was drenched in tears because his Galera Cluster did not work any more and he could not make it work any more.

Upsss! What has happened?

A bit of the background of this case: The customer wanted to do a rolling-restart of the Galera Cluster under load because of an Operating System upgrade which requires a reboot of the system.

Lets have a look at the MySQL error log to see what was going on. Customer restarted server with NodeC:

12:20:42 NodeC: normal shutdown --> Group 2/2 12:20:46 NodeC: shutdown complete 12:22:09 NodeC: started 12:22:15 NodeC: start replication 12:22:16 NodeC: CLOSED -> OPEN 12:22:16 all : Group 2/3 component all PRIMARY 12:22:17 NodeC: Gap in state sequence. Need state transfer. 12:22:18 all : Node 1 (NodeC) requested state transfer from '*any*'. Selected 0 (NodeB)(SYNCED) as donor. 12:22:18 NodeB: Shifting SYNCED -> DONOR/DESYNCED (TO: 660966498) 12:22:19 NodeC: Shifting PRIMARY -> JOINER (TO: 660966498) 12:22:19 NodeC: Receiving IST: 14761 writesets, seqnos 660951493-660966254 12:22:21 NodeC: 0 (NodeB): State transfer to 1 (NodeC) complete.

Everything went fine so far NodeC came up again and did an IST as expected. But then the first operational error happened: The customer did not wait to reboot NodeB until NodeC was completely recovered. It seems like NodeC took some time for the IST recovery. This should be checked on all nodes with SHOW GLOBAL STATUS LIKE 'wsrep%';...

12:22:21 NodeC: Member 0 (NodeB) synced with group. 12:22:21 NodeB: Shifting JOINED -> SYNCED (TO: 660966845) 12:22:21 NodeB: Synchronized with group, ready for connections --> NodeC seems not to be ready yet! 12:23:21 NodeB: Normal shutdown 12:23:21 all : Group 1/2 12:23:21 NodeC: Aborted (core dumped)

And now Murphy was acting already the first time: We hit a situation in the Galera Cluster which is not covered as expected. Now we have 2 nodes out of 3 not working. As a result the Cluster gets a quorum loss (non-Primary, more than 50% of nodes disappeared) and does not reply to any SQL queries any more. This is a bug because both nodes left the cluster gracefully. The third node should have stayed primary:

12:23:21 NodeB: Received SELF-LEAVE. Closing connection. 12:23:23 NodeB: Shifting CLOSED -> DESTROYED (TO: 660973981) 12:23:25 NodeB: Shutdown complete 12:23:29 NodeC: mysqld_safe WSREP: sleeping 15 seconds before restart 12:23:37 NodeA: Received NON-PRIMARY. 12:23:44 NodeC: mysqld_safe mysqld restarted 12:23:48 NodeC: Shifting CLOSED -> OPEN 12:23:48 NodeC: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 2 12:23:48 NodeC: Received NON-PRIMARY. 12:23:48 NodeA: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 2 12:23:48 NodeA: Received NON-PRIMARY. 12:24:30 NodeB: mysqld_safe Starting mysqld daemon 12:24:36 NodeB: Start replication 12:24:37 NodeB: Received NON-PRIMARY.

As a result the customer decided to shutdown the whole cluster. Which was not necessary but is a acceptable approach:

12:27:55 NodeB: /usr/sbin/mysqld: Normal shutdown 12:27:58 NodeB: /usr/sbin/mysqld: Shutdown complete 12:28:14 NodeA: /usr/sbin/mysqld: Normal shutdown 12:28:19 NodeA: /usr/sbin/mysqld: Shutdown complete 12:31:45 NodeC: /usr/sbin/mysqld: Normal shutdown 12:31:49 NodeC: /usr/sbin/mysqld: Shutdown complete

We experience a complete cluster outage now. An then the next operational error happened: The customer has chosen the node (NodeC) with the worst (= oldest) data as the starting node for the new Cluster:

12:31:55 NodeC: Starting mysqld daemon 12:31:58 NodeC: PRIMARY, 1/1 12:31:58 NodeC: /usr/sbin/mysqld: ready for connections. 12:33:29 NodeB: mysqld_safe Starting mysqld daemon 12:33:33 NodeB: PRIMARY, 1/2

An alternative approach would have been to run the command SET GLOBAL wsrep_provider_options='pc.bootstrap=yes'; on the node (NodeA) with the most recent data...
After connecting NodeB (with the newer state) requested an state transfer from the older NodeC:

12:33:35 all : Node 1 (NodeB) requested state transfer from '*any*'. Selected 0 (NodeC)(SYNCED) as donor. 12:33:35 NodeC: Shifting SYNCED -> DONOR/DESYNCED (TO: 660982149) 12:33:35 NodeC: IST request --> Should be SST, why IST? 12:33:35 NodeB: Shifting PRIMARY -> JOINER (TO: 660982149) 12:33:35 NodeB: Receiving IST: 7914 writesets, seqnos 660973981-660981895 12:33:36 NodeB: Slave SQL: Could not execute Write_rows event on table test.test; Duplicate entry '8994678' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 102, Error_code: 1062

And now Mister Murphy is acting a second time: We hit another situation: The newer node requests an IST from the older node which has progressed in the meanwhile to an even newer state. So the newer joiner node receives data from the older donor node which causes an AUTO_INCREMENT Primary Key violation. As a consequence the node crashes:

12:33:36 NodeB: receiving IST failed, node restart required: Failed to apply app buffer: äJR#, seqno: 660974010, status: WSREP_FATAL 12:33:36 NodeB: Closed send monitor. 12:33:37 NodeB: Closing slave action queue. 12:33:37 NodeB: Aborted (core dumped) 12:33:37 NodeC: PRIMARY 1/1 12:33:44 NodeC: Shifting DONOR/DESYNCED -> JOINED (TO: 660983204) 12:33:59 NodeB: mysqld_safe mysqld restarted 12:34:04 NodeB: Shifting CLOSED -> OPEN 12:34:07 NodeB: Aborted (core dumped) ... Loop

This situation keeps the node NodeB now in a crashing loop. Restarted by the mysqld_safe process requesting an IST. This is another bug which is fixed in a newer Galera MySQL (5.5.33). And now the next operational error happened: Instead of killing NodeB and forcing an SST by deleting the grastat.dat file They started the third node as well...

12:37:12 NodeA: mysqld_safe Starting mysqld daemon ... --> code dumped ... Loop

NodeB and NodeA both have the same problem now...

As a result: Node NodeA and NodeB are now looping in a crash. But at least the node NodeC was up and running all the time.

Learnings

• Most important: Have an ntpd service running on all Cluster nodes to not mess up the times on different nodes while investigating in errors. This makes problem solving much easier...
• In case of split-brain or quorum loss choose the node with the most recent data as your initial Cluster node.
• If you have a Cluster in split-brain you do not have to restart it. You can bring the node out of split-brain with pc.bootstrap=yes variable if you found out which node is the most recent one.
• Analyse error log files carefully to understand what went wrong. Forcing an SST only takes a few seconds.
• Upgrade your software regularly to not hit old known bugs. The rule Do not touch a running system! does not apply here because we are already touching the running system! So regular upgrade from time to time can be very helpful!
• Be familiar with operational issues of your Cluster software. A Cluster does not only mean high-availability. It means also you have to train your staff to handle it.
• It is always valuable to have support for your business critical systems.

In MySQL we have the typical behaviour that we open and close connections very often and rapidly. So we have very short-living connections to the server. This can lead in extreme cases to the situation that the maximum number of TCP ports are exhausted.

The maximum number of TCP ports we can find with:

# cat /proc/sys/net/ipv4/ip_local_port_range 32768 61000

In this example we can have in maximum (61000 - 32768 = 28232) connections concurrently open.

When a TCP connections closes the port cannot be reused immediately afterwards because the Operating System has to wait for the duration of the TIME_WAIT interval (maximum segment lifetime, MSL). This we can see with the command:

# netstat -nat Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:10051 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:10051 127.0.0.1:60756 TIME_WAIT tcp 0 0 127.0.0.1:10050 127.0.0.1:50191 TIME_WAIT tcp 0 0 127.0.0.1:10050 127.0.0.1:52186 ESTABLISHED tcp 0 0 127.0.0.1:10051 127.0.0.1:34445 TIME_WAIT

The reason for waiting is that packets may arrive out of order or be retransmitted after the connection has been closed. CLOSE_WAIT indicates that the other side of the connection has closed the connection. TIME_WAIT indicates that this side has closed the connection. The connection is being kept around so that any delayed packets can be matched to the connection and handled appropriately.

The Maximum Segment Lifetime can be found as follows:

# cat /proc/sys/net/ipv4/tcp_fin_timeout 60

This basically means your system cannot guarantee more than ((61000 - 32768) / 60 = 470) ports at any given time.

Solutions

There are several strategies out of this problem:

• Open less frequently connections to your MySQL database. Put more payload into one connection. Often Connection Pooling is used to achieve this.
• Increasing the port range. Setting the range to 15000 61000 is pretty common these days (extreme tuning: 1024 - 65535).
• Increase the availability by decreasing the FIN timeout.

Those values can be changed online with:

# echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout # echo 15000 65000 > /proc/sys/net/ipv4/ip_local_port_range

Or permanently by adding it to /etc/sysctl.conf

An other possibility to change this behaviour is to use tcp_tw_recycle and tcp_tw_reuse. By default they are disabled:

# cat /proc/sys/net/ipv4/tcp_tw_recycle 0 # cat /proc/sys/net/ipv4/tcp_tw_reuse 0

These parameters allow fast cycling of sockets in TIME_WAIT state and re-using them. But before you do this change make sure that this does not conflict with the protocols that you would use for the application that needs these ports.

The tcp_tw_recycle could cause some problems when using load balancers:

tcp_tw_reuse Allow to reuse TIME_WAIT sockets for new connections when it is safe from protocol viewpoint. Default value is 0.
It should not be changed without advice/request of technical experts. tcp_tw_recycle Enable fast recycling TIME_WAIT sockets. Default value is 0. It should not be changed without advice/request of technical experts.
Literature

• TCP State Transition Diagram
• Transmission Control Protocol
• TIME_WAIT and its design implications for protocols and scalable client server systems
• Linux documentation about network parameters

Taxonomy upgrade extras: galerastartstop

It took me quite a while to find out how the beast Galera Arbitrator (garbd) works. To safe your time here a short summary:

How to start Galera Arbitrator (garbd) shell> ./garbd --address gcomm://192.168.13.1,192.168.13.2 --group "Our Galera Cluster" --log /tmp/garbd.log --daemon
How to stop Galera Arbitrator (gardb) shell> killall garbd
How to start Galera Arbitrator (garbd) with a configuration file shell>./garbd --cfg /tmp/garb.cnf --daemon

The configuration file looks as follows:

# # /etc/mysql/garb.cnf # address = gcomm://127.0.0.1:5671,127.0.0.1:5672,127.0.0.1:5673 group = Our Galera Cluster options = gmcast.listen_addr=tcp://127.0.0.1:5674 log = /tmp/garbd.log

A service start/stop script can be found at: galera-src/garb/files/agrb.sh and galera-src/garb/files/garb.cnf

Our bigger customers where we help to deploy Galera Cluster for MySQL set-ups have some commercial hardware (e.g. F5 or Cisco) for load balancing instead of software load balancers.

For those hardware load balancer it is not possible to see if a Galera node is available or not because the MySQL daemon is still running and responding on port 3306 but the service is not available nonetheless.
So the load balancer still serves the Galera node while he feeds for example a joiner node with a SST. This would lead to application errors which is unlovely.

One can try somehow to teach the load balancer to find out if a Galera Cluster node is really available or not. But this requires a more sophisticated load balancer, know-how how to teach the load balancer the new behaviour and possible interaction between the MySQL node and the load balancer. See our other discussion for this mater.

An other concept we hit on this week is that we could also block the port 3306 of the MySQL node with firewall rules (iptables). Then the hardware load balancer does not see anybody listening on port 3306 any more and assumes that this IP address should not be served any more.

We also learned this week that the REJECT rule is better than the DROP rule when we want to have fast response time for immediate elimination of traffic.

The script block_galera_node.sh has to be hooked as before into the wsrep_notify_cmd variable and an additional sudoers rule has to be added for the mysql user.

# # /etc/sudoers.d/mysql # chmod 0440 # mysql ALL = (root) NOPASSWD: /sbin/iptables

We are interested to hear your experience and your opinion about this approach.

Recently a forum question [ 1 ] got my attention:

Is there any performance issue with Union?

I used union all sometime back and it was performance issue just to make an opinion that we should used union in query.

The question itself was not too interesting because the answer is easy: It depends. But I wanted to see if there was an improvement in this common problem over time in MySQL.

Test set-up

So I prepared a little test to simulate some of the possible scenarios:

CREATE TABLE `u` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `a` int(10) unsigned DEFAULT NULL, `b` int(10) unsigned DEFAULT NULL, `c` int(10) unsigned DEFAULT NULL, `d` int(10) unsigned DEFAULT NULL, PRIMARY KEY (`id`), KEY `a` (`a`), KEY `b` (`b`), KEY `c` (`c`), KEY `d` (`d`) ) ENGINE=InnoDB ; INSERT INTO u SELECT NULL, ROUND(RAND()*10, 0), ROUND(RAND()*10, 0), ROUND(RAND()*1000000, 0), ROUND(RAND()*1000000, 0); INSERT INTO u SELECT NULL, ROUND(RAND()*10, 0), ROUND(RAND()*10, 0), ROUND(RAND()*1000000, 0), ROUND(RAND()*1000000, 0) FROM u; ... 1 mio rows ANALYZE TABLE u;

With this table we can simulate the OR problem with low and high selectivity.

Running the tests

We did the tests with MySQL (5.0 - 5.7), Percona Server (5.6) and MariaDB (5.5, 10.0) for the following queries:

EXPLAIN SELECT * FROM u WHERE a = 5 OR b = 5; EXPLAIN SELECT * FROM u WHERE a = 5 OR c = 500001; EXPLAIN SELECT * FROM u WHERE c = 500001 OR d = 500001;

We are interested in what the optimizer is doing and what the performance of the queries is. The following results came out:

Query 1 Query 2 Query 3 Database version rows avg. time QEP rows avg. time QEP rows avg. time QEP MySQL 5.0.92 194402 390 ms 1 104876 230 ms 2 6 < 10 ms 3 MySQL 5.1.66 194402 410 ms 1 104876 240 ms 2 6 < 10 ms 3 MySQL 5.5.24 194402 420 ms 1 104876 370 ms 1 6 < 10 ms 3 MariaDB 5.5.32 194402 460 ms 1 104876 420 ms 1 6 < 10 ms 3 MySQL 5.6.12 194402 440 ms 2 104876 240 ms 2 6 < 10 ms 3 Percona 5.6.12-60.40 194402 450 ms 2 104876 240 ms 2 6 < 10 ms 3 MySQL 5.7.1 194402 420 ms 2 104876 220 ms 2 6 < 10 ms 3 MariaDB 10.0.3 194402 450 ms 1 104876 400 ms 1 6 < 10 ms 3 Different Query Execution Plans (QEP)

• QEP 1: +----+-------------+-------+------+---------------+------+---------+------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+---------------+------+---------+------+---------+-------------+ | 1 | SIMPLE | u | ALL | a,b | NULL | NULL | NULL | 1049134 | Using where | +----+-------------+-------+------+---------------+------+---------+------+---------+-------------+
  
• QEP 2: +----+-------------+-------+-------------+---------------+------+---------+------+--------+-------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+-------------+---------------+------+---------+------+--------+-------------------------------+ | 1 | SIMPLE | u | index_merge | a,c | a,c | 5,5 | NULL | nnnnnn | Using union(a,c); Using where | +----+-------------+-------+-------------+---------------+------+---------+------+--------+-------------------------------+
  
• QEP 3: +----+-------------+-------+-------------+---------------+------+---------+------+------+-------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+-------------+---------------+------+---------+------+------+-------------------------------+ | 1 | SIMPLE | u | index_merge | c,d | c,d | 5,5 | NULL | n | Using union(c,d); Using where | +----+-------------+-------+-------------+---------------+------+---------+------+------+-------------------------------+
  
Conclusion
• Single query performance went down from 5 - 50% (in one case increased by 5%) over time (MySQL releases). But we can see some impacts on optimizer improvements.
• Newer MySQL releases are not necessarily faster for single-query performance than older ones. Most of the MySQL users are not running more than 1 or 2 concurrent queries. For them scalability improvements are not really an issue.
• There seems to be some changes in the Optimizer some for good, some for bad, depending on the release or branch/fork you are using. So test carefully when you change the release or branch/fork.
• And: Do not believe the whole marketing yelling but do your own testing...

Maybe you experienced some strange behaviour with MySQL: Everything is installed correctly and should work. But it does not.

Symptoms we have seen:

• MySQL starts/stops properly when started/stopped with service mysqld restart but MySQL does not start when a server is rebooted.
• Or after upgrading MySQL binaries mysqld will not start at all any more.
• Or after relocating MySQL datadir or changing default port MySQL does not start any more.

shell> service mysqld start MySQL Daemon failed to start. Starting mysqld: [FAILED] shell> grep mysqld /var/log/boot.log Starting mysqld: [FAILED]

If you are lucky you get some error message like: ERROR! The server quit without updating PID file (/data/mysql/server.pid). or:

130620 9:49:14 [ERROR] Can't start server : Bind on unix socket: Permission denied 130620 9:49:14 [ERROR] Do you already have another mysqld server running on socket: /var/lib/mysql/mysql.sock ? 130620 9:49:14 [ERROR] Aborting

This typically happens when you relocate the MySQL data files (datadir), change port, socket, log file, pid file or similar.

The reason for this problem is not too easy to find. You see some traces in /var/log/boot.log. And if you know where to look for you will find something in /var/log/audit/audit.log. But without knowing where to look and what to look for it is quite hard.
If you are lucky the setroubleshoot utility is installed. This will report problems in the syslog (/var/log/messages).

The cause of this problem might be the Secure Linux (SELinux) feature!

SELinux [1], [2], [3] is typically used in Red Hat, CentOS and Fedora Linux. On Debian, Ubuntu and SuSE you have a similar solution called AppArmor.

To see if SELinux is enabled just run the following command:

shell> sestatus SELinux status: enabled SELinuxfs mount: /selinux Current mode: enforcing Mode from config file: enforcing Policy version: 24 Policy from config file: targeted

To disable SELinux you just have to run the following command:

shell> setenforce 0

And to make this change persistent you have to change it in the following configuration file:

# # /etc/selinux/config # # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=permissive # SELINUXTYPE= can take one of these two values: # targeted - Targeted processes are protected, # mls - Multi Level Security protection. SELINUXTYPE=targeted

But possibly you want to move the MySQL datadir to an other location without disabling SELinux? To achieve this proceed with the following steps:

The simple way

If you have just moved datadir or the MySQL port the Blog article SELinux and MySQL of Jeremy Smyth is a good starting point.

Complicated way

If you want to create an other or a new MySQL instance or do some other stuff you have to do some more things manually (possibly there is also an automated way?):

First it is recommended to install the setroubleshoot utility. Then with the command:

shell> tail /var/log/messages Jun 20 09:38:53 ip-10-39-25-184 setroubleshoot: SELinux is preventing /bin/mkdir from write access on the directory /var/lib. For complete SELinux messages. run sealert -l ef8eae63-7ec3-4b22-87e0-5774120726c3

You will find what is going wrong. Follow the instructions:

shell> sealert -l ef8eae63-7ec3-4b22-87e0-5774120726c3 SELinux is preventing /bin/mkdir from write access on the directory /var/lib. ***** Plugin catchall_labels (83.8 confidence) suggests ******************** If you want to allow mkdir to have write access on the lib directory Then you need to change the label on /var/lib Do # semanage fcontext -a -t FILE_TYPE '/var/lib' where FILE_TYPE is one of the following: var_log_t, mysqld_var_run_t, mysqld_db_t, root_t. Then execute: restorecon -v '/var/lib' ***** Plugin catchall (17.1 confidence) suggests *************************** If you believe that mkdir should be allowed write access on the lib directory by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep mkdir /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp

until MySQL starts properly. And also test a reboot of the machine!

Taxonomy upgrade extras: galeraclusterMySQL ClusterHigh Availability

Recently we had to set-up a 3-node Galera Cluster with a Load Balancer in front of it. Because Galera Cluster nodes (mysqld) still reply to TCP requests on port 3306 when they are expelled from the Cluster it is not sufficient to just leave it to the Load Balancer to check the port if a Galera node is properly running or not.

We used the wsrep_notify_cmd variable to hook our own script into the Galera Cluster which disables each Node on the Load Balancer when its state changed.

# my.cnf # [mysqld] wsrep_notify_cmd = /usr/local/bin/lvs_control.sh

The whole Galera Cluster Architecture looks as follows:

As Load Balancer we used the IPVS Load Balancer from the Linux Virtual Server (LVS) Project. This Load Balancer was made highly available with keepalived.

Our script to take a Galera Node out of the Load Balancer was the following:

#!/bin/bash -eu # # /etc/mysql/conf.d/wsrep.cnf # # [mysqld] # wsrep_notify_cmd = /usr/local/bin/lvs_control.sh # LOG="/tmp/lvs_control.log" LBIP="192.168.0.89" VIP="192.168.0.99" PORT="3306" LBUSER="galera" LBUSER="root" ETC="/etc/mysql/conf.d/wsrep.cnf" ETC="/home/mysql/data/mysql-5.5-wsrep-23.7-a/my.cnf" MYIP='' WEIGHT="100" DATE=$(date '+%Y-%m-%d %H:%M:%S') echo $DATE >>$LOG regex='^.*=\s*([0-9]+.[0-9]+.[0-9]+.[0-9]+).*' str=$(grep "^wsrep_node_incoming_address" $ETC 2>>$LOG) if [[ $str =~ $regex ]] ; then MYIP=${BASH_REMATCH[1]} else echo "Cannot find IP address in $str" >>$LOG exit 1 fi while [ $# -gt 0 ] ; do case $1 in --status) STATUS=$2 shift ;; --uuid) CLUSTER_UUID=$2 shift ;; --primary) PRIMARY=$2 shift ;; --index) INDEX=$2 shift ;; --members) MEMBERS=$2 shift ;; esac shift done # echo $* >> $LOG echo $STATUS >> $LOG # Undefined means node is shutting down # Synced means node is ready again if [ "$STATUS" != "Synced" ] ; then cmd="ssh $LBUSER@$LBIP 'sudo /sbin/ipvsadm -e -t $VIP:$PORT -r $MYIP -w 0'" else cmd="ssh $LBUSER@$LBIP 'sudo /sbin/ipvsadm -e -t $VIP:$PORT -r $MYIP -w $WEIGHT'" fi echo $cmd >>$LOG eval $cmd >>$LOG 2>&1 echo "ret=$?" >>$LOG exit 0

We assume that the same script can be used with little modifications for the Galera Load Balancer as well.

FromDual is looking for professional, enthusiastic and experienced people who:

• Know MySQL, Percona Server or MariaDB extensively
• Are Familiar with the open source eco-system
• Know how to operate database systems, as a DBA or a DevOps
• Understand what can go wrong in operating a database
• Are happy to work autonomously, remotely and to communicate with IRC, Skype, Mail and Phone
• Are comfortable on Linux systems
• Are team players, keen to contribute to the growth of the company
• Are Comfortable dealing direct with clients and
• Look for new challenges

Job description

We are looking for full-time MySQL support engineers (female or male) to primarily take care of our MySQL support services and help our customers operating their MySQL databases (remote-DBA and emergency interventions).

You are well trained and have good experience in:

• Operating critical highly available MySQL production databases mostly on Linux.
• Running MySQL-Replication in all variants is your daily business.
• The working of the most used MySQL HA set-up's and how to fix them efficiently if problems occur. (If you are already experienced in running Galera Cluster this would be a plus!
• Open Source Technologies (LAMP stack, etc.)
• Bash scripting and you can do some simple programs in at least one popular programming/scripting language (Perl, PHP, ...).

You will be in direct contact with the customers and you need good antennae to listen to them, know how to respond and get the answers to their real problems. You also have to be proactive when something goes wrong and direct the customer back to the right track.

You need to have good communication skills and be an active team player.

To fulfil your job you have to work in European Time Zones. You can organize your working time flexible within certain ranges. Participating in the on call duty is expected. FromDual is a completely virtual company and relocation is not needed (home office). Good English verbally and in writing is a must. Most of our current customers speak German and having German skills is a plus.

Beside being our support engineer we expect you to improve your knowledge and skills and to contribute to improving our monitoring solution, our database controlling solution and our other tools. Further, we expect that you write regular technical articles and give help wherever it is needed or requested...

You should be prepared to work, think and act autonomously most of the time and to teach yourself (using Google, MySQL documentation, testing etc.). If you are ever stuck, your colleagues at FromDual will assist you.

If you need somebody holding your hand all the time, FromDual is not a good choice for you.

Who is FromDual?

FromDual is the leading independent and professional MySQL database consulting and service company in Europe with its Headquarters in Switzerland.

Our customers are mostly located in Europe and range from small start-up companies to some of the top-500 companies of Europe.

You will be joining us at an exciting time. We are growing and we need like-minded people to grow with us, individually and collectively. As our horizons expand, we need our team to expand in its skills, knowledge and expertise.

Applying to join FromDual could be the best decision you make.

How to continue

If you are interested in this opportunity and if you feel you are a good "fit" (we know that there will not be a 100% match!) we would be glad to hear from you.

Please send your true CV with your salary expectation and a list of your open source involvements, blog articles, slides, tweets etc. to jobs@fromdual.com. If you want to know more about this job opportunity or if you want to speak with me, please call me at +41 79 830 09 33 (Oli Sennhauser, CTO). Only candidates, NO head hunters please!

After we received and screened your CV we will invite you to prove your technical skills by taking an exam in operating MySQL. If you pass the exam you will be invited for the final interviews.

This job opportunity is open until May 31st 2013.

Taxonomy upgrade extras: galeramyisaminnodbstorage engine

Switching from MySQL/MyISAM to Galera Cluster requires that all tables (except those from the mysql, information_schema and performance_schema) are using the InnoDB Storage Engine.

For altering the Storage Engine of the tables we wrote a script (alter_engine.pl) long time ago already. Because we have made many of those switches recently we have extended its functionality.

New features

• Recognizes VIEW's and does NOT try to alter their Storage Engine (bug).
• Script is MySQL version aware. Complain if too old MySQL version is used.
• Find tables without a Primary Key.
• Check for too long InnoDB Primary Keys
• Check for FULLTEXT indexes in MySQL 5.1 and 5.5 and write a note if version is older.

Example ./alter_engine.pl User [root] : Password [] : secret Schema from (or all) [test] : all Engine to [InnoDB] : Version is : 5.6.10 MR Version is: 050610 The following tables might not have a Primary Key: +--------------+----------------------+ | table_schema | table_name | +--------------+----------------------+ | test | innodb_table_monitor | | test | log_event | | test | parent | | test | t | +--------------+----------------------+ The tables above not having a Primary Key will negatively affect perfor- mance and data consistency in MySQL Master/Slave replication and Galera Cluster replication. The following tables might have a too long Primary Key for InnoDB (> 767 bytes): +--------------+------------+-------------+ | table_schema | table_name | column_name | +--------------+------------+-------------+ | test | test | data | +--------------+------------+-------------+ The following tables might have a FULLTEXT index (which is only supported in MySQL 5.6 and newer): +--------------+------------+-------------+ | table_schema | table_name | column_name | +--------------+------------+-------------+ | test | test | data | +--------------+------------+-------------+ Output written to /tmp/alter_table_all.sql After reviewing it you can apply it with mysql --user=root --password=secret

From time to time some maintenance work on the MySQL database has to be done. During the maintenance window we do not want to have application traffic on the database.

Sometimes it is hard to shut down all applications spread over the whole company. Or we want to allow only some specific hosts to access mysql from remote (for example the monitoring system or the backup server).

For this purpose we can use the Linux packet filtering.

To see what packet filtering rules are available we can run the following command:

iptables -L INPUT -v

To close the MySQL port on all interfaces we use:

iptables -A INPUT -p tcp --dport mysql -j DROP

and to open the MySQL port again after the maintenance window:

iptables -D INPUT -p tcp --dport mysql -j DROP

With the -i option we can restrict the rule to a specific interface for example eth0 and with the option -s we can specify a specific source only. Or with a ! -s we can implement an inverse rule (all but).

Taxonomy upgrade extras: galeracluster

A while ago it was pretty inconvenient to start a complete Galera Cluster from scratch. Rolling restart an such things are already working well but bootstrapping was a pain.

With Galera v2.2 new functionality came in. We tried it out and it did not work as documented. :-( Thanks to Teemu's help we found there was a documentation bug in the Galera documentation.

The settings which were working for us are:

wsrep_cluster_address = "gcomm://192.168.1.2,192.168.1.3?pc.wait_prim=no"

And when all 3 nodes of the Galera Cluster are started and ready to join you can run:

SET GLOBAL wsrep_provider_options="pc.bootstrap=1";

I hope we can go life on Thursday with the new Telco VoIP Cluster for 2500 employees...

Have fun and enjoy an even better Galera Cluster for MySQL!

Taxonomy upgrade extras: Backup Restore RecoveryBackup

Some MySQL customers do not want to use the root user for mysqldump backups. For this user you have to grant the following minimal MySQL privileges:

mysqldump --single-transaction (InnoDB) CREATE USER 'backup'@'localhost' IDENTIFIED BY 'secret'; GRANT SELECT, SHOW VIEW, RELOAD, REPLICATION CLIENT, EVENT, TRIGGER ON *.* TO 'backup'@'localhost';
mysqldump --lock-all-tables (MyISAM) GRANT LOCK TABLES ON *.* TO 'backup'@'localhost';

If we missed a privilege please let us know.

Taxonomy upgrade extras: innodb

In this weeks MySQL workshop we were discussing, beside other things, about the innodb_file_per_table parameter and its advantages of enabling it. In addition there was a discussion if the InnoDB system tablespace file can be shrinked once it has been grown very large or not. We all know the answer: The InnoDB system tablespace file does never shrink again.

But why should it not be possible? Other databases like for example Oracle can shrink or even get rid of tablespace files... After some philosophising about it we came to the conclusion that we should give it a try if this is possible with InnoDB as well.

The scenario we considered was the following: You inherit a MySQL database with InnoDB tables but innodb_file_per_table was set to 0. So all the tables are located in the InnoDB tablespace file. And only a small amount of space is left on the device and there is a lot of free space in the InnoDB system tablespace file. The database itself is much too big to dump and restore and we want to get rid of the one big InnoDB system tablespace file and have many small tablespace files as we get them with innodb_file_per_table = 1.

So what we did is the following: We created InnoDB tables inside the InnoDB system tablespace (ibdata1) and bloat them up. Then we altered them to be placed in their own tablespace files by OPTIMIZE TABLE. And now the tricky part starts: How can we shrink the InnoDB system tablespace file to free the disk space again?

CAUTION: This is a prove of concept and should never be used on a production system!!!

First we move all tables out of the InnoDB system tablespace (with innodb_file_per_table = 1):

mysqlcheck --optimize --all-databases --user=root ... note : Table does not support optimize, doing recreate + analyze instead status : OK ...

Now all tables have been moved out of the system tablespace, but the file is still about 674 Mbyte in size:

ll ibdata1 -rw-rw----. 1 mysql mysql 706740224 Dec 6 23:37 ibdata1

Then we search for empty blocks at the end of the InnoDB data files:

innochecksum -v -d ibdata1 file ibdata1 = 706740224 bytes (43136 pages)... checking pages in range 0 to 43135 page 0: log sequence number: first = 3558400819; second = 3558400819 page 0: old style: calculated = 148443420; recorded = 148443420 page 0: new style: calculated = 4252778336; recorded = 4252778336 ... page 42508: log sequence number: first = 0; second = 0 page 42508: old style: calculated = 1371122432; recorded = 0 page 42508: new style: calculated = 1575996416; recorded = 0 ... page 43135: log sequence number: first = 0; second = 0 page 43135: old style: calculated = 1371122432; recorded = 0 page 43135: new style: calculated = 1575996416; recorded = 0

In ideal case we should also find blocks which are not used any more but not blanked out. Theses 627 blocks (of 16k = 10 Mbyte) can easily be removed...

Next we shrink the InnoDB system tablespace file after stopping the mysqld:

printf '' | dd of=ibdata1 bs=16384 seek=42508 ll ibdata1 -rw-rw----. 1 mysql mysql 696451072 Dec 6 23:42 ibdata1

As a next step we have to change the number of blocks in the header of the InnoDB system tablespace file. This can be done with a tool like hexedit (aptitude install hexedit). We have to change at position 0x0030 the value from 43136 (0xA880) to 42508 (0xA60C):

hexdump -C -n 256 ibdata1 00000000 fd 7c 3f 60 00 00 00 00 00 00 00 00 00 00 00 00 |.|?`............| 00000010 00 00 00 00 d4 18 e3 33 00 08 00 00 00 00 d4 18 |.......3........| 00000020 e4 13 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000030 a8 80 00 00 a6 c0 00 00 00 00 00 00 01 21 00 00 |.............!..|

Otherwise we would get an error like:

InnoDB: Error: tablespace size stored in header is 43146 pages, but InnoDB: the sum of data file sizes is only 42508 pages

It looks like InnoDB itself corrects somehow the block number to a 0x100 boundary (4 Mbyte) later.

As the next step we have to fix the new style check sum (at position 0x0000) and the old style check sum (at position 0x3FFC). You have to do this until innochecksum does not complain anymore:

innochecksum -d -p 0 ibdata1 file ibdata1 = 696451072 bytes (42508 pages)... checking pages in range 0 to 0 page 0: log sequence number: first = 3558400819; second = 3558400819 page 0: old style: calculated = 2354503790; recorded = 2354503790 page 0: new style: calculated = 3427457314; recorded = 3587772574

When you have done this the database should be ready to start.

The tables later on can be possibly transferred with the transportable tablespace feature which comes with MySQL 5.6.

I have not found a good way yet to find the highest used block in the tablespace file. So it is a wild guess which is dangerous. Especially because some InnoDB UNDO LOG blocks seems to be located there at very high positions:

SELECT page_type, MAX(page_number) AS max_page_number FROM information_schema.innodb_buffer_page WHERE space = 0 AND page_number != 0 GROUP BY page_type ORDER BY max_page_number; +-------------------+-----------------+ | page_type | max_page_number | +-------------------+-----------------+ | TRX_SYSTEM | 5 | | SYSTEM | 300 | | BLOB | 9366 | | EXTENT_DESCRIPTOR | 32768 | | IBUF_BITMAP | 32769 | | INODE | 42123 | | INDEX | 45229 | | ALLOCATED | 45247 | | UNDO_LOG | 45503 | +-------------------+-----------------+

It would be good if we have a method to relocate those blocks somehow...

To verify that everything works I have tried to increase the system tablespace again. This seems to work if the number of blocks is dividable by 256 (4 Mbyte, or 128 2 Mbyte?). But growing the system tablespace again should not be the intention.

Further according to our tests this method of shrinking the InnoDB system tablespace seems to work with MySQL 5.1, 5.5 and 5.6.

Thanks to Ralf, Torsten and Stefan for assistance!

It would be nice to get some feedback from the InnoDB and Percona guys about how this feature could be implemented correctly...

And finally: Do not blame and beat me. I know that this is an evil hack, but I like to play in my sandbox as I want!

Taxonomy upgrade extras: userlogin triggerlogon triggerloggingaudithosterhosting

MySQL hosting providers can easily loose the overview over their customers and which user or schema is still in use and which not.

The MySQL database becomes bigger and bigger, uses more and more RAM and disk space and the backup takes longer and longer.

In this situation it would be nice to know which MySQL database user has logged in within the last 6 months for example. MySQL database users who did not login within a defined period can be backuped and removed from the production MySQL database.

The following MySQL login trigger helps to track the login of all non-super privileged MySQL users.

First we need a table where to log the login of the users:

-- DROP DATABASE tracking; CREATE DATABASE tracking; use tracking; -- DROP TABLE IF EXISTS login_tracking; CREATE TABLE login_tracking ( user VARCHAR(16) , host VARCHAR(60) , ts TIMESTAMP , PRIMARY KEY (user, host) ) engine = MyISAM;

Then we need a MySQL stored procedure which does the logging of the login:

-- DROP PROCEDURE IF EXISTS login_trigger; DELIMITER // CREATE PROCEDURE login_trigger() SQL SECURITY DEFINER BEGIN INSERT INTO login_tracking (user, host, ts) VALUES (SUBSTR(USER(), 1, instr(USER(), '@')-1), substr(USER(), instr(USER(), '@')+1), NOW()) ON DUPLICATE KEY UPDATE ts = NOW(); END; // DELIMITER ;

Then we have to grant the EXECUTE privilege to all users of the database which do not have the SUPER privilege. MySQL users with the SUPER privilege are not logged with the init_connect login trigger hook:

-- REVOKE EXECUTE ON PROCEDURE tracking.login_trigger FROM 'oli'@'%'; GRANT EXECUTE ON PROCEDURE tracking.login_trigger TO 'oli'@'%';

Those GRANTSs can be created with the following query:

tee /tmp/grants.sql SELECT CONCAT("GRANT EXECUTE ON PROCEDURE tracking.login_trigger TO '", user, "'@'", host, "';") AS query FROM mysql.user WHERE Super_priv = 'N'; notee +---------------------------------------------------------------------------------+ | query | +---------------------------------------------------------------------------------+ | GRANT EXECUTE ON PROCEDURE tracking.login_trigger TO 'oli'@'localhost'; | | GRANT EXECUTE ON PROCEDURE tracking.login_trigger TO 'replication'@'127.0.0.1'; | | GRANT EXECUTE ON PROCEDURE tracking.login_trigger TO 'oli'@'%'; | | GRANT EXECUTE ON PROCEDURE tracking.login_trigger TO ''@'localhost'; | +---------------------------------------------------------------------------------+

As the last step we have to activate the stored procedure by hooking it into the login trigger hook:

-- SET GLOBAL init_connect=""; SET GLOBAL init_connect="CALL tracking.login_trigger()";

If something went wrong with the login trigger you find the needed information in the MySQL error log.

Reporting

To find out, which users have logged in we can run the following query:

SELECT * FROM tracking.login_tracking; +------+-----------+---------------------+ | user | host | ts | +------+-----------+---------------------+ | oli | localhost | 2012-11-30 15:36:39 | +------+-----------+---------------------+

To find at what time a user has logged in last you can run:

SELECT u.user, u.host, l.ts FROM mysql.user AS u LEFT JOIN tracking.login_tracking AS l ON u.user = l.user AND l.host = u.host WHERE u.Super_priv = 'N'; +-------------+-----------+---------------------+ | user | host | ts | +-------------+-----------+---------------------+ | oli | localhost | 2012-12-01 09:55:33 | | replication | 127.0.0.1 | NULL | | crm | 127.0.0.1 | NULL | +-------------+-----------+---------------------+

And to find users which are logged but could not be found from the mysql user table you can run:

SELECT l.user, l.host FROM tracking.login_tracking AS l LEFT JOIN mysql.user AS u ON u.user = l.user AND l.host = u.host WHERE u.user IS NULL;