You are here
News
FromDual Performance Monitor for MySQL and MariaDB 0.10.5 has been released
FromDual has the pleasure to announce the release of the new version 0.10.5 of its popular Database Performance Monitor for MySQL, MariaDB, Galera Cluster and Percona Server fpmmm.
You can download fpmmm from here.
In the inconceivable case that you find a bug in fpmmm please report it to our Bug-tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
This release contains various bug fixes and improvements. The previous release had some major bugs so we recommend to upgrade...
New installation of fpmmm v0.10.5Please follow our mpm installation guide. A specific fpmmm installation guide will follow with the next version.
Prerequisites CentOS 6# yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm yum update yum install zabbix-sender
CentOS 7 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/7/x86_64/zabbix-release-2.2-1.el7.noarch.rpm yum update yum install zabbix-sender
Ubuntu 14.04
# apt-get install php5-cli php5-mysqlnd php5-curl # cat << _EOF >/etc/php5/cli/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF apt-get install zabbix-agent
OpenSuSE 13.1
# zypper install php5 php5-posix php5-mysql php5-pcntl php5-curl #cat << _EOF >/etc/php5/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF zypper addrepo http://download.opensuse.org/repositories/server:/monitoring/openSUSE_13.1 server_monitoring zypper update zypper install zabbix-agent
Upgrade from fpmmm 0.10.x to fpmmm 0.10.5 # cd /opt # tar xf /download/fpmmm-0.10.5.tar.gz # rm -f fpmmm # ln -s fpmmm-0.10.5 fpmmm
The following templates in your Zabbix monitor should be replaced. Before you replace the templates it is a good idea to first delete all triggers...
- tpl/Template_FromDual.MySQL.fpmmm.xml
- tpl/Template_FromDual.MySQL.innodb.xml
- tpl/Template_FromDual.MySQL.master.xml
- tpl/Template_FromDual.MySQL.myisam.xml
- tpl/Template_FromDual.MySQL.mysql.xml
- tpl/Template_FromDual.MySQL.server.xml
- tpl/Template_FromDual.MySQL.slave.xml
- Better and more verbose error handling in various modules.
- Directory for log file is created automatically if it does not exist yet.
- All broken SQL queries (from 0.10.4) fixed again.
- Add delay for not so frequent changing data.
- Several triggers which complained after restart are fixed now.
- Connections to database were now reduced to the minimum.
- Links for templates fixed.
- Innodb_flush_log_at_trx_commit, log_queries_not_using_indexes and character_set_server triggers disabled by default.
- Also sendCachedData is now checked for too big cache file. Bug from swd.
- Slave error messages are caught and sent to the monitor.
- Warning is written to the log file if slave module is configured without being a slave.
- New slave status is reported correctly now.
- Seconds_behind_master is now only sent when running.
- New trigger for binlog_format = MIXED and replication filtering added.
- Severity increased on STATEMENT based filtering added.
- Regexp bug fixed.
- Master without binary log fixed.
- Old broken triggers fixed.
- IOPS graph added.
- Device sda5 removed.
- I/O statistics calculation improved.
- I/O r/w wait experimental items implemented.
- CPU count added.
- NUMA and virtualization information added.
- innodb_flush_method item added.
- Trigger for innodb_flush_method added.
- innodb_force_recovery trigger severity increased.
- innodb_log_files_in_group item added for log traffic threshold.
- InnoDB transaction log traffic trigger and graph added.
For subscriptions of commercial use of fpmmm please get in contact with us.
Taxonomy upgrade extras: mysqlperformancemonitormonitoringfpmmmmaasperformance monitormpmreleaseFromDual Performance Monitor for MySQL and MariaDB 0.10.5 has been released
FromDual has the pleasure to announce the release of the new version 0.10.5 of its popular Database Performance Monitor for MySQL, MariaDB, Galera Cluster and Percona Server fpmmm.
You can download fpmmm from here.
In the inconceivable case that you find a bug in fpmmm please report it to our Bug-tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
This release contains various bug fixes and improvements. The previous release had some major bugs so we recommend to upgrade...
New installation of fpmmm v0.10.5Please follow our mpm installation guide. A specific fpmmm installation guide will follow with the next version.
Prerequisites CentOS 6# yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm yum update yum install zabbix-sender
CentOS 7 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/7/x86_64/zabbix-release-2.2-1.el7.noarch.rpm yum update yum install zabbix-sender
Ubuntu 14.04
# apt-get install php5-cli php5-mysqlnd php5-curl # cat << _EOF >/etc/php5/cli/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF apt-get install zabbix-agent
OpenSuSE 13.1
# zypper install php5 php5-posix php5-mysql php5-pcntl php5-curl #cat << _EOF >/etc/php5/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF zypper addrepo http://download.opensuse.org/repositories/server:/monitoring/openSUSE_13.1 server_monitoring zypper update zypper install zabbix-agent
Upgrade from fpmmm 0.10.x to fpmmm 0.10.5 # cd /opt # tar xf /download/fpmmm-0.10.5.tar.gz # rm -f fpmmm # ln -s fpmmm-0.10.5 fpmmm
The following templates in your Zabbix monitor should be replaced. Before you replace the templates it is a good idea to first delete all triggers...
- tpl/Template_FromDual.MySQL.fpmmm.xml
- tpl/Template_FromDual.MySQL.innodb.xml
- tpl/Template_FromDual.MySQL.master.xml
- tpl/Template_FromDual.MySQL.myisam.xml
- tpl/Template_FromDual.MySQL.mysql.xml
- tpl/Template_FromDual.MySQL.server.xml
- tpl/Template_FromDual.MySQL.slave.xml
- Better and more verbose error handling in various modules.
- Directory for log file is created automatically if it does not exist yet.
- All broken SQL queries (from 0.10.4) fixed again.
- Add delay for not so frequent changing data.
- Several triggers which complained after restart are fixed now.
- Connections to database were now reduced to the minimum.
- Links for templates fixed.
- Innodb_flush_log_at_trx_commit, log_queries_not_using_indexes and character_set_server triggers disabled by default.
- Also sendCachedData is now checked for too big cache file. Bug from swd.
- Slave error messages are caught and sent to the monitor.
- Warning is written to the log file if slave module is configured without being a slave.
- New slave status is reported correctly now.
- Seconds_behind_master is now only sent when running.
- New trigger for binlog_format = MIXED and replication filtering added.
- Severity increased on STATEMENT based filtering added.
- Regexp bug fixed.
- Master without binary log fixed.
- Old broken triggers fixed.
- IOPS graph added.
- Device sda5 removed.
- I/O statistics calculation improved.
- I/O r/w wait experimental items implemented.
- CPU count added.
- NUMA and virtualization information added.
- innodb_flush_method item added.
- Trigger for innodb_flush_method added.
- innodb_force_recovery trigger severity increased.
- innodb_log_files_in_group item added for log traffic threshold.
- InnoDB transaction log traffic trigger and graph added.
For subscriptions of commercial use of fpmmm please get in contact with us.
Max_used_connections per user/account
How many connections can be opened concurrently against my MySQL or MariaDB database can be configured and checked with the following command:
SHOW GLOBAL VARIABLES LIKE 'max_connections'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | max_connections | 505 | +-----------------+-------+If this limit was ever reached in the past can be checked with:
SHOW GLOBAL STATUS LIKE 'max_use%'; +----------------------+-------+ | Variable_name | Value | +----------------------+-------+ | Max_used_connections | 23 | +----------------------+-------+But on MySQL instances with many different applications (= databases/schemas) and thus many different users it is a bit more complicated to find out which of these users have connected how many times concurrently. We can configure how many connections one specific user can have at maximum at the same time with:
SHOW GLOBAL VARIABLES LIKE 'max_user_connections'; +----------------------+-------+ | Variable_name | Value | +----------------------+-------+ | max_user_connections | 500 | +----------------------+-------+Further we can limit one specific user with:
GRANT USAGE ON *.* TO 'repl'@'%' WITH MAX_CONNECTIONS_PER_HOUR 100 MAX_USER_CONNECTIONS 10;and check with:
SELECT User, Host, max_connections, max_user_connections FROM mysql.user; +------+---------------+-----------------+----------------------+ | User | Host | max_connections | max_user_connections | +------+---------------+-----------------+----------------------+ | root | localhost | 0 | 0 | | repl | % | 100 | 10 | | repl | 192.168.1.139 | 0 | 0 | +------+---------------+-----------------+----------------------+But we have currently no chance to check if this limit was reached or nearly reached in the past...
A feature request for this was opened at MySQL wit bug #77888
SolutionIf you cannot wait for the implementation here we have a little workaround:
DROP TABLE IF EXISTS mysql.`max_used_connections`; CREATE TABLE mysql.`max_used_connections` ( `USER` char(16) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL, `HOST` char(60) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL, `MAX_USED_CONNECTIONS` bigint(20) NOT NULL, PRIMARY KEY (`USER`, `HOST`) USING HASH ) ENGINE=MEMORY DEFAULT CHARSET=utf8 ; DROP EVENT IF EXISTS mysql.gather_max_used_connections; -- event_scheduler = on CREATE DEFINER=root@localhost EVENT mysql.gather_max_used_connections ON SCHEDULE EVERY 10 SECOND DO INSERT INTO mysql.max_used_connections SELECT user, host, current_connections FROM performance_schema.accounts WHERE user IS NOT NULL AND host IS NOT NULL ON DUPLICATE KEY UPDATE max_used_connections = IF(current_connections > max_used_connections, current_connections, max_used_connections) ; SELECT * FROM mysql.max_used_connections; +--------+-----------+----------------------+ | USER | HOST | MAX_USED_CONNECTIONS | +--------+-----------+----------------------+ | root | localhost | 4 | | zabbix | localhost | 21 | +--------+-----------+----------------------+Caution: Because we used a MEMORY table those values are reset at every MySQL restart (as it happens with the PERFORMANCE_SCHEMA or the INFORMATION_SCHEMA).
MySQL Environment MyEnv 1.2.1 has been released
FromDual has the pleasure to announce the release of the new version 1.2.1 of its popular MySQL, Galera Cluster, MariaDB and Percona Server multi-instance environment MyEnv.
The new MyEnv can be downloaded here.
In the inconceivable case that you find a bug in the MyEnv please report it to our bug tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
Upgrade from 1.1.x to 1.2.1 # cd ${HOME}/product # tar xf /download/myenv-1.2.1.tar.gz # rm -f myenv # ln -s myenv-1.2.1 myenvIf you are using plug-ins for showMyEnvStatus create all the links in the new directory structure:
cd ${HOME}/product/myenv ln -s ../../utl/oem_agent.php plg/showMyEnvStatus/Changes in MyEnv 1.2.1 MyEnv
- Bug in myenv.server fixed (chmod user:).
- Distribution check for CentOS 7.1 fixed (bug report Jörg).
- Directory /var/run/mysqld is created by myenv.server
- Missing function output added.
- Output changes unified, execute with LC_ALL=C made language independent.
- Template my.cnf extended by hostname and instance name.
- Skip in init script implemented, (bug report Jörg).
- MyEnv my.cnf.template merged with website.
- Template AUTO_INCREMENT values fixed.
- Galera Cluster variable causal read in my.cnf template updated.
- Functions separated into own library.
- All templates moved from etc to tpl.
- my.cnf template provided with hostname tag.
- Skript alter_engine.php now supports socket.
- Check in alter_engine.php for Primary Key length of 767 bytes was removed because it was wrong.
- Skript decrypt_mylogin_cnf.php added.
- Script log_maintenance.php added.
- Usage and --help added to block_galera_node.sh.
For subscriptions of commercial use of MyEnv please get in contact with us.
Taxonomy upgrade extras: MyEnvoperationMySQL Operationsmulti instanceconsolidationtestingreleaseMySQL Environment MyEnv 1.2.1 has been released
FromDual has the pleasure to announce the release of the new version 1.2.1 of its popular MySQL, Galera Cluster, MariaDB and Percona Server multi-instance environment MyEnv.
The new MyEnv can be downloaded here.
In the inconceivable case that you find a bug in the MyEnv please report it to our bug tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
Upgrade from 1.1.x to 1.2.1 # cd ${HOME}/product # tar xf /download/myenv-1.2.1.tar.gz # rm -f myenv # ln -s myenv-1.2.1 myenvIf you are using plug-ins for showMyEnvStatus create all the links in the new directory structure:
cd ${HOME}/product/myenv ln -s ../../utl/oem_agent.php plg/showMyEnvStatus/Changes in MyEnv 1.2.1 MyEnv
- Bug in myenv.server fixed (chmod user:).
- Distribution check for CentOS 7.1 fixed (bug report Jörg).
- Directory /var/run/mysqld is created by myenv.server
- Missing function output added.
- Output changes unified, execute with LC_ALL=C made language independent.
- Template my.cnf extended by hostname and instance name.
- Skip in init script implemented, (bug report Jörg).
- MyEnv my.cnf.template merged with website.
- Template AUTO_INCREMENT values fixed.
- Galera Cluster variable causal read in my.cnf template updated.
- Functions separated into own library.
- All templates moved from etc to tpl.
- my.cnf template provided with hostname tag.
- Skript alter_engine.php now supports socket.
- Check in alter_engine.php for Primary Key length of 767 bytes was removed because it was wrong.
- Skript decrypt_mylogin_cnf.php added.
- Script log_maintenance.php added.
- Usage and --help added to block_galera_node.sh.
For subscriptions of commercial use of MyEnv please get in contact with us.
Taxonomy upgrade extras: myenvoperationMySQL Operationsmulti instanceconsolidationtestingupgradereleaseMySQL Environment MyEnv 1.2.1 has been released
FromDual has the pleasure to announce the release of the new version 1.2.1 of its popular MySQL, Galera Cluster, MariaDB and Percona Server multi-instance environment MyEnv.
The new MyEnv can be downloaded here.
In the inconceivable case that you find a bug in the MyEnv please report it to our bug tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
Upgrade from 1.1.x to 1.2.1 # cd ${HOME}/product # tar xf /download/myenv-1.2.1.tar.gz # rm -f myenv # ln -s myenv-1.2.1 myenvIf you are using plug-ins for showMyEnvStatus create all the links in the new directory structure:
cd ${HOME}/product/myenv ln -s ../../utl/oem_agent.php plg/showMyEnvStatus/Changes in MyEnv 1.2.1 MyEnv
- Bug in myenv.server fixed (chmod user:).
- Distribution check for CentOS 7.1 fixed (bug report Jörg).
- Directory /var/run/mysqld is created by myenv.server
- Missing function output added.
- Output changes unified, execute with LC_ALL=C made language independent.
- Template my.cnf extended by hostname and instance name.
- Skip in init script implemented, (bug report Jörg).
- MyEnv my.cnf.template merged with website.
- Template AUTO_INCREMENT values fixed.
- Galera Cluster variable causal read in my.cnf template updated.
- Functions separated into own library.
- All templates moved from etc to tpl.
- my.cnf template provided with hostname tag.
- Skript alter_engine.php now supports socket.
- Check in alter_engine.php for Primary Key length of 767 bytes was removed because it was wrong.
- Skript decrypt_mylogin_cnf.php added.
- Script log_maintenance.php added.
- Usage and --help added to block_galera_node.sh.
For subscriptions of commercial use of MyEnv please get in contact with us.
The Upcoming Leap Second
The press, be it the general daily newspaper or the computer magazines, is currently informing the public about an upcoming leap second, which will be taken in the night from June 30 to July 1 at 00:00:00 UTC. While we Europeans will enjoy our well-deserved sleep then, this will be at 5 PM (17:00) local time on June 30 for Califormia people, and during the morning of July 1 for people in China, Japan, Korea, or Australia. (Other countries not mentioned for the sake of brevity.) This is different from last time, when the leap second was taken in the night from Saturday to Sunday (2012-July-1 00:00:00 UTC), so it was a weekend everywhere on the globe.
We have got several requests from our customers about this upcoming leap second, whether they need to take any special precautions or whether they "are safe". Well, obviously nobody is "safe" from the leap second in the sense that it would circumvent them, everybody will encounter it on their systems. The concern is whether they have to expect any trouble.
For the people operating MySQL (or any other DBMS), this issue is threefold:
Let us look at the operating system first, dealing with Linux only.
(All other operating systems don't show up with significant numbers in our customer base.)
Linux measures the time in seconds (since Jan 1, 1970, 00:00:00 UTC), and it does not include the leap seconds in this counting.
When the leap second is taken, the timestamp value (those seconds) will simply not be advanced at the end of the regular second, but it will re-use its current value in the leap second.
As a result, the conversion of timestamps into common time reckoning will still produce values from 0 to 59 for the minute as well as for the second, there will not be a 60.
Another consequence is that there is no way to tell the leap second from the preceding regular one.
MySQL always takes the time information from the Linux kernel, so it will also use the same timestamp value (or "now()" result) for both the regular and the leap second. The MySQL manual describes this on its own leap second page, which has become inaccurate by some recent code changes: We could not reproduce the results given there (and will probably file a documentation bug about this). However, that difference does not affect the principle of the page's first paragraph.
About the application, it is hard to claim anything - there are too many of them. However, it is obvious that an application might run into trouble if it managed to store a new timestamp value every second and assumed they are distinct: they will not be (unless the application manages to skip the leap second). Let's hope any application programmer creating such a high-resolution application was aware of the problem.
So does it all look fine?
Not completely:
Following the last leap second (just three years ago), several administrators noticed that their machines became extraordinarily busy, which even let the power consumption rise significantly.
This was caused by a bug in the Linux kernel, it let user processes resume immediately if they were trying to wait on a high-resolution timer.
The exact details are beyond the scope of this article, your favourite search engine will provide you with more than enough references if you ask it.
For our purposes, the important fact is that this also happened to MySQL server processes ("mysqld") because InnoDB was doing such operations.
Back then, Sheeri K. Cabral (well-known in the MySQL blogosphere) published a cure which her team had discovered:
This loop can be broken by simply setting the Linux kernel's clock to the current time.
date -s "`date`"
While this looks like a no-op at first sight, I assume that it will reset the sub-second time information and so will have a small effect, which obviously is sufficient to terminate that loop.
(In practice, you should stop your NTP daemon before changing the date, and restart it afterwards. The exact command will depend on your Linux distribution.)
Well, this was three years ago. The issue was reported to kernel developers, a fix was developed (which gives credit to Sheeri's report) and applied to newer kernels, and it w as also backported to older kernels. From my search, it seems that all reasonably maintained installations should have received that fix. For example, I have seen notices that Ubuntu 12.04 (since kernel 3.2.0-29.46) and 14.04 do have it, as does RedHat 6.4 (since kernel 2.6.32-298); for other distributions, I did not search.
In addition to a current kernel, you obviously need current packages including the leap second information. Typically, this would be "ntp", "ntpdate", "tzdata", and related ones. Don't forget to restart your NTP daemon after installing these updates!
So I wish all our readers that the leap second may not get them into trouble, or that (if worst comes to worst) the cure published by Sheeri may get them out.
However, I have to remind you of the old truth which is attributed to Mark Twain (sometimes also to Niels Bohr):
"It is difficult to make predictions, especially about the future."
The Upcoming Leap Second
The press, be it the general daily newspaper or the computer magazines, is currently informing the public about an upcoming leap second, which will be taken in the night from June 30 to July 1 at 00:00:00 UTC. While we Europeans will enjoy our well-deserved sleep then, this will be at 5 PM (17:00) local time on June 30 for Califormia people, and during the morning of July 1 for people in China, Japan, Korea, or Australia. (Other countries not mentioned for the sake of brevity.) This is different from last time, when the leap second was taken in the night from Saturday to Sunday (2012-July-1 00:00:00 UTC), so it was a weekend everywhere on the globe.
We have got several requests from our customers about this upcoming leap second, whether they need to take any special precautions or whether they "are safe". Well, obviously nobody is "safe" from the leap second in the sense that it would circumvent them, everybody will encounter it on their systems. The concern is whether they have to expect any trouble.
For the people operating MySQL (or any other DBMS), this issue is threefold:
Let us look at the operating system first, dealing with Linux only.
(All other operating systems don't show up with significant numbers in our customer base.)
Linux measures the time in seconds (since Jan 1, 1970, 00:00:00 UTC), and it does not include the leap seconds in this counting.
When the leap second is taken, the timestamp value (those seconds) will simply not be advanced at the end of the regular second, but it will re-use its current value in the leap second.
As a result, the conversion of timestamps into common time reckoning will still produce values from 0 to 59 for the minute as well as for the second, there will not be a 60.
Another consequence is that there is no way to tell the leap second from the preceding regular one.
MySQL always takes the time information from the Linux kernel, so it will also use the same timestamp value (or "now()" result) for both the regular and the leap second. The MySQL manual describes this on its own leap second page, which has become inaccurate by some recent code changes: We could not reproduce the results given there (and will probably file a documentation bug about this). However, that difference does not affect the principle of the page's first paragraph.
About the application, it is hard to claim anything - there are too many of them. However, it is obvious that an application might run into trouble if it managed to store a new timestamp value every second and assumed they are distinct: they will not be (unless the application manages to skip the leap second). Let's hope any application programmer creating such a high-resolution application was aware of the problem.
So does it all look fine?
Not completely:
Following the last leap second (just three years ago), several administrators noticed that their machines became extraordinarily busy, which even let the power consumption rise significantly.
This was caused by a bug in the Linux kernel, it let user processes resume immediately if they were trying to wait on a high-resolution timer.
The exact details are beyond the scope of this article, your favourite search engine will provide you with more than enough references if you ask it.
For our purposes, the important fact is that this also happened to MySQL server processes ("mysqld") because InnoDB was doing such operations.
Back then, Sheeri K. Cabral (well-known in the MySQL blogosphere) published a cure which her team had discovered:
This loop can be broken by simply setting the Linux kernel's clock to the current time.
date -s "`date`"
While this looks like a no-op at first sight, I assume that it will reset the sub-second time information and so will have a small effect, which obviously is sufficient to terminate that loop.
(In practice, you should stop your NTP daemon before changing the date, and restart it afterwards. The exact command will depend on your Linux distribution.)
Well, this was three years ago. The issue was reported to kernel developers, a fix was developed (which gives credit to Sheeri's report) and applied to newer kernels, and it w as also backported to older kernels. From my search, it seems that all reasonably maintained installations should have received that fix. For example, I have seen notices that Ubuntu 12.04 (since kernel 3.2.0-29.46) and 14.04 do have it, as does RedHat 6.4 (since kernel 2.6.32-298); for other distributions, I did not search.
In addition to a current kernel, you obviously need current packages including the leap second information. Typically, this would be "ntp", "ntpdate", "tzdata", and related ones. Don't forget to restart your NTP daemon after installing these updates!
So I wish all our readers that the leap second may not get them into trouble, or that (if worst comes to worst) the cure published by Sheeri may get them out.
However, I have to remind you of the old truth which is attributed to Mark Twain (sometimes also to Niels Bohr):
"It is difficult to make predictions, especially about the future."
The Upcoming Leap Second
The press, be it the general daily newspaper or the computer magazines, is currently informing the public about an upcoming leap second, which will be taken in the night from June 30 to July 1 at 00:00:00 UTC. While we Europeans will enjoy our well-deserved sleep then, this will be at 5 PM (17:00) local time on June 30 for Califormia people, and during the morning of July 1 for people in China, Japan, Korea, or Australia. (Other countries not mentioned for the sake of brevity.) This is different from last time, when the leap second was taken in the night from Saturday to Sunday (2012-July-1 00:00:00 UTC), so it was a weekend everywhere on the globe.
We have got several requests from our customers about this upcoming leap second, whether they need to take any special precautions or whether they "are safe". Well, obviously nobody is "safe" from the leap second in the sense that it would circumvent them, everybody will encounter it on their systems. The concern is whether they have to expect any trouble.
For the people operating MySQL (or any other DBMS), this issue is threefold:
Let us look at the operating system first, dealing with Linux only.
(All other operating systems don't show up with significant numbers in our customer base.)
Linux measures the time in seconds (since Jan 1, 1970, 00:00:00 UTC), and it does not include the leap seconds in this counting.
When the leap second is taken, the timestamp value (those seconds) will simply not be advanced at the end of the regular second, but it will re-use its current value in the leap second.
As a result, the conversion of timestamps into common time reckoning will still produce values from 0 to 59 for the minute as well as for the second, there will not be a 60.
Another consequence is that there is no way to tell the leap second from the preceding regular one.
MySQL always takes the time information from the Linux kernel, so it will also use the same timestamp value (or "now()" result) for both the regular and the leap second. The MySQL manual describes this on its own leap second page, which has become inaccurate by some recent code changes: We could not reproduce the results given there (and will probably file a documentation bug about this). However, that difference does not affect the principle of the page's first paragraph.
About the application, it is hard to claim anything - there are too many of them. However, it is obvious that an application might run into trouble if it managed to store a new timestamp value every second and assumed they are distinct: they will not be (unless the application manages to skip the leap second). Let's hope any application programmer creating such a high-resolution application was aware of the problem.
So does it all look fine?
Not completely:
Following the last leap second (just three years ago), several administrators noticed that their machines became extraordinarily busy, which even let the power consumption rise significantly.
This was caused by a bug in the Linux kernel, it let user processes resume immediately if they were trying to wait on a high-resolution timer.
The exact details are beyond the scope of this article, your favourite search engine will provide you with more than enough references if you ask it.
For our purposes, the important fact is that this also happened to MySQL server processes ("mysqld") because InnoDB was doing such operations.
Back then, Sheeri K. Cabral (well-known in the MySQL blogosphere) published a cure which her team had discovered:
This loop can be broken by simply setting the Linux kernel's clock to the current time.
date -s "`date`"
While this looks like a no-op at first sight, I assume that it will reset the sub-second time information and so will have a small effect, which obviously is sufficient to terminate that loop.
(In practice, you should stop your NTP daemon before changing the date, and restart it afterwards. The exact command will depend on your Linux distribution.)
Well, this was three years ago. The issue was reported to kernel developers, a fix was developed (which gives credit to Sheeri's report) and applied to newer kernels, and it w as also backported to older kernels. From my search, it seems that all reasonably maintained installations should have received that fix. For example, I have seen notices that Ubuntu 12.04 (since kernel 3.2.0-29.46) and 14.04 do have it, as does RedHat 6.4 (since kernel 2.6.32-298); for other distributions, I did not search.
In addition to a current kernel, you obviously need current packages including the leap second information. Typically, this would be "ntp", "ntpdate", "tzdata", and related ones. Don't forget to restart your NTP daemon after installing these updates!
So I wish all our readers that the leap second may not get them into trouble, or that (if worst comes to worst) the cure published by Sheeri may get them out.
However, I have to remind you of the old truth which is attributed to Mark Twain (sometimes also to Niels Bohr):
"It is difficult to make predictions, especially about the future."
FromDual Backup Manager for MySQL 1.2.2 has been released
FromDual has the pleasure to announce the release of the new version 1.2.2 of the popular Backup Manager for MySQL and MariaDB (fromdual_bman).
You can download the FromDual Backup Manager from here.
In the inconceivable case that you find a bug in the Backup Manager please report it to our Bugtracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
Upgrade from 1.2.x to 1.2.2 # cd ${HOME}/product # tar xf /download/fromdual_brman-1.2.2.tar.gz # rm -f fromdual_brman # ln -s fromdual_brman-1.2.2 fromdual_brmanChanges in FromDual Backup Manager 1.2.2 FromDual Backup Manager
It contains mainly fixes with brman catalog and physical backups.
You can verify your current FromDual Backup Manager version with the following command:
fromdual_bman --version- Archiving with physical backup bug fixed.
- Connect replaced by OO style and error exit fixed.
- Create catalog fixed.
- Archivedir without archive option does not make sense.
Wir suchen Dich: MySQL/MariaDB DBA für FromDual Support
FromDual ist das führende unabhängige Beratungs- und Dienstleistungsunternehmen für MySQL, Galera Cluster, MariaDB und Percona Server in Europa mit Hauptsitz in der Schweiz.
Unsere Kunden stammen hauptsächlich aus Europa und reichen vom kleinen Start-Up bis zur europäischen Top-500 Firma. Sie erhalten von uns Support bei Datenbank-Problemen, direkte Eingriffe als remote-DBA, Schulung für ihre DBAs und Entwickler sowie Beratung bei Architektur- und Design-Entscheidungen. Ausserdem entwickeln wir Tools rund um MySQL, schreiben Blog-Artikel und halten Vorträge bei Konferenzen.
Da unsere qualitativ guten Dienstleistungen immer mehr Kunden anziehen, brauchen wir Kollegen (m/w), welche selbst und mit uns wachsen wollen.
StellenbeschreibungWir suchen deutschsprachige Mitarbeiter (Sie oder Ihn) auf Junior- oder Senior-Level für Dienstleistungen rund um MySQL (hauptsächlich Support und remote-DBA Arbeiten) in Vollzeit. Primär solltest Du sicherstellen, dass die geschäftskritischen MySQL-Datenbanken unserer Kunden wie am Schnürchen laufen - und falls nicht, diese schnell wieder ans Laufen kriegen...
Unser/e "Wunschkandidat/in"
- hat Erfahrung im Betrieb kritischer und hoch verfügbarer produktiver Datenbanken hauptsächlich auf Linux,
- kennt Replikation in allen Variationen aus der täglichen Arbeit,
- weiß, wie die meist verbreiteten MySQL-HA-Setups funktionieren und wie man sie wieder effizient repariert, wenn ein Problem auftritt,
- ist sattelfest in SQL,
- bringt Erfahrung mit Galera Cluster mit,
- kann Bash skripten und einfache Programme in mindestens einer verbreiteten Programmier-/Skripting-Sprache (PHP, Bash, ...) erstellen.
Wir suchen Verstärkung, die von soliden Grundlagen aus auf dem Weg zu diesem Ideal ist.
Was wir von Dir erwarten:
- Kenntnisse in MySQL, Percona Server oder MariaDB oder Bereitschaft, sich diese anzueignen
- wissen, wie man kritische Datenbank-Systeme betreibt
- Verständnis, was beim Betrieb von Datenbanken falsch laufen kann
- selbständige Arbeitsweise (remote) mit Kommunikation über IRC, Skype, Mail und Telefon
- Kenntnisse des Linux Systems
DBA- oder DevOps-Erfahrungen wären z.B. eine gute fachliche Basis.
Du schätzt den direkten Kontakt mit Kunden, hast ein gutes Gespür für deren Probleme, kannst zuhören und findest schnell die eigentlichen Probleme. Du bist gewohnt, proaktiv zu handeln bevor etwas passiert, und führst den Kunden wieder auf den richtigen Pfad zurück.
Um Deine Arbeit erledigen zu können, arbeitest Du in einer europäischen Zeitzone. Deine Arbeitszeit kannst Du, der betrieblichen Situation entsprechend, flexibel gestalten. Wir erwarten, dass Du Deinen Beitrag zum Bereitschaftsdienst leistest. FromDual hat voraussichtlich keine Büroräumlichkeiten in Deinem Wohnort. Ein Umzug ist jedoch nicht notwendig: Wir ermöglichen Dir das Arbeiten von zu Hause aus oder unterstützen Dich bei der Suche einer geeigneten Arbeitsräumlichkeit in Deiner Nähe. Gute schriftliche und mündliche Englischkenntnisse sind erforderlich.
Was wir Dir bieten:- Deinen Leistungen angemessenes Gehalt.
- Möglichkeit Dich zum Top MySQL-Datenbankspezialisten zu entwickeln.
- Selbständiges Arbeiten.
- Verantwortung für Deine Projekte und Kunden zu übernehmen.
- Gute Kameradschaft im Team, sowie lockerer und angenehmer Umgang.
- Stellenbezogene Weiterbildungsmöglichkeiten.
- Teilnahme an Open Source Anlässen.
- Arbeit von Deinem bevorzugten Wohnort aus.
Du solltest in der Lage sein, die meiste Zeit selbständig zu arbeiten, zu denken und zu handeln und Dir neues Wissen selbständig anzueignen (durch Web-Suche, die MySQL-Dokumentation, Ausprobieren, etc.). Solltest Du dennoch einmal nicht weiterkommen, werden Dir Deine Kollegen von FromDual gerne helfen.
Wenn Du jemanden brauchst, der Dir die ganze Zeit Dein Händchen hält, ist FromDual nicht die richtige Wahl.
Wie geht es weiter
Wenn Du an dieser Chance interessiert bist und Du denkst, dass Du die passende Kandidatin oder der passende Kandidat bist, würden wir uns freuen, von Dir zu hören. Wir wissen, dass niemand 100% auf diese Stellenbeschreibung passt!
Bitte schicke Deinen ungeschönten Lebenslauf mit Deinen Gehaltsvorstellungen an jobs@fromdual.com. Wenn Du mehr über diese Stelle erfahren oder wenn Du mit mir persönlich sprechen möchtest, ruf mich bitte an unter +41 79 830 09 33 (Oli Sennhauser, CTO). Bitte nur Bewerber, KEINE Headhunter!
Nachdem Du uns Deinen Lebenslauf zugeschickt hast, darfst Du Deine Fähigkeiten in einem kleinen MySQL-Test unter Beweis zu stellen. Nach bestandenem Test laden wir Dich für die finalen Interviews ein.
FromDual Performance Monitor for MySQL and MariaDB 0.10.4 has been released
FromDual has the pleasure to announce the release of the new version 0.10.4 of its popular Database Performance Monitor for MySQL, MariaDB, Galera Cluster and Percona Server fpmmm.
You can download fpmmm from here.
In the inconceivable case that you find a bug in fpmmm please report it to our Bug-tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
This release contains various minor bug fixes and improvements.
New installation of fpmmm v0.10.4Please follow our mpm installation guide. A specific fpmmm installation guide will follow with the next version.
Prerequisites CentOS 6 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm yum update yum install zabbix-senderCentOS 7 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/7/x86_64/zabbix-release-2.2-1.el7.noarch.rpm yum update yum install zabbix-sender
Ubuntu 14.04 # apt-get install php5-cli php5-mysqlnd php5-curl # cat << _EOF >/etc/php5/cli/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF apt-get install zabbix-agent
OpenSuSE 13.1 # zypper install php5 php5-posix php5-mysql php5-pcntl php5-curl #cat << _EOF >/etc/php5/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF zypper addrepo http://download.opensuse.org/repositories/server:/monitoring/openSUSE_13.1 server_monitoring zypper update zypper install zabbix-agent
Upgrade from fpmmm 0.10.x to fpmmm 0.10.4 # cd /opt # tar xf /download/fpmmm-0.10.4.tar.gz # rm -f fpmmm # ln -s fpmmm-0.10.4 fpmmm
Changes in fpmmm v0.10.4 Security module
- Privilege problem in security module caught and error message written.
- preg_replace error fixed (FromDualMySQLagent.inc).
- Bug in master module fixed.
- Deadlock and foreign key error files move from /tmp to cacheBase.
- Deadlock and foreign key error catching fixed (bug #162).
- fpmmm version check added in template.
- fpmmm templates upgrade to Zabbix version 2.0.9. This means, they do not work with Zabix 1.8 any more!
- zabbix_sender return code 0 caught correctly now.
- Minor bug fixes and typos fixed.
- Bug in caching data fixed.
For subscriptions of commercial use of fpmmm please get in contact with us.
Taxonomy upgrade extras: mysqlperformancemonitormonitoringfpmmmmaasperformance monitorreleaseFromDual Performance Monitor for MySQL and MariaDB 0.10.4 has been released
FromDual has the pleasure to announce the release of the new version 0.10.4 of its popular Database Performance Monitor for MySQL, MariaDB, Galera Cluster and Percona Server fpmmm.
You can download fpmmm from here.
In the inconceivable case that you find a bug in fpmmm please report it to our Bug-tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
This release contains various minor bug fixes and improvements.
New installation of fpmmm v0.10.4Please follow our mpm installation guide. A specific fpmmm installation guide will follow with the next version.
Prerequisites CentOS 6 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm yum update yum install zabbix-senderCentOS 7 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/7/x86_64/zabbix-release-2.2-1.el7.noarch.rpm yum update yum install zabbix-sender
Ubuntu 14.04 # apt-get install php5-cli php5-mysqlnd php5-curl # cat << _EOF >/etc/php5/cli/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF apt-get install zabbix-agent
OpenSuSE 13.1 # zypper install php5 php5-posix php5-mysql php5-pcntl php5-curl #cat << _EOF >/etc/php5/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF zypper addrepo http://download.opensuse.org/repositories/server:/monitoring/openSUSE_13.1 server_monitoring zypper update zypper install zabbix-agent
Upgrade from fpmmm 0.10.x to fpmmm 0.10.4 # cd /opt # tar xf /download/fpmmm-0.10.4.tar.gz # rm -f fpmmm # ln -s fpmmm-0.10.4 fpmmm
Changes in fpmmm v0.10.4 Security module
- Privilege problem in security module caught and error message written.
- preg_replace error fixed (FromDualMySQLagent.inc).
- Bug in master module fixed.
- Deadlock and foreign key error files move from /tmp to cacheBase.
- Deadlock and foreign key error catching fixed (bug #162).
- fpmmm version check added in template.
- fpmmm templates upgrade to Zabbix version 2.0.9. This means, they do not work with Zabix 1.8 any more!
- zabbix_sender return code 0 caught correctly now.
- Minor bug fixes and typos fixed.
- Bug in caching data fixed.
For subscriptions of commercial use of fpmmm please get in contact with us.
Taxonomy upgrade extras: mysqlperformancemonitormonitoringfpmmmmaasperformance monitorreleaseFromDual Performance Monitor for MySQL and MariaDB 0.10.4 has been released
FromDual has the pleasure to announce the release of the new version 0.10.4 of its popular Database Performance Monitor for MySQL, MariaDB, Galera Cluster and Percona Server fpmmm.
You can download fpmmm from here.
In the inconceivable case that you find a bug in fpmmm please report it to our Bug-tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
This release contains various minor bug fixes and improvements.
New installation of fpmmm v0.10.4Please follow our mpm installation guide. A specific fpmmm installation guide will follow with the next version.
Prerequisites CentOS 6 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm yum update yum install zabbix-senderCentOS 7 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/7/x86_64/zabbix-release-2.2-1.el7.noarch.rpm yum update yum install zabbix-sender
Ubuntu 14.04 # apt-get install php5-cli php5-mysqlnd # cat << _EOF >/etc/php5/cli/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF apt-get install zabbix-agent
OpenSuSE 13.1 # zypper install php5 php5-posix php5-mysql php5-pcntl php5-curl #cat << _EOF >/etc/php5/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF zypper addrepo http://download.opensuse.org/repositories/server:/monitoring/openSUSE_13.1 server_monitoring zypper update zypper install zabbix-agent
Upgrade from fpmmm 0.10.x to fpmmm 0.10.4 # cd /opt # tar xf /download/fpmmm-0.10.4.tar.gz # rm -f fpmmm # ln -s fpmmm-0.10.4 fpmmm
Changes in fpmmm v0.10.4 Security module
- Privilege problem in security module caught and error message written.
- preg_replace error fixed (FromDualMySQLagent.inc).
- Bug in master module fixed.
- Deadlock and foreign key error files move from /tmp to cacheBase.
- Deadlock and foreign key error catching fixed (bug #162).
- fpmmm version check added in template.
- fpmmm templates upgrade to Zabbix version 2.0.9. This means, they do not work with Zabix 1.8 any more!
- zabbix_sender return code 0 caught correctly now.
- Minor bug fixes and typos fixed.
- Bug in caching data fixed.
For subscriptions of commercial use of fpmmm please get in contact with us.
Controlling worldwide manufacturing plants with MySQL
A MySQL customer of FromDual has different manufacturing plants spread across the globe. They are operated by local companies. FromDuals customer wants to maintain the manufacturing receipts centralized in a MySQL database in the Head Quarter in Europe. Each manufacturing plant should only see their specific data.
Manufacturing log information should be reported backup to European Head Quarter MySQL database.
The process was designed as follows:
Preparation of Proof of Concept (PoC)To simulate all cases we need different schemas. Some which should be replicated, some which should NOT be replicated:
CREATE DATABASE finance; CREATE TABLE finance.accounting ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `data` varchar(255) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `data_rename` (`data`) ); CREATE DATABASE crm; CREATE TABLE crm.customer ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `data` varchar(255) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `data_rename` (`data`) ); CREATE DATABASE erp; -- Avoid specifying Storage Engine here!!! CREATE TABLE erp.manufacturing_data ( id INT UNSIGNED NOT NULL AUTO_INCREMENT , manufacture_plant VARCHAR(32) , manufacture_info VARCHAR(255) , PRIMARY KEY (id) , KEY (manufacture_plant) ); CREATE TABLE erp.manufacturing_log ( id INT UNSIGNED NOT NULL AUTO_INCREMENT , manufacture_plant VARCHAR(32) , log_data VARCHAR(255) , PRIMARY KEY (id) , KEY (manufacture_plant) );MySQL replication architecture
Before you start with such complicated MySQL set-ups it is recommended to make a little sketch of what you want to build:
Preparing the Production Master database (Prod M1)To make use of all the new and cool features of MySQL we used the new GTID replication. First we set up a Master (Prod M1) and its fail-over System (Prod M2) in the customers Head Quarter:
# /etc/my.cnf [mysqld] binlog_format = row # optional log_bin = binary-log # mandatory, also on Slave! log_slave_updates = on # mandatory gtid_mode = on # mandatory enforce_gtid_consistency = on # mandatory server-id = 39 # mandatoryThis step requires a system restart (one minute downtime).
Preparing the Production Master standby database (Prod M2)On Master (Prod M1):
GRANT REPLICATION SLAVE ON *.* TO 'replication'@'192.168.1.%' IDENTIFIED BY 'secret'; mysqldump -u root --set-gtid-purged=on --master-data=2 --all-databases --triggers --routines --events > /tmp/full_dump.sqlOn Slave (Prod M2):
CHANGE MASTER TO MASTER_HOST='192.168.1.39', MASTER_PORT=3306 , MASTER_USER='replication', MASTER_PASSWORD='secret' , MASTER_AUTO_POSITION=1; RESET MASTER; -- On SLAVE! system mysql -u root < /tmp/full_dump.sql START SLAVE;To make it easier for a Slave to connect to its master we set a VIP in front of those 2 database servers (VIP Prod). This VIP should be used by all applications in the head quarter and also the filter engines.
Set-up filter engines (Filter BR and Filter CN)To make sure every manufacturing plant sees only the data it is allowed to see we need a filtering engine between the production site and the manufacturing plant (Filter BR and Filter CN).
To keep this filter engine lean we use a MySQL instance with all tables converted to the Blackhole Storage Engine:
# /etc/my.cnf [mysqld] binlog_format = row # optional log_bin = binary-log # mandatory, also on Slave! log_slave_updates = on # mandatory gtid_mode = on # mandatory enforce_gtid_consistency = on # mandatory server-id = 36 # mandatory default_storage_engine = blackholeOn the production master (Prod M1) we get the data as follows:
mysqldump -u root --set-gtid-purged=on --master-data=2 --triggers --routines --events --no-data --databases erp > /tmp/erp_dump_nd.sqlThe Filter Engines (Filter BR and CN) are set-up as follows::
-- Here we can use the VIP! CHANGE MASTER TO master_host='192.168.1.33', master_port=3306 , master_user='replication', master_password='secret' , master_auto_position=1; RESET MASTER; -- On SLAVE! system cat /tmp/erp_dump_nd.sql | sed 's/ ENGINE=[a-zA-Z]*/ ENGINE=blackhole/' | mysql -u root START SLAVE;Do not forget to also create the replication user on the filter engines.
GRANT REPLICATION SLAVE ON *.* TO 'replication'@'192.168.1.%' IDENTIFIED BY 'secret';Filtering out all non ERP schemata
We only want the erp schema to be replicated to the manufacturing plants, not the crm or the finance application. This we achieve with the following option on the filter engines:
# /etc/my.cnf [mysqld] replicate_do_db = erp replicate_ignore_table = erp.manufacturing_logMySQL row filtering
To achieve row filtering we use TRIGGERS. Make sure they are not replicated further down the hierarchy:
SET SESSION SQL_LOG_BIN = 0; use erp DROP TRIGGER IF EXISTS filter_row; delimiter // CREATE TRIGGER filter_row BEFORE INSERT ON manufacturing_data FOR EACH ROW BEGIN IF ( NEW.manufacture_plant != 'China' ) THEN SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Row was filtered out.' , CLASS_ORIGIN = 'FromDual filter trigger' , SUBCLASS_ORIGIN = 'filter_row' , CONSTRAINT_SCHEMA = 'erp' , CONSTRAINT_NAME = 'filer_row' , SCHEMA_NAME = 'erp' , TABLE_NAME = 'manufacturing_data' , COLUMN_NAME = '' , MYSQL_ERRNO = 1644 ; END IF; END; // delimiter ; SET SESSION SQL_LOG_BIN = 0;Up to now this would cause to stop replication for every filtered row. To avoid this we tell the Filtering Slaves to skip this error number:
# /etc/my.cnf [mysqld] slave_skip_errors = 1644Attaching production manufacturing Slaves (Man BR M1 and Man CN M1)
When we have finished everything on our head quarter site. We can start with the manufacturing sites (BR and CN):
On Master (Prod M1):
mysqldump -u root --set-gtid-purged=on --master-data=2 --triggers --routines --events --where='manufacture_plant="Brazil"' --databases erp > /tmp/erp_dump_br.sql mysqldump -u root --set-gtid-purged=on --master-data=2 --triggers --routines --events --where='manufacture_plant="China"' --databases erp > /tmp/erp_dump_cn.sqlOn the Manufacturing Masters (Man BR M1 and Man BR M2). Here we do NOT use a VIP because we think a blackhole storage engine is robust enough as master:
CHANGE MASTER TO master_host='192.168.1.43', master_port=3306 , master_user='replication', master_password='secret' , master_auto_position=1; RESET MASTER; -- On SLAVE! system cat /tmp/erp_dump_br.sql | mysql -u root START SLAVE;The standby manufacturing (Man BR M2 and Man CN M2) database is created in the same way as the production manufacturing database on the master.
Testing replication from HQ to manufacturing plantsFirst we make sure, crm and finance is not replicated out and replication also does not stop (on Prod M1):
INSERT INTO finance.accounting VALUES (NULL, 'test data over VIP', NULL); INSERT INTO finance.accounting VALUES (NULL, 'test data over VIP', NULL); INSERT INTO crm.customer VALUES (NULL, 'test data over VIP', NULL); INSERT INTO crm.customer VALUES (NULL, 'test data over VIP', NULL); UPDATE finance.accounting SET data = 'Changed data'; UPDATE crm.customer SET data = 'Changed data'; DELETE FROM finance.accounting WHERE id = 1; DELETE FROM crm.customer WHERE id = 1; SELECT * FROM finance.accounting; SELECT * FROM crm.customer; SHOW SLAVE STATUS\GThe schema filter seems to work correctly. Then we check if also the row filter works correctly. For this we have to run the queries in statement based replication (SBR)! Otherwise the trigger would not fire:
use mysql INSERT INTO erp.manufacturing_data VALUES (NULL, 'China', 'Highly secret manufacturing info as RBR.'); INSERT INTO erp.manufacturing_data VALUES (NULL, 'Brazil', 'Highly secret manufacturing info as RBR.'); -- This needs SUPER privilege... :-( SET SESSION binlog_format = STATEMENT; -- Caution those rows will NOT be replicated!!! -- See filter rules for SBR INSERT INTO erp.manufacturing_data VALUES (NULL, 'China', 'Highly secret manufacturing info as SBR lost.'); INSERT INTO erp.manufacturing_data VALUES (NULL, 'Brazil', 'Highly secret manufacturing info as SBR lost.'); use erp INSERT INTO manufacturing_data VALUES (NULL, 'China', 'Highly secret manufacturing info as SBR.'); INSERT INTO manufacturing_data VALUES (NULL, 'Brazil', 'Highly secret manufacturing info as SBR.'); INSERT INTO manufacturing_data VALUES (NULL, 'Germany', 'Highly secret manufacturing info as SBR.'); INSERT INTO manufacturing_data VALUES (NULL, 'Switzerland', 'Highly secret manufacturing info as SBR.'); SET SESSION binlog_format = ROW; SELECT * FROM erp.manufacturing_data;Production data back to head quarter
Now we have to take care about the production data on their way back to the HQ. To achieve this we use the new MySQL 5.7 feature called multi source replication. For multi source replication the replication repositories must be kept in tables instead of files:
# /etc/my.cnf [mysqld] master_info_repository = TABLE # mandatory relay_log_info_repository = TABLE # mandatoryThen we have to configure 2 replication channels from Prod M1 to their specific manufacturing masters over the VIP (VIP BR and VIP CN):
CHANGE MASTER TO MASTER_HOST='192.168.1.98', MASTER_PORT=3306 , MASTER_USER='replication', MASTER_PASSWORD='secret' , MASTER_AUTO_POSITION=1 FOR CHANNEL "manu_br"; CHANGE MASTER TO MASTER_HOST='192.168.1.99', MASTER_PORT=3306 , MASTER_USER='replication', MASTER_PASSWORD='secret' , MASTER_AUTO_POSITION=1 FOR CHANNEL "manu_cn"; START SLAVE FOR CHANNEL 'manu_br'; START SLAVE FOR CHANNEL 'manu_cn'; SHOW SLAVE STATUS FOR CHANNEL 'manu_br'\G SHOW SLAVE STATUS FOR CHANNEL 'manu_cn'\GAvoid to configure and activate the channels on Prod M2 as well.
Testing back replication from manufacturing plantsBrazil on Man BR M1:
INSERT INTO manufacturing_log VALUES (1, 'Production data from Brazil', 'data');China on Man CN M1:
INSERT INTO manufacturing_log VALUES (2, 'Production data from China', 'data');For testing:
SELECT * FROM manufacturing_log;Make sure you do not run into conflicts (Primary Key, AUTO_INCREMENTS). Make sure filtering is defined correctly!
To check the different channel states you can use the following command:
SHOW SLAVE STATUS\G or SELECT ras.channel_name, ras.service_state AS 'SQL_thread', ras.remaining_delay , CONCAT(user, '@', host, ':', port) AS user , rcs.service_state AS IO_thread, REPLACE(received_transaction_set, '\n', '') AS received_transaction_set FROM performance_schema.replication_applier_status AS ras JOIN performance_schema.replication_connection_configuration AS rcc ON rcc.channel_name = ras.channel_name JOIN performance_schema.replication_connection_status AS rcs ON ras.channel_name = rcs.channel_name ;Troubleshooting Inject empty transaction
If you try to skip a transaction as you did earlier (SQL_SLAVE_SKIP_COUNTER) you will face some problems:
STOP SLAVE; ERROR 1858 (HY000): sql_slave_skip_counter can not be set when the server is running with @@GLOBAL.GTID_MODE = ON. Instead, for each transaction that you want to skip, generate an empty transaction with the same GTID as the transactionTo skip the next transaction you have find the ones applied so far:
SHOW SLAVE STATUS\G ... Executed_Gtid_Set: c3611091-f80e-11e4-99bc-28d2445cb2e9:1-20then tell MySQL to skip this by injecting a new empty transaction:
SET SESSION GTID_NEXT='c3611091-f80e-11e4-99bc-28d2445cb2e9:21'; BEGIN; COMMIT; SET SESSION GTID_NEXT='AUTOMATIC'; SHOW SLAVE STATUS\G ... Executed_Gtid_Set: c3611091-f80e-11e4-99bc-28d2445cb2e9:1-21 START SLAVE;Revert from GTID-based replication to file/position-based replication
If you want to fall-back from MySQL GTID-based replication to file/position-based replication this is quite simple:
CHANGE MASTER TO MASTER_AUTO_POSITION = 0;MySQL Support and Engineering
If you need some help or support our MySQL support and engineering team is happy to help you.
FromDual Performance Monitor for MySQL and MariaDB 0.10.1 has been released
FromDual has the pleasure to announce the release of the new version 0.10.1 of its popular Database Performance Monitor for MySQL, MariaDB, Galera Cluster and Percona Server fpmmm.
You can download fpmmm from here.
In the inconceivable case that you find a bug in fpmmm please report it to our Bug-tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
This release contains various minor bug fixes and improvements. Mainly it was a rewrite from Perl into PHP.
New installation of fpmmm v0.10.1Please follow our mpm installation guide. A specific fpmmm installation guide will follow with the next version.
Prerequisites CentOS 6 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm yum update yum install zabbix-senderCentOS 7 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/7/x86_64/zabbix-release-2.2-1.el7.noarch.rpm yum update yum install zabbix-sender
Ubuntu 14.04 # apt-get install php5-cli php5-mysqlnd # cat << _EOF >/etc/php5/cli/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF apt-get install zabbix-agent
OpenSuSE 13.1 # zypper install php5 php5-posix php5-mysql php5-pcntl php5-curl #cat << _EOF >/etc/php5/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF zypper addrepo http://download.opensuse.org/repositories/server:/monitoring/openSUSE_13.1 server_monitoring zypper update zypper install zabbix-agent
Upgrade from mpm 0.x to fpmmm 0.10.1 # cd /opt # tar xf /download/fpmmm-0.10.1.tar.gz # rm -f fpmmm # ln -s fpmmm-0.10.1 fpmmm
You further have to replace all mysql_performance_monitor_agent.pl hooks (in Zabbix agent) by bin/fpmmm.
Changes in fpmmm v0.10.1 fpmmm agent- Defaults in fpmmm.conf.template adapted.
- Item sending prepared for empty strings.
- Log of cachedData more verbose.
- zabbix_sender stuff fixed and made more robust.
- New naming convention fpmmm implemented.
- Connection problems and error catching fixed.
- NDB cluster and PBXT stuff removed and locking made more robust.
- zabbix-sender on SuSE should work now as well. Bug #149.
- exec/system replaced by my_exec.
- Debug parameter replaced by LogLevel parameter, including all templates and warning for deprecation.
- export LC_ALL=C added to all O/S calls to make it language independent.
- Usage added (--help).
- All Perl code replaced by PHP code.
- Bug fix on cache clean-up on corruption.
- Mac OSX (darwin) support added, partially.
- Changed defaults away from /tmp This is no good location for persistent things because of RedHat tmpcleaner job.
- Debugging info added for missing lock file.
- none.
- Regexp pattern fixed for mr_versions.
- Trailing / from datadir is removed.
- Innodb buffer pool hit ratio rounded to 2 digits.
- Innodb_os_log_written item added.
- InnoDB hash searches item fixed.
- Galera status now gathered with causal reads off.
- Memcached stuff activated.
- Bug report on new DRBD fixed.
- Security template data typo fixed.
- Templates moved from xml folder to tpl folder and other file structure and renaming.
- InnoDB Flush Log at Transaction Commit Serverity lowered from Warning to Information.
For subscriptions of commercial use of fpmmm please get in contact with us.
Taxonomy upgrade extras: mysqlperformancemonitormonitoringfpmmmmaasperformance monitorreleaseFromDual Performance Monitor for MySQL and MariaDB 0.10.1 has been released
FromDual has the pleasure to announce the release of the new version 0.10.1 of its popular Database Performance Monitor for MySQL, MariaDB, Galera Cluster and Percona Server fpmmm.
You can download fpmmm from here.
In the inconceivable case that you find a bug in fpmmm please report it to our Bug-tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
This release contains various minor bug fixes and improvements. Mainly it was a rewrite from Perl into PHP.
New installation of fpmmm v0.10.1Please follow our mpm installation guide. A specific fpmmm installation guide will follow with the next version.
Prerequisites CentOS 6 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm yum update yum install zabbix-senderCentOS 7 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/7/x86_64/zabbix-release-2.2-1.el7.noarch.rpm yum update yum install zabbix-sender
Ubuntu 14.04 # apt-get install php5-cli php5-mysqlnd # cat << _EOF >/etc/php5/cli/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF apt-get install zabbix-agent
OpenSuSE 13.1 # zypper install php5 php5-posix php5-mysql php5-pcntl php5-curl #cat << _EOF >/etc/php5/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF zypper addrepo http://download.opensuse.org/repositories/server:/monitoring/openSUSE_13.1 server_monitoring zypper update zypper install zabbix-agent
Upgrade from mpm 0.x to fpmmm 0.10.1 # cd /opt # tar xf /download/fpmmm-0.10.1.tar.gz # rm -f fpmmm # ln -s fpmmm-0.10.1 fpmmm
You further have to replace all mysql_performance_monitor_agent.pl hooks (in Zabbix agent) by bin/fpmmm.
Changes in fpmmm v0.10.1 fpmmm agent- Defaults in fpmmm.conf.template adapted.
- Item sending prepared for empty strings.
- Log of cachedData more verbose.
- zabbix_sender stuff fixed and made more robust.
- New naming convention fpmmm implemented.
- Connection problems and error catching fixed.
- NDB cluster and PBXT stuff removed and locking made more robust.
- zabbix-sender on SuSE should work now as well. Bug #149.
- exec/system replaced by my_exec.
- Debug parameter replaced by LogLevel parameter, including all templates and warning for deprecation.
- export LC_ALL=C added to all O/S calls to make it language independent.
- Usage added (--help).
- All Perl code replaced by PHP code.
- Bug fix on cache clean-up on corruption.
- Mac OSX (darwin) support added, partially.
- Changed defaults away from /tmp This is no good location for persistent things because of RedHat tmpcleaner job.
- Debugging info added for missing lock file.
- none.
- Regexp pattern fixed for mr_versions.
- Trailing / from datadir is removed.
- Innodb buffer pool hit ratio rounded to 2 digits.
- Innodb_os_log_written item added.
- InnoDB hash searches item fixed.
- Galera status now gathered with causal reads off.
- Memcached stuff activated.
- Bug report on new DRBD fixed.
- Security template data typo fixed.
- Templates moved from xml folder to tpl folder and other file structure and renaming.
- InnoDB Flush Log at Transaction Commit Serverity lowered from Warning to Information.
For subscriptions of commercial use of fpmmm please get in contact with us.
Taxonomy upgrade extras: mysqlperformancemonitormonitoringfpmmmmaasperformance monitorreleaseFromDual Performance Monitor for MySQL and MariaDB 0.10.1 has been released
FromDual has the pleasure to announce the release of the new version 0.10.1 of its popular Database Performance Monitor for MySQL, MariaDB, Galera Cluster and Percona Server fpmmm.
You can download fpmmm from here.
In the inconceivable case that you find a bug in fpmmm please report it to our Bug-tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
This release contains various minor bug fixes and improvements. Mainly it was a rewrite from Perl into PHP.
New installation of fpmmm v0.10.1Please follow our mpm installation guide. A specific fpmmm installation guide will follow with the next version.
Prerequisites CentOS 6 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm yum update yum install zabbix-senderCentOS 7 # yum install php-cli php-process php-mysqli # cat << _EOF >/etc/php.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/7/x86_64/zabbix-release-2.2-1.el7.noarch.rpm yum update yum install zabbix-sender
Ubuntu 14.04 # apt-get install php5-cli php5-mysqlnd # cat << _EOF >/etc/php5/cli/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF apt-get install zabbix-agent
OpenSuSE 13.1 # zypper install php5 php5-posix php5-mysql php5-pcntl php5-curl #cat << _EOF >/etc/php5/conf.d/fpmmm.ini variables_order = "EGPCS" date.timezone = 'Europe/Zurich' _EOF zypper addrepo http://download.opensuse.org/repositories/server:/monitoring/openSUSE_13.1 server_monitoring zypper update zypper install zabbix-agent
Upgrade from mpm 0.x to fpmmm 0.10.1 # cd /opt # tar xf /download/fpmmm-0.10.1.tar.gz # rm -f fpmmm # ln -s fpmmm-0.10.1 fpmmm
You further have to replace all mysql_performance_monitor_agent.pl hooks (in Zabbix agent) by bin/fpmmm.
Changes in fpmmm v0.10.1 fpmmm agent- Defaults in fpmmm.conf.template adapted.
- Item sending prepared for empty strings.
- Log of cachedData more verbose.
- zabbix_sender stuff fixed and made more robust.
- New naming convention fpmmm implemented.
- Connection problems and error catching fixed.
- NDB cluster and PBXT stuff removed and locking made more robust.
- zabbix-sender on SuSE should work now as well. Bug #149.
- exec/system replaced by my_exec.
- Debug parameter replaced by LogLevel parameter, including all templates and warning for deprecation.
- export LC_ALL=C added to all O/S calls to make it language independent.
- Usage added (--help).
- All Perl code replaced by PHP code.
- Bug fix on cache clean-up on corruption.
- Mac OSX (darwin) support added, partially.
- Changed defaults away from /tmp This is no good location for persistent things because of RedHat tmpcleaner job.
- Debugging info added for missing lock file.
- none.
- Regexp pattern fixed for mr_versions.
- Trailing / from datadir is removed.
- Innodb buffer pool hit ratio rounded to 2 digits.
- Innodb_os_log_written item added.
- InnoDB hash searches item fixed.
- Galera status now gathered with causal reads off.
- Memcached stuff activated.
- Bug report on new DRBD fixed.
- Security template data typo fixed.
- Templates moved from xml folder to tpl folder and other file structure and renaming.
- InnoDB Flush Log at Transaction Commit Serverity lowered from Warning to Information.
For subscriptions of commercial use of fpmmm please get in contact with us.
MySQL Environment MyEnv 1.2.0 has been released
FromDual has the pleasure to announce the release of the new version 1.2.0 of its popular MySQL, Galera Cluster, MariaDB and Percona Server multi-instance environment MyEnv.
The new MyEnv can be downloaded here.
In the inconceivable case that you find a bug in the MyEnv please report it to our bug tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
Upgrade from 1.1.x to 1.2.0 # cd ${HOME}/product # tar xf /download/myenv-1.2.0.tar.gz # rm -f myenv # ln -s myenv-1.2.0 myenvIf you are using plug-ins for showMyEnvStatus create all the links in the new directory structure:
cd ${HOME}/product/myenv ln -s ../../utl/oem_agent.php plg/showMyEnvStatus/Changes in MyEnv 1.2.0 MyEnv
- Some minor fixes on init script.
- Introduction of states production, quality, testing and development.
- Colored prompt added for stage production databases.
- All /tmp/*sock moved to /var/run/mysqld.
- up output prepared for bind-address awareness.
- lsb_release dummy implemented.
- Advice for 2 sudo commands in row was wrong.
- Port conflict resolution more verbose.
- Init script replacement check added. Replace init script is distribution aware and auto executable.
- Installer made mode modular and prepared for automatization.
- Installer allows now to run as other user than mysql.
- Bugs in preparing myenv.conf fixed and instance name removed.
- Install routine made more distribution aware.
- Split partition (split_partition.php) improved.
- Fix of DROP PARTITION (drop_partition.php) was executed in wrong order (new before old).
- Alter Engine script rewritten into PHP (alter_engine.php).
- Alter Engine script handles ROW_FORMAT=FIXED problem correctly now.
- Alter Engine script handles too large Primary Key better.
- Alter Engine script recognizes tables with AUTO_INCREMENT column not at 1st position.
- Purge Binary Log rewritten into PHP (purge_binary_log.php).
- compare_status_files.pl added to compare output of 2 SHOW GLOBAL STATUS.
For subscriptions of commercial use of MyEnv please get in contact with us.
Taxonomy upgrade extras: MyEnvoperationMySQL Operationsmulti instanceconsolidationtestingreleaseMySQL Environment MyEnv 1.2.0 has been released
FromDual has the pleasure to announce the release of the new version 1.2.0 of its popular MySQL, Galera Cluster, MariaDB and Percona Server multi-instance environment MyEnv.
The new MyEnv can be downloaded here.
In the inconceivable case that you find a bug in the MyEnv please report it to our bug tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.
Upgrade from 1.1.x to 1.2.0 # cd ${HOME}/product # tar xf /download/myenv-1.2.0.tar.gz # rm -f myenv # ln -s myenv-1.2.0 myenvIf you are using plug-ins for showMyEnvStatus create all the links in the new directory structure:
cd ${HOME}/product/myenv ln -s ../../utl/oem_agent.php plg/showMyEnvStatus/Changes in MyEnv 1.2.0 MyEnv
- Some minor fixes on init script.
- Introduction of states production, quality, testing and development.
- Colored prompt added for stage production databases.
- All /tmp/*sock moved to /var/run/mysqld.
- up output prepared for bind-address awareness.
- lsb_release dummy implemented.
- Advice for 2 sudo commands in row was wrong.
- Port conflict resolution more verbose.
- Init script replacement check added. Replace init script is distribution aware and auto executable.
- Installer made mode modular and prepared for automatization.
- Installer allows now to run as other user than mysql.
- Bugs in preparing myenv.conf fixed and instance name removed.
- Install routine made more distribution aware.
- Split partition (split_partition.php) improved.
- Fix of DROP PARTITION (drop_partition.php) was executed in wrong order (new before old).
- Alter Engine script rewritten into PHP (alter_engine.php).
- Alter Engine script handles ROW_FORMAT=FIXED problem correctly now.
- Alter Engine script handles too large Primary Key better.
- Alter Engine script recognizes tables with AUTO_INCREMENT column not at 1st position.
- Purge Binary Log rewritten into PHP (purge_binary_log.php).
- compare_status_files.pl added to compare output of 2 SHOW GLOBAL STATUS.
For subscriptions of commercial use of MyEnv please get in contact with us.
Taxonomy upgrade extras: myenvoperationMySQL Operationsmulti instanceconsolidationtestingupgraderelease