Feed aggregator

Playing with MariaDB Vector for initial AI tests

Shinguz - Wed, 2024-08-28 10:51

Artificial intelligence (AI) and vector databases are on everyone's lips these days. Since MariaDB will soon be coming onto the market with vector database functionality, as a database consultant I thought it was about time I got to grips with the subject so that I have at least a hint of what it's all about...

Since I'm not so much of a theoretician but rather like to do something practical, I've built a small "AI" prototype that anyone can build very quickly and easily on their laptop (without a GPU)...

I also took the liberty of stealing the graphs from the MariaDB Foundation presentation (see sources at the end).

Downloading the MariaDB database with vector functionality

There are no MariaDB packages with vector functionality yet, but the source code is already available. So you can quickly build the binaries yourself. This took just under an hour on my old box. Once the binaries are built, you can make a tarball out of them:

# tar xf mariadb-11.6.0_vector.tar.gz # cd mariadb-11.6.0_vector/ # cmake . # make # make package

The MariaDB database then only needs to be started.

The model

To show the concept of tokenisation I decided to build an AI for URLs and to show the concept of different models and their potential for improvement I built a very stupid model in PHP, which simply decomposes a URL.

The question that this model should be able to answer is: "Give me similar URLs to the following URL."

The corresponding table looks like this:

DROP TABLE IF EXISTS `urls`; -- TRUNCATE TABLE is NOT sufficient!!! CREATE TABLE `urls` ( `id` INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY , `url` varchar(1024) DEFAULT NULL , `title` varchar(2000) DEFAULT NULL , `embedding` blob NOT NULL , VECTOR KEY `embedding` (`embedding`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 ;

The model fromdual_llm_v1 can be downloaded here.

This diagram from the MariaDB Foundation shows roughly how the whole thing works:

Training the AI

The database is then trained: The URL is taken as given and the title can be read out using an HTML scraper, for example. Here are 8 training datasets:

The vectors are then generated using our model:

(./fromdual_llm_v1.php https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source ./fromdual_llm_v1.php https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ ./fromdual_llm_v1.php https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf ./fromdual_llm_v1.php https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/ ./fromdual_llm_v1.php https://mariadb.org/projects/mariadb-vector/ ./fromdual_llm_v1.php https://metacpan.org/pod/Perl::Tokenizer ./fromdual_llm_v1.php https://www.qwak.com/post/utilizing-llms-with-embedding-stores ./fromdual_llm_v1.php https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma) | grep '^\[' [0.2, 0.0107421875, 0, 0, 0, 0.0006103515625, 0.00054931640625, 0] [0.2, 0.0107421875, 0, 0, 0, 0.00262451171875, 0, 0] [0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0] [0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0] [0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0] [0.2, 0.01171875, 0, 0, 0, 0.001220703125, 0, 0] [0.2, 0.01171875, 0, 0, 0, 0.0025634765625, 0, 0] [0.2, 0.009765625, 0, 0, 0, 0.00323486328125, 0, 0]

The database is now fed (trained) with these vectors:

INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source' , 'Download MariaDB Server' , VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00262451171875, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/' , 'Creating the MariaDB Binary Tarball' , VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0006103515625, 0.00054931640625, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf' , 'MariaDB Vector' , VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/' , 'MariaDB Vector preview is out' , VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://mariadb.org/projects/mariadb-vector/' , 'MariaDB Vector' , VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://metacpan.org/pod/Perl::Tokenizer' , 'Perl::Tokenizer - A tiny Perl code tokenizer' , VEC_FromText('[0.2, 0.01171875, 0, 0, 0, 0.001220703125, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://www.qwak.com/post/utilizing-llms-with-embedding-stores' , 'Integrating Vector Databases with LLMs: A Hands-On Guide' , VEC_FromText('[0.2, 0.01171875, 0, 0, 0, 0.0025634765625, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma' , 'LLM Model Enhanced with Vector DB' , VEC_FromText('[0.2, 0.009765625, 0, 0, 0, 0.00323486328125, 0, 0]') );

Here is an overview of what is now in the database:

SELECT id, url, title, VEC_ToText(embedding) FROM urls ; +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+ | 1 | https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source | Download MariaDB Server | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002625,0.000000,0.000000] | | 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] | | 3 | https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] | | 4 | https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/ | MariaDB Vector preview is out | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] | | 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] | | 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] | | 7 | https://www.qwak.com/post/utilizing-llms-with-embedding-stores | Integrating Vector Databases with LLMs: A Hands-On Guide | [0.200000,0.011719,0.000000,0.000000,0.000000,0.002563,0.000000,0.000000] | | 8 | https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma | LLM Model Enhanced with Vector DB | [0.200000,0.009766,0.000000,0.000000,0.000000,0.003235,0.000000,0.000000] | +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+
Search in the MariaDB vector database

Now comes the exciting part of the whole story: Can we also find something in our MariaDB vector database with URLs?

How this works schematically can again be seen in the MariaDB Foundation diagram:

The first attempt is a perfect match:

./fromdual_llm_v1.php https://mariadb.org/projects/mariadb-vector/ [0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0] SELECT id, url, title, VEC_ToText(embedding) FROM urls ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0]')) LIMIT 3 ; +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] | | 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] | | 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+

The first row matches 100%. Then the results get much worse relatively quickly...

Second attempt a similar URL:

./fromdual_llm_v1.php https://mariadb.com/kb/en/e4201/ [0.2, 0.0107421875, 0, 0, 0, 0.00079345703125, 0, 0] SELECT id, url, title, VEC_ToText(embedding) FROM urls ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00079345703125, 0, 0]')) LIMIT 3 ; +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] | | 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] | | 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+

Here I would expect only mariadb URLs among the first 3 hits. But this is not the case. So our model still has room for improvement here!

And another similar URL:

./fromdual_llm_v1.php https://mariadb.com/kb/en/vec_totext/ [0.2, 0.0107421875, 0, 0, 0, 0.0010986328125, 0, 0] SELECT id, url, title, VEC_ToText(embedding) FROM urls ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0010986328125, 0, 0]')) LIMIT 3 ; +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] | | 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] | | 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+

Same problem here. The hostname is not weighted enough. You can/must probably play with the scatter that mariadb.org and mariadb.com generate.

And last but not least, a URL that does not appear in the data set at all:

./fromdual_llm_v1.php https://www.mongodb.com/blog/post/vector-search-llm-essentials-what-when-why [0.2, 0.0146484375, 0, 0, 0, 0.00323486328125, 0, 0] SELECT id, url, title, VEC_ToText(embedding) FROM urls ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0146484375, 0, 0, 0, 0.00323486328125, 0, 0]')) LIMIT 5 ; +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+ | 7 | https://www.qwak.com/post/utilizing-llms-with-embedding-stores | Integrating Vector Databases with LLMs: A Hands-On Guide | [0.200000,0.011719,0.000000,0.000000,0.000000,0.002563,0.000000,0.000000] | | 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] | | 3 | https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] | | 4 | https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/ | MariaDB Vector preview is out | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] | | 1 | https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source | Download MariaDB Server | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002625,0.000000,0.000000] | +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+

Here the result seems to be completely arbitrary. But if you compare the vector of the query with the vectors of the results, the order makes sense... Are the dimensions in the vector evaluated from left to right? After all, the distance between two points in 8-dimensional space is to be determined...

Improvements in the model

The results of our AI are not yet particularly impressive. On the one hand, this is certainly due to the very limited amount of data, on the other hand, we have not yet modelled very important criteria in our model or have used completely nonsensical criteria.

Suggestions for improvement for a next model: The hostname could also be tokenised so that mariadb.com and mariadb.org are closer together.

The length of hostname, path, query and fragment is certainly not a particularly clever criterion for mapping the similarity of URLs. Much more intelligence would be needed in the model here. A function 1/CRC32(dim) might already provide slightly better results?

The title could be included, or at least the most important words (nouns, verbs) from the title.

The document type (MIME type) could be included: Is a PDF more similar to another PDF than to a CSV file or a static HMTL page or a dynamic PHP page?

Points that stood out while playing

The number of dimensions in a vector seem to be set on the first INSERT. If you subsequently enter data with a different vector length, the following error appears:

INSERT INTO products (name, description, embedding) VALUES ( 'Coffee Machine' , 'Built to make the best coffee you can imagine' , VEC_FromText('[0.2, 0.013671875, 0, 0, 0, 6.103515625E-5, 0, 0]') ); ERROR 1366 (22007): Incorrect vector value: '...' for column `test`.`products`.`embedding` at row 1

Changing the vector length is currently NOT possible with a TRUNCATE TABLE command. The table must be dropped (DROP TABLE) and created again (CREATE TABLE).

However, searching with a shorter vector is possible:

SELECT id, url, title, VEC_ToText(embedding) FROM urls ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875]')) LIMIT 3 ; +----+-------------------------------------------------------------------------+-------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+-------------------------------------------------------------------------+-------------------------------------+---------------------------------------------------------------------------+ | 8 | https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma | LLM Model Enhanced with Vector DB | [0.200000,0.009766,0.000000,0.000000,0.000000,0.003235,0.000000,0.000000] | | 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] | | 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] | +----+-------------------------------------------------------------------------+-------------------------------------+---------------------------------------------------------------------------+

I cannot (yet) judge whether the result is so useful.

Sources

Taxonomy upgrade extras: mariadb ai vector artificial intelligence

Spielen mit MariaDB Vector für erste KI-Tests

Oli Sennhauser - Tue, 2024-08-27 21:50

Künstliche Intelligenz (KI) und Vektor-Datenbanken sind heute in aller Munde. Da MariaDB demnächst auch mit Vektor-Datenbank-Funktionalität auf den Markt kommt, habe ich es als Datenbank-Berater für an der Zeit befunden mich etwas mit dem Thema zu beschäftigen, damit ich wenigstens einen Hauch Ahnung davon habe um was es geht...

Da ich nicht so der Theoretiker bin sondern eher gerne etwas praktisches mache, habe ich einen kleinen "KI" Prototypen gebaut, den jeder auf seinem Laptop (ohne GPU) sehr schnell und einfach nachbauen kann...

Ich habe mir auch erlaubt die Graphen aus dem Vortrag der MariaDB Foundation zu klauen (siehe Quellen am Ende).

Herunterladen der MariaDB Datenbank mit Vektor Funktionalität

Noch gib es keine MariaDB Pakete mit Vektor-Funktionalität, aber der Quellcode ist bereits verfügbar. Also baut man sich die Binaries halt schnell selber. Dies hat auf meiner alten Kiste eine knappe Stunde gedauert. Wenn die Binaries gebaut sind, kann man sich einen Tarball draus machen:

# tar xf mariadb-11.6.0_vector.tar.gz # cd mariadb-11.6.0_vector/ # cmake . # make # make package

Anschliessend muss die MariaDB Datenbank nur noch gestartet werden.

Das Modell

Um das Konzept des Tokenisierens zu zeigen habe ich mir vorgenommen ein KI für URLs zu bauen und um das Konzept der verschiedenen Modelle und deren Verbesserungspotenzial zu zeigen habe ich eine ganz dummes Modell in PHP gebaut, welches ganz einfach und simple eine URL zerlegt.

Die Frage, die mit diesem Modell beantwortet werden können sollte lautet: "Gib mir ähnliche URLs zur folgenden URL."

Die dazu gehörende Tabelle sieht wie folgt aus:

DROP TABLE IF EXISTS `urls`; -- TRUNCATE TABLE is NOT sufficient!!! CREATE TABLE `urls` ( `id` INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY , `url` varchar(1024) DEFAULT NULL , `title` varchar(2000) DEFAULT NULL , `embedding` blob NOT NULL , VECTOR KEY `embedding` (`embedding`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 ;

Das Modell fromdual_llm_v1 kann man hier herunter laden.

Wie das Ganze in etwa funktioniert kann man an diesem Schaubild der MariaDB Foundation entnehmen:

Trainieren der KI

Dann wird die Datenbank trainiert: Die URL wird als gegeben angenommen und der Title kann z.B. über einen HTML Scraper ausgelesen werden. Hier 8 Trainings-Datensätze:

Anschliessend werden mittels unseres Models die Vektoren generiert:

(./fromdual_llm_v1.php https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source ./fromdual_llm_v1.php https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ ./fromdual_llm_v1.php https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf ./fromdual_llm_v1.php https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/ ./fromdual_llm_v1.php https://mariadb.org/projects/mariadb-vector/ ./fromdual_llm_v1.php https://metacpan.org/pod/Perl::Tokenizer ./fromdual_llm_v1.php https://www.qwak.com/post/utilizing-llms-with-embedding-stores ./fromdual_llm_v1.php https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma) | grep '^\[' [0.2, 0.0107421875, 0, 0, 0, 0.0006103515625, 0.00054931640625, 0] [0.2, 0.0107421875, 0, 0, 0, 0.00262451171875, 0, 0] [0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0] [0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0] [0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0] [0.2, 0.01171875, 0, 0, 0, 0.001220703125, 0, 0] [0.2, 0.01171875, 0, 0, 0, 0.0025634765625, 0, 0] [0.2, 0.009765625, 0, 0, 0, 0.00323486328125, 0, 0]

Mit diesen Vektoren wird jetzt die Datenbank gefüttert (trainiert):

INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source' , 'Download MariaDB Server' , VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00262451171875, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/' , 'Creating the MariaDB Binary Tarball' , VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0006103515625, 0.00054931640625, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf' , 'MariaDB Vector' , VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/' , 'MariaDB Vector preview is out' , VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://mariadb.org/projects/mariadb-vector/' , 'MariaDB Vector' , VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://metacpan.org/pod/Perl::Tokenizer' , 'Perl::Tokenizer - A tiny Perl code tokenizer' , VEC_FromText('[0.2, 0.01171875, 0, 0, 0, 0.001220703125, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://www.qwak.com/post/utilizing-llms-with-embedding-stores' , 'Integrating Vector Databases with LLMs: A Hands-On Guide' , VEC_FromText('[0.2, 0.01171875, 0, 0, 0, 0.0025634765625, 0, 0]') ); INSERT INTO `urls` (id, url, title, embedding) VALUES ( NULL , 'https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma' , 'LLM Model Enhanced with Vector DB' , VEC_FromText('[0.2, 0.009765625, 0, 0, 0, 0.00323486328125, 0, 0]') );

Hier mal zuerst ein Überblick, was jetzt in der Datenbank drin steht:

SELECT id, url, title, VEC_ToText(embedding) FROM urls ; +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+ | 1 | https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source | Download MariaDB Server | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002625,0.000000,0.000000] | | 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] | | 3 | https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] | | 4 | https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/ | MariaDB Vector preview is out | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] | | 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] | | 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] | | 7 | https://www.qwak.com/post/utilizing-llms-with-embedding-stores | Integrating Vector Databases with LLMs: A Hands-On Guide | [0.200000,0.011719,0.000000,0.000000,0.000000,0.002563,0.000000,0.000000] | | 8 | https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma | LLM Model Enhanced with Vector DB | [0.200000,0.009766,0.000000,0.000000,0.000000,0.003235,0.000000,0.000000] | +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+
Suche in der MariaDB Vektor Datenbank

Jetzt kommt der spannend Teil der ganzen Geschichte: Finden wir auch was in unserer MariaDB Vektor Datenbank mit URLs?

Wie das schematisch vor sich geht kann wieder dem Schaubild der MariaDB Foundation entnommen werden:

Als erster Versuch ein perfekter Treffer:

./fromdual_llm_v1.php https://mariadb.org/projects/mariadb-vector/ [0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0] SELECT id, url, title, VEC_ToText(embedding) FROM urls ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0]')) LIMIT 3 ; +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] | | 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] | | 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+

Die erste Zeile stimmt zu 100% überein. Dann werden die Resultate relativ schnell viel schlechter...

Zweiter Versuch eine ähnliche URL:

./fromdual_llm_v1.php https://mariadb.com/kb/en/e4201/ [0.2, 0.0107421875, 0, 0, 0, 0.00079345703125, 0, 0] SELECT id, url, title, VEC_ToText(embedding) FROM urls ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00079345703125, 0, 0]')) LIMIT 3 ; +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] | | 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] | | 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+

Hier würde ich unter den ersten 3 Treffern nur mariadb URLs erwarten. Dies ist aber nicht der Fall. Hier hat unser Modell also noch Verbesserungspotential!

Und noch eine andere ähnliche URL:

./fromdual_llm_v1.php https://mariadb.com/kb/en/vec_totext/ [0.2, 0.0107421875, 0, 0, 0, 0.0010986328125, 0, 0] SELECT id, url, title, VEC_ToText(embedding) FROM urls ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0010986328125, 0, 0]')) LIMIT 3 ; +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+ | 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] | | 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] | | 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] | +----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+

Selbes Problem hier. Der Hostname wird zu wenig gewichtet. Wahrscheinlich kann/muss man hier mit der Streuung spielen welche mariadb.org und mariadb.com erzeugen.

Und zu guter letzt eine URL die gar nicht im Datensatz vorkommt:

./fromdual_llm_v1.php https://www.mongodb.com/blog/post/vector-search-llm-essentials-what-when-why [0.2, 0.0146484375, 0, 0, 0, 0.00323486328125, 0, 0] SELECT id, url, title, VEC_ToText(embedding) FROM urls ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0146484375, 0, 0, 0, 0.00323486328125, 0, 0]')) LIMIT 5 ; +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+ | 7 | https://www.qwak.com/post/utilizing-llms-with-embedding-stores | Integrating Vector Databases with LLMs: A Hands-On Guide | [0.200000,0.011719,0.000000,0.000000,0.000000,0.002563,0.000000,0.000000] | | 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] | | 3 | https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] | | 4 | https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/ | MariaDB Vector preview is out | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] | | 1 | https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source | Download MariaDB Server | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002625,0.000000,0.000000] | +----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+

Hier scheint das Resultat völlig willkürlich zu sein. Wenn man sich den Vektor der Abfrage mit den Vektoren der Resultate vergleicht macht die Reihenfolge aber schon Sinn... Werden die Dimensionen im Vektor von links nach rechts ausgewertet? Es soll ja der Abstand von zwei Punkten im 8-dimensionalen Raum bestimmt werden...

Verbesserungen im Modell

Die Resultate unserer KI sind noch nicht so sonderlich berauschend. Einerseits liegt das sicher an der sehr beschränkten Datenmenge, andererseits haben wir auch sehr wichtige Kriterien in unserem Model noch nicht abgebildet oder völlig unsinnige Kriterien verwendet.

Verbesserungsvorschläge für ein nächstes Modell: Der hostname könnte zusätzlich noch tokenisiert werden, damit könnte man erreichen dass mariadb.com und mariadb.org näher beieinander liegen.

Die Länge von hostname, path, query und fragment ist sicherlich kein sonderlich schlaues Kriterium um eine Änhlichkeit von URLs abzubilden. Hier wäre also wesentlich mehr Intelligenz im Modell vonnöten. Eine Funktion 1/CRC32(dim) liefert ev. bereits minimal bessere Resultate?

titel könnte mit einbezogen werden, oder zumindest die wichtigsten Worte (Nomen, Verben) aus dem Titel.

Der Dokumententyp (MIME type) könnte mit einbezogen werden: Ist ein PDF ähnlicher zu einem anderen PDF als zu einer CSV Datei oder einer statischen HMTL Seite oder einer dynamischen PHP Seite?

Punkte die beim Spielen aufgefallen sind

Die Anzahl der Dimensionen in einem Vektor scheinen beim ersten INSERT festgelegt zu werden. Wenn man anschliessend Daten mit einer anderen Vektorlänge eingibt erscheint folgender Fehler:

INSERT INTO products (name, description, embedding) VALUES ( 'Coffee Machine' , 'Built to make the best coffee you can imagine' , VEC_FromText('[0.2, 0.013671875, 0, 0, 0, 6.103515625E-5, 0, 0]') ); ERROR 1366 (22007): Incorrect vector value: '...' for column `test`.`products`.`embedding` at row 1

Ein Änderen der Vector-Länge ist mit einem TRUNCATE TABLE Befehl zur Zeit noch NICHT möglich. Die Tabelle muss gelöscht (DROP TABLE) und wieder kreiert (CREATE TABLE) werden.

Das Suchen mit einem kürzeren Vector geht hingegen:

SELECT id, url, title, VEC_ToText(embedding) FROM urls ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875]')) LIMIT 3 ; +----+-------------------------------------------------------------------------+-------------------------------------+---------------------------------------------------------------------------+ | id | url | title | VEC_ToText(embedding) | +----+-------------------------------------------------------------------------+-------------------------------------+---------------------------------------------------------------------------+ | 8 | https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma | LLM Model Enhanced with Vector DB | [0.200000,0.009766,0.000000,0.000000,0.000000,0.003235,0.000000,0.000000] | | 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] | | 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] | +----+-------------------------------------------------------------------------+-------------------------------------+---------------------------------------------------------------------------+

Ob das Resultat so sinnvoll ist, kann ich (noch) nicht beurteilen.

Quellen

Taxonomy upgrade extras: mariadb ki ai vector artificial intelligence künstliche intelligenz vektor

Categories: FromDual all (de), FromDual TechFeed (de)

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.3.1 has been released

Shinguz - Mon, 2024-08-12 16:25

FromDual has the pleasure to announce the release of the new version 2.3.1 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

The new FromDual Backup and Recovery Manager can be downloaded from here. The FromDual Repositories were updated. How to install and use the Backup and Recovery Manager is described in FromDual Backup and Recovery Manager (brman) installation guide.

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please report it to the FromDual Bugtracker or just send us an email.

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.3.1 shell> cd ${HOME}/product shell> tar xf /download/brman-2.3.1.tar.gz shell> rm -f brman shell> ln -s brman-2.3.1 brman
Changes in FromDual Backup and Recovery Manager 2.3.1

This release is a new minor release. It contains mainly bug fixes. We have tried to maintain backward-compatibility with the 1.2, 2.0, 2.1 and 2.2 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

shell> fromdual_bman --version shell> bman --version shell> rman --version
General

Testing: LXC library updated.
Testing: Test preparation improved.
General: CHANGELOG updated.
General: rc made unique.
Documentation: Documentation improved.
Documentation: Create user documented.
Documentation: New release notes added to documentation.
Packaging: Distro Debian 10 removed.

FromDual Backup Manager (bman)

Utility mariabackup replaced by mariadb-backup.
Branch guessing improved.
Code refactored.
No realistic password is used any more.
Error message for missing client utilities made more distro specific.

FromDual Recovery Manager (rman)

Branch guessing improved.
Code refactored.
No realistic password is used any more.
Error message for missing client utilities made more distro specific.
Typo error fixed, tests refactored.
In some error cases all processes where killed instead of just the dummy database daemon. This is caught correctly now.
Error message improved to get more information in case of errors.
Physical schema backup leads to dump which is wrong. Now this is caught and aborted with an error.

Subscriptions for commercial use of FromDual Backup and Recovery Manager you can get from from us.

Taxonomy upgrade extras: Backup Restore Recovery pitr brman release bman rman FromDual Backup and Recovery Manager

Categories: FromDual all (en), FromDual TechFeed (en), MySQL Tech-Feed (en)

Partial physical database restore for MariaDB and MySQL

Shinguz - Tue, 2024-07-02 18:13

What is it about?

When describing backup and restore scenarios, a full backup and a full restore of the database instance (mariadbd/mysqld) are usually assumed. This means that the entire database instance, including all databases (schemas), is backed up and restored.

In practice, however, the situation is often different: An entire database instance is not to be restored, but only individual databases or even individual tables, because only these have broken.

In many cases, this can be done quite easily with the tools mariadb-dump/mariadb or mysqldump/mysql (logical backup). However, if the database or table is very large, the restore will not be completed in a reasonable time (some minutes to a few hours).

This is exactly where the so-called partial physical restore comes into play. Partial stands for one or more tables (or an entire database), physical for: Individual SQL statements are not executed, but the data files are physically restored. In this scenario, very large amounts of data can be restored very quickly, provided the appropriate infrastructure is in place. Rule of thumb: On fat hardware: 1 Tbyte per hour. In this way, database restores can be carried out very quickly.

MariaDB and MySQL already offer this functionality out of the box. The mechanism is reasonably practicable for individual tables (see Restore partial backup). For entire databases with possibly dozens or hundreds of tables, however, the on-board mechanism is very cumbersome and error-prone.

Use case

This is exactly where the new functionality of the FromDual Backup and Recovery Manager (brman) v2.3.0 comes into play: it considerably simplifies the partial physical database restore.

A second scenario in which this new functionality can also be used is when moving a large database from one database instance to another (e.g. from Dev to Prod).

Preparations for the partial physical database restore

In order to be able to restore a database, a clean backup must of course first be available. This can either be created with the FromDual Backup Manager (bman):

PORT=3306 BACKUPNAME=bck_full_2024-07-01 BACKUPDIR=/tmp/bck ./brman/bin/bman --target=brman:secret@127.0.0.1:${PORT} --type=full --mode=physical --policy=daily --backupdir=${BACKUPDIR} --backup-name=${BACKUPNAME} --no-compress

or you can simply create the backup with the MariaDB (mariadb-backup) or MySQL on-board tools (xtrabackup):

PORT=3306 BACKUPNAME=bck_full_2024-07-01 BACKUPDIR=/tmp/bck POLICY=daily mariadb-backup --user=brman --password=secret --host=127.0.0.1 --port=${PORT} --backup --target-dir=${BACKUPDIR}/${POLICY}/${BACKUPNAME} mariadb-backup --user=brman --password=secret --host=127.0.0.1 --port=${PORT} --prepare --target-dir=${BACKUPDIR}/${POLICY}/${BACKUPNAME}
Partial physical database restore

To perform a partial physical database restore, the database must be running, in contrast to a complete physical restore.

The partial physical database restore is then simple:

PORT=3306 DATADIR=/var/lib/mysql BACKUPNAME=bck_full_2024-07-01 BACKUPDIR=/tmp/bck ./brman/bin/rman --target=brman:secret@127.0.0.1:${PORT} --type=schema --mode=physical --policy=daily --schema=test --log=/tmp/rman.log --backupdir=${BACKUPDIR} --datadir=${DATADIR} --backup-name=${BACKUPNAME} ... Start restore at 2024-07-01 16:29:48 Backup with tool mariabackup version 10.11.8 (from path /home/mysql/product/mariadb-10.11/bin/mariabackup). Parent: We are the parent. Our child is: 63712. Waiting for database daemon... Child: We are the child: Starting database daemon... Child: Change ownership of database files (/tmp/bck/daily/bck_full_2024-07-01) to mysql Child: /home/mysql/product/mariadb-10.11/bin/mariadbd --no-defaults --user=mysql --basedir=/home/mysql/product/mariadb-10.11 --datadir=/tmp/bck/daily/bck_full_2024-07-01 --log-error=/tmp/my.err --port=3360 --socket=/tmp/my.sock --lower-case-table-names=0 Parent: Tables not InnoDB or sequences: 0 Parent: Tables with partitions: 0 Parent: Tables with full-text index: 0 Parent: InnoDB table `test` found to restore Parent: Dump database test Parent: /home/mysql/product/mariadb-10.11/bin/mariadb-dump --user=brman --host=127.0.0.1 --port=3360 --routines --events --triggers --no-data --skip-lock-tables --add-drop-database --databases test Parent: Shutdown backup database. Restore empty database test Prepare and export tables: /home/mysql/product/mariadb-10.11/bin/mariabackup --user=brman --host=127.0.0.1 --port=3321 --prepare --export --databases=test --target-dir=/tmp/bck/daily/bck_full_2024-07-01 SET SESSION foreign_key_checks = 0 SET SESSION sql_log_bin = off Restore table test ALTER TABLE `test`.`test` DISCARD TABLESPACE cp /tmp/bck/daily/bck_full_2024-07-01/test/test.cfg /home/mysql/database/mariadb-1011/data/test/test.cfg cp /tmp/bck/daily/bck_full_2024-07-01/test/test.ibd /home/mysql/database/mariadb-1011/data/test/test.ibd chown mysql: /home/mysql/database/mariadb-1011/data/test/test.cfg /home/mysql/database/mariadb-1011/data/test/test.ibd ALTER TABLE `test`.`test` IMPORT TABLESPACE rm /home/mysql/database/mariadb-1011/data/test/test.cfg rm /tmp/bck/daily/bck_full_2024-07-01/test/test.cfg ---------------------------------------- WARNING: You should restart the database now! Otherwise possible future backups may fail. See MDEV-34418 (https://jira.mariadb.org/browse/MDEV-34418). ---------------------------------------- Restore time was: 0d 0h 0' 2" End restore at 2024-07-01 16:29:50 (rc=0)

For MariaDB, it is recommended to restart the database afterwards until the bug MDEV-34418: mariadb-backup fails on database which was partially restored with mariadb-backup is fixed. This step can be omitted for MySQL.

Restrictions

At present, the following restrictions still apply to the partial physical restore of databases with rman:

Only entire databases can be restored. Restoring individual tables is not yet implemented. Use the basic on-board tools for this.
Restoring partitioned tables is not yet implemented. Use the basic on-board tools for partitioned tables.
A subsequent point-in-time recovery of the database is not yet implemented and must be carried out manually.
A partial physical database restore for an entire Galera cluster is not yet implemented and must be performed manually. In this case, a restore to one Galera node and a subsequent synchronisation of the other nodes using SST is recommended.
With a physical partial database restore, a pseudo instance is started on the backup files. This pseudo instance requires a free port 3360.
The backup files must already be in a consistent state (--prepare).
With a partial physical database restore, a logical backup of the database is created without the data on the pseudo instance. This backup is restored to the instance to be repaired. This means that all objects (views, triggers, functions, procedures, events, etc.) that were created AFTER the complete physical backup are deleted before the partial physical database restore and are then no longer available.
The original database instance from which the backup was created and the instance on which the restore is performed must have the same setting for lower_case_table_names.
All three, the backup, the database instance and the rman tool must be located on the same machine.
The backup must currently still be available in uncompressed form.

Literature

Taxonomy upgrade extras: partial restore Restore database schema

Categories: FromDual all (en), FromDual TechFeed (en), MySQL Tech-Feed (en)

Partieller physischer Datenbank-Restore für MariaDB und MySQL

Oli Sennhauser - Mon, 2024-07-01 16:52

Um was geht es?

Bei der Beschreibung von Backup- und /Restore-Szenarien wird in der Regel immer von einem vollständigen Backup (full backup) und einem vollständigen Restore (full restore) der Datenbankinstanz (mariadbd/mysqld) ausgegangen. Das bedeutet, dass die gesamte Datenbankinstanz inklusive aller Datenbanken (Schemata) gesichert und wiederhergestellt wird.

In der Praxis sieht die Situation jedoch oft anders aus: Es soll nicht eine ganze Datenbankinstanz wiederhergestellt werden, sondern nur einzelne Datenbanken oder gar einzelne Tabellen, weil nur diese kaputt gegangen sind.

Dies kann in vielen Fällen recht einfach mit den Tools mariadb-dump/mariadb oder mysqldump/mysql (logisches Backup) bewerkstelligt werden. Wenn die Datenbank oder die Tabelle jedoch sehr gross ist, wird die Wiederherstellung nicht in angemessener Zeit (einige Minuten bis wenige Stunden) abgeschlossen sein.

Genau in diesem Fall kommt der sogenannte partielle physische Restore ins Spiel. Partiell steht für eine oder mehrere Tabellen (oder eine ganze Datenbank), physisch für: Es werden nicht einzelne SQL-Anweisungen ausgeführt, sondern die Datenfiles werden physisch zurückgespielt. In diesem Szenario können, eine entsprechende Infrastruktur vorausgesetzt, sehr schnell sehr grosse Datenbestände zurückgespielt werden. Faustregel: Auf fetter Hardware: 1 Tbyte pro Stunde. Auf diese Weise können Datenbank-Restores sehr schnell durchgeführt werden.

MariaDB und MySQL bieten diese Funktionalität bereits von Haus aus an. Für einzelne Tabellen ist der Mechanismus einigermassen praktikabel (siehe Restore partial Backup). Für ganze Datenbanken mit möglicherweise Dutzenden oder Hunderten von Tabellen ist der Bord-Mechanismus jedoch sehr umständlich und fehleranfällig.

Anwendungsfall

Genau hier kommt die neue Funktionalität des FromDual Backup and Recovery Managers (brman) v2.3.0 ins Spiel: Er vereinfacht den partiellen physischen Datenbank-Restore erheblich.

Ein zweites Szenario, in welchem diese neue Funktionalität ebenfalls genutzt werden kann, ist der Umzug einer grossen Datenbank von einer Datenbankinstanz in eine andere Datenbankinstanz (z. B. von Dev nach Prod).

Vorbereitungen für den partiellen physischen Datenbank-Restore

Um eine Datenbank wiederherstellen zu können, muss natürlich zunächst ein sauberes Backup vorliegen. Dieses kann entweder mit dem FromDual Backup Manager (bman) erstellt werden:

PORT=3306 BACKUPNAME=bck_full_2024-07-01 BACKUPDIR=/tmp/bck ./brman/bin/bman --target=brman:secret@127.0.0.1:${PORT} --type=full --mode=physical --policy=daily --backupdir=${BACKUPDIR} --backup-name=${BACKUPNAME} --no-compress

oder man kann das Backup auch einfach mit den MariaDB- (mariadb-backup) oder MySQL-Bordmitteln (xtrabackup) erstellen:

PORT=3306 BACKUPNAME=bck_full_2024-07-01 BACKUPDIR=/tmp/bck POLICY=daily mariadb-backup --user=brman --password=secret --host=127.0.0.1 --port=${PORT} --backup --target-dir=${BACKUPDIR}/${POLICY}/${BACKUPNAME} mariadb-backup --user=brman --password=secret --host=127.0.0.1 --port=${PORT} --prepare --target-dir=${BACKUPDIR}/${POLICY}/${BACKUPNAME}
Partieller physischer Datenbank-Restore

Um einen partiellen physischen Datenbank-Restore durchzuführen, muss die Datenbank, im Gegensatz zum vollständigen physischen Restore, laufen.

Der partielle physische Datenbank-Restore ist dann einfach:

PORT=3306 DATADIR=/var/lib/mysql BACKUPNAME=bck_full_2024-07-01 BACKUPDIR=/tmp/bck ./brman/bin/rman --target=brman:secret@127.0.0.1:${PORT} --type=schema --mode=physical --policy=daily --schema=test --log=/tmp/rman.log --backupdir=${BACKUPDIR} --datadir=${DATADIR} --backup-name=${BACKUPNAME} ... Start restore at 2024-07-01 16:29:48 Backup with tool mariabackup version 10.11.8 (from path /home/mysql/product/mariadb-10.11/bin/mariabackup). Parent: We are the parent. Our child is: 63712. Waiting for database daemon... Child: We are the child: Starting database daemon... Child: Change ownership of database files (/tmp/bck/daily/bck_full_2024-07-01) to mysql Child: /home/mysql/product/mariadb-10.11/bin/mariadbd --no-defaults --user=mysql --basedir=/home/mysql/product/mariadb-10.11 --datadir=/tmp/bck/daily/bck_full_2024-07-01 --log-error=/tmp/my.err --port=3360 --socket=/tmp/my.sock --lower-case-table-names=0 Parent: Tables not InnoDB or sequences: 0 Parent: Tables with partitions: 0 Parent: Tables with full-text index: 0 Parent: InnoDB table `test` found to restore Parent: Dump database test Parent: /home/mysql/product/mariadb-10.11/bin/mariadb-dump --user=brman --host=127.0.0.1 --port=3360 --routines --events --triggers --no-data --skip-lock-tables --add-drop-database --databases test Parent: Shutdown backup database. Restore empty database test Prepare and export tables: /home/mysql/product/mariadb-10.11/bin/mariabackup --user=brman --host=127.0.0.1 --port=3321 --prepare --export --databases=test --target-dir=/tmp/bck/daily/bck_full_2024-07-01 SET SESSION foreign_key_checks = 0 SET SESSION sql_log_bin = off Restore table test ALTER TABLE `test`.`test` DISCARD TABLESPACE cp /tmp/bck/daily/bck_full_2024-07-01/test/test.cfg /home/mysql/database/mariadb-1011/data/test/test.cfg cp /tmp/bck/daily/bck_full_2024-07-01/test/test.ibd /home/mysql/database/mariadb-1011/data/test/test.ibd chown mysql: /home/mysql/database/mariadb-1011/data/test/test.cfg /home/mysql/database/mariadb-1011/data/test/test.ibd ALTER TABLE `test`.`test` IMPORT TABLESPACE rm /home/mysql/database/mariadb-1011/data/test/test.cfg rm /tmp/bck/daily/bck_full_2024-07-01/test/test.cfg ---------------------------------------- WARNING: You should restart the database now! Otherwise possible future backups may fail. See MDEV-34418 (https://jira.mariadb.org/browse/MDEV-34418). ---------------------------------------- Restore time was: 0d 0h 0' 2" End restore at 2024-07-01 16:29:50 (rc=0)

Bei MariaDB empfiehlt es sich, die Datenbank anschliessend neu zu starten, bis der Bug MDEV-34418: mariadb-backup fails on database which was partially restored with mariadb-backup behoben ist. Für MySQL kann dieser Schritt entfallen.

Einschränkungen

Zur Zeit sind noch folgende Einschränkungen beim partiellen physischen Restore von Datenbanken mit rman zu beachten:

Es können nur ganze Datenbanken zurückgesichert werden. Das zurücksichern von einzelne Tabellen ist derzeit noch nicht implementiert. Verwenden Sie hierfür die Basis-Bordwerkzeuge.
Die Wiederherstellung partitionierter Tabellen ist derzeit noch nicht implementiert. Verwenden Sie dazu die Basis-Bordwerkzeuge für partitionierte Tabellen.
Ein anschliessendes Point-in-Time-Recovery der Datenbank ist noch nicht implementiert und muss manuell durchgeführt werden.
Ein partielles physisches Datenbank-Restore für einen gesamten Galera-Cluster ist derzeit noch nicht implementiert und muss manuell durchgeführt werden. In diesem Fall wird ein Restore auf einen Galera-Knoten und eine anschliessende Synchronisation der anderen Knoten mittels SST empfohlen.
Beim physischen partiellen Datenbank-Restore wird eine Pseudo-Instanz auf den Backup-Dateien gestartet. Diese Pseudoinstanz benötigt einen freien Port 3360.
Die Backup-Dateien müssen bereits in einem konsistenten Zustand vorliegen (--prepare).
Beim partiellen physischen Datenbank-Restore wird ein logisches Backup der Datenbank ohne die Daten auf der Pseudo-Instanz erstellt. Diese Sicherung wird auf die zu reparierende Instanz zurückgespielt. Das bedeutet, dass alle Objekte (Views, Trigger, Functions, Procedures, Events, etc.), die NACH dem vollständigen physischen Backup erstellt wurden, vor dem partiellen physischen Datenbank-Restore gelöscht werden und anschliessend nicht mehr vorhanden sind.
Die ursprüngliche Datenbankinstanz, von der das Backup erstellt wurde, und die Instanz, auf der der Restore durchgeführt wird, müssen die gleiche Einstellung für lower_case_table_names haben.
Sowohl das Backup als auch die Datenbankinstanz und das rman-Werkzeug müssen sich auf derselben Maschine befinden.
Das Backup muss zur Zeit noch in unkomprimierter Form vorliegen.

Literatur

Taxonomy upgrade extras: partial restore Restore database schema

Categories: FromDual all (de), FromDual TechFeed (de)

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.3.0 has been released

Shinguz - Tue, 2024-06-25 15:12

FromDual has the pleasure to announce the release of the new version 2.3.0 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

The new FromDual Backup and Recovery Manager can be downloaded from here. The FromDual Repositories were updated. How to install and use the Backup and Recovery Manager is described in FromDual Backup and Recovery Manager (brman) installation guide.

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please report it to the FromDual Bugtracker or just send us an email.

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.3.0 shell> cd ${HOME}/product shell> tar xf /download/brman-2.3.0.tar.gz shell> rm -f brman shell> ln -s brman-2.3.0 brman
Changes in FromDual Backup and Recovery Manager 2.3.0

This release is a new major release. The most important new feature is the partial physical restore. We have tried to maintain backward-compatibility with the 1.2, 2.0, 2.1 and 2.2 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

shell> fromdual_bman --version shell> bman --version shell> rman --version
General

Packaging: Release el9 added and debian format increase from 9 to 10 because of debian12.
Documentation prepared for Asciidoctor.
Distribtion dependent things fixed.
Build script improved.
Typos fixed.
All print replaced by standard PHP echo commands.
Redhat replace by Red Hat and CentOS by Rocky Linux.
Copyright year updated to 2024.
Library myEnv updated (getDistribution, lsb_release removed).

FromDual Backup Manager (bman)

Bug in sftp transfer for archiving files fixed.
Also create subdirectories only once if necessary in physical backup.
Archive location directory is only created once in physical backup. This avoids errors and makes the backup slightly faster.
In function createArchiveDestinationDirectory php-ssh2 calls where replaced by the more stable scp/sftp calls.
Bug in archive destination directory creation fixed and example improved.
Library LXClib added for testing.
Number of lines in dump parsing increased because of changes in MySQL 8.0.
Binlog backup debug output made nicer.
Retention time increased from 800 days to 1200 days (more than 3 years), tests added for this, newer DB versions added to tests.
Advices apt-get and yum replace by apt and dnf.
Error was not returned correctly when bman aborts because of MyISAM tables. Fixed.
Syntax error fixed.
flock on fpmmm file added.
BINLOG MONITOR privilege check added for MariaDB 10.11.
Code clean-up.
Example enhanced.
Output made more clear for trouble shooting (doCleanup).

FromDual Recovery Manager (rman)

Various bugs fixed found during automated testing.
Some minor fixes about timing and repeatable testing, warnings and messages improved.
mariadb_backup_binlog_info case (MariaDB 11.4) added for physical full backup.
Code refactored and case when backup is compressed caught.
Function doPhysicalRestore renamed to doFullPhysicalRestore.
Split restore functions in their own files.
sql_log_bin is used for schema restore on partial physical restore.
New example for partial physical restore added.
New method for partial physical restore implemented.
Partial physical schema restore added.

Subscriptions for commercial use of FromDual Backup and Recovery Manager you can get from from us.

Taxonomy upgrade extras: Backup Restore Recovery pitr brman release bman rman FromDual Backup and Recovery Manager

Categories: FromDual all (en), FromDual TechFeed (en), MySQL Tech-Feed (en)

Shrinking the InnoDB system tablespace

Shinguz - Mon, 2024-06-17 14:45

One feature that really excited me in the new MariaDB 11.4 LTS release is the shrinking of the system tablespace (ibdata1). I have been eagerly waiting for this feature since around 2006 and now it has finally arrived with MariaDB 11.4.
Actually, this feature has been available since MariaDB 11.2 IR (June 2023).

Unfortunately, the announcement of this feature came a little too short. In the MariaDB release notes it says succinctly:

The InnoDB system tablespace is now shrunk by reclaiming unused space at startup (MDEV-14795)

From the MariaDB 11.2.0 Release Notes.

The reasons why this file can grow immeasurably have actually been known for a long time and the measures against it are also clear (see literature). But again and again we see MariaDB users out in the field who didn't realise the problem or realised it too late and are now left with an ibdata1 file that is far too large...

How can the problem be provoked?

The problem can be provoked by creating a table in the system tablespace:

SQL> SET global innodb_file_per_table = off; SQL> CREATE TABLE `test` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `data` varchar(128) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(), PRIMARY KEY (`id`) ) ENGINE=InnoDB;

and then filling it with data:

SQL> INSERT INTO test SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW() ; SQL> INSERT INTO test SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW() FROM test LIMIT 1000000 ; ...

While the table is being filled, you can observe how the file ibdata1 swells on the file system:

$ while [ 1 ] ; do ll -h ibdata1 ; sleep 5 ; done -rw-rw---- 1 mysql mysql 12M Jun 2 13:57 ibdata1 -rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1 -rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1 -rw-rw---- 1 mysql mysql 140M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 204M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 268M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 332M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 396M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 460M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 524M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 588M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 652M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 716M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 780M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 844M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 908M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 972M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.1G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.2G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.3G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:00 ibdata1

If the ibdata1 file is large enough, you can move the table from the system tablespace to a dedicated tablespace:

SQL> SET global innodb_file_per_table = off; SQL> ALTER TABLE test.test FORCE; Query OK, 0 rows affected (33.764 sec)

And you can see how the new file is created:

$ ll -h ibdata1 test/* -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:01 ibdata1 -rw-rw---- 1 mysql mysql 1.1K Jun 12 14:01 test/#sql-alter-dca30-12.frm -rw-rw---- 1 mysql mysql 696M Jun 12 14:01 test/#sql-alter-dca30-12.ibd -rw-rw---- 1 mysql mysql 1.1K Jun 12 13:56 test/test.frm -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:01 ibdata1 -rw-rw---- 1 mysql mysql 1.1K Jun 12 14:01 test/test.frm -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:02 test/test.ibd

So now we have once the data but twice as much space used.

And how can you make the system tablespace smaller again?

Unfortunately, this information is somewhat hidden and has to be gathered from the documentation and the MariaDB Jira issues (see literature):

SQL> SET GLOBAL innodb_fast_shutdown=0; SQL> SHUTDOWN;

When shutting down, you can see the corresponding entries in the MariaDB error log:

[Note] bin/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown [Note] InnoDB: FTS optimize thread exiting. [Note] InnoDB: Truncating system tablespace from 90880 to 768 pages [Note] InnoDB: System tablespace truncated successfully [Note] InnoDB: Starting shutdown... [Note] InnoDB: Dumping buffer pool(s) to /home/mysql/database/mariadb-114/data/ib_buffer_pool [Note] InnoDB: Restricted to 2016 pages due to innodb_buf_pool_dump_pct=25 [Note] InnoDB: Buffer pool(s) dump completed at 240612 14:11:11 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" [Note] InnoDB: Shutdown completed; log sequence number 4011132308; transaction id 139 [Note] bin/mariadbd: Shutdown complete

And if you look at the file ibdata1 on disc afterwards, it is as small as it was at the beginning of the experiment:

$ ll ibdata1* -h -rw-rw---- 1 mysql mysql 12M Jun 12 14:11 ibdata1
Literature

Shrinking the InnoDB system tablespace

Shinguz - Mon, 2024-06-17 14:45

One feature that really excited me in the new MariaDB 11.4 LTS release is the shrinking of the system tablespace (ibdata1). I have been eagerly waiting for this feature since around 2006 and now it has finally arrived with MariaDB 11.4.
Actually, this feature has been available since MariaDB 11.2 IR (June 2023).

Unfortunately, the announcement of this feature came a little too short. In the MariaDB release notes it says succinctly:

The InnoDB system tablespace is now shrunk by reclaiming unused space at startup (MDEV-14795)

From the MariaDB 11.2.0 Release Notes.

The reasons why this file can grow immeasurably have actually been known for a long time and the measures against it are also clear (see literature). But again and again we see MariaDB users out in the field who didn't realise the problem or realised it too late and are now left with an ibdata1 file that is far too large...

How can the problem be provoked?

The problem can be provoked by creating a table in the system tablespace:

SQL> SET global innodb_file_per_table = off; SQL> CREATE TABLE `test` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `data` varchar(128) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(), PRIMARY KEY (`id`) ) ENGINE=InnoDB;

and then filling it with data:

SQL> INSERT INTO test SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW() ; SQL> INSERT INTO test SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW() FROM test LIMIT 1000000 ; ...

While the table is being filled, you can observe how the file ibdata1 swells on the file system:

$ while [ 1 ] ; do ll -h ibdata1 ; sleep 5 ; done -rw-rw---- 1 mysql mysql 12M Jun 2 13:57 ibdata1 -rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1 -rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1 -rw-rw---- 1 mysql mysql 140M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 204M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 268M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 332M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 396M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 460M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 524M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 588M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 652M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 716M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 780M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 844M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 908M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 972M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.1G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.2G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.3G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:00 ibdata1

If the ibdata1 file is large enough, you can move the table from the system tablespace to a dedicated tablespace:

SQL> SET global innodb_file_per_table = off; SQL> ALTER TABLE test.test FORCE; Query OK, 0 rows affected (33.764 sec)

And you can see how the new file is created:

$ ll -h ibdata1 test/* -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:01 ibdata1 -rw-rw---- 1 mysql mysql 1.1K Jun 12 14:01 test/#sql-alter-dca30-12.frm -rw-rw---- 1 mysql mysql 696M Jun 12 14:01 test/#sql-alter-dca30-12.ibd -rw-rw---- 1 mysql mysql 1.1K Jun 12 13:56 test/test.frm -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:01 ibdata1 -rw-rw---- 1 mysql mysql 1.1K Jun 12 14:01 test/test.frm -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:02 test/test.ibd

So now we have once the data but twice as much space used.

And how can you make the system tablespace smaller again?

Unfortunately, this information is somewhat hidden and has to be gathered from the documentation and the MariaDB Jira issues (see literature):

SQL> SET GLOBAL innodb_fast_shutdown=0; SQL> SHUTDOWN;

When shutting down, you can see the corresponding entries in the MariaDB error log:

[Note] bin/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown [Note] InnoDB: FTS optimize thread exiting. [Note] InnoDB: Truncating system tablespace from 90880 to 768 pages [Note] InnoDB: System tablespace truncated successfully [Note] InnoDB: Starting shutdown... [Note] InnoDB: Dumping buffer pool(s) to /home/mysql/database/mariadb-114/data/ib_buffer_pool [Note] InnoDB: Restricted to 2016 pages due to innodb_buf_pool_dump_pct=25 [Note] InnoDB: Buffer pool(s) dump completed at 240612 14:11:11 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" [Note] InnoDB: Shutdown completed; log sequence number 4011132308; transaction id 139 [Note] bin/mariadbd: Shutdown complete

And if you look at the file ibdata1 on disc afterwards, it is as small as it was at the beginning of the experiment:

$ ll ibdata1* -h -rw-rw---- 1 mysql mysql 12M Jun 12 14:11 ibdata1
Literature

Verkleinern des InnoDB-System-Tablespaces

Oli Sennhauser - Wed, 2024-06-12 15:27

Ein Feature, das mich im neuen MariaDB 11.4 LTS Release wirklich begeistert hat, ist das Verkleinern bzw. Schrumpfen des System-Tablespaces (ibdata1). Auf dieses Feature habe ich seit ca. 2006 sehnsüchtig gewartet und nun ist es mit MariaDB 11.4 endlich gekommen.
Eigentlich gibt es dieses Feature schon seit dem MariaDB 11.2 IR (Juni 2023).

Leider ist die Ankündigung dieses Features etwas zu kurz gekommen. In den MariaDB Release Notes heisst es lapidar:

The InnoDB system tablespace is now shrunk by reclaiming unused space at startup (MDEV-14795)

Aus den MariaDB 11.2.0 Release Notes.

Die Gründe, warum dieses Datei ins Unermessliche wachsen kann, sind eigentlich schon lange bekannt und die Massnahmen dagegen sind auch klar (siehe Literatur). Nur sehen wir immer wieder MariaDB-Anwender draussen im Feld, die das Problem nicht oder zu spät auf dem Schirm hatten und nun mit einer viel zu grossen ibdata1-Datei da stehen...

Wie kann das Problem provoziert werden?

Das Problem kann provoziert werden, indem man eine Tabelle im System-Tablespace anlegt:

SQL> SET global innodb_file_per_table = off; SQL> CREATE TABLE `test` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `data` varchar(128) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(), PRIMARY KEY (`id`) ) ENGINE=InnoDB;

und diese dann mit Daten befüllt:

SQL> INSERT INTO test SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW() ; SQL> INSERT INTO test SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW() FROM test LIMIT 1000000 ; ...

Während die Tabelle gefüllt wird, kann man auf dem Dateisystem beobachten, wie die Datei ibdata1 anschwillt:

$ while [ 1 ] ; do ll -h ibdata1 ; sleep 5 ; done -rw-rw---- 1 mysql mysql 12M Jun 2 13:57 ibdata1 -rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1 -rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1 -rw-rw---- 1 mysql mysql 140M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 204M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 268M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 332M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 396M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 460M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 524M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 588M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 652M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 716M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 780M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 844M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 908M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 972M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.1G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.2G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.3G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:00 ibdata1

Wenn die Datei ibdata1 gross genug ist, kann man die Tabelle vom System-Tablespace in einen dedizierten Tablespace verschieben:

SQL> SET global innodb_file_per_table = off; SQL> ALTER TABLE test.test FORCE; Query OK, 0 rows affected (33.764 sec)

Und man sieht, wie die neue Datei aufgebaut wird:

$ ll -h ibdata1 test/* -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:01 ibdata1 -rw-rw---- 1 mysql mysql 1.1K Jun 12 14:01 test/#sql-alter-dca30-12.frm -rw-rw---- 1 mysql mysql 696M Jun 12 14:01 test/#sql-alter-dca30-12.ibd -rw-rw---- 1 mysql mysql 1.1K Jun 12 13:56 test/test.frm -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:01 ibdata1 -rw-rw---- 1 mysql mysql 1.1K Jun 12 14:01 test/test.frm -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:02 test/test.ibd

Wir haben jetzt also einmal die Daten aber doppelte so viel Platz verbraucht.

Und wie kann man den System-Tablespace wieder verkleinern?

Diese Information ist leider etwas versteck und muss aus der Dokumentation und den MariaDB Jira Issues (siehe Literatur) zusammengesucht werden:

SQL> SET GLOBAL innodb_fast_shutdown=0; SQL> SHUTDOWN;

Beim Herunterfahren sieht man die entsprechenden Einträge im MariaDB Error Log:

[Note] bin/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown [Note] InnoDB: FTS optimize thread exiting. [Note] InnoDB: Truncating system tablespace from 90880 to 768 pages [Note] InnoDB: System tablespace truncated successfully [Note] InnoDB: Starting shutdown... [Note] InnoDB: Dumping buffer pool(s) to /home/mysql/database/mariadb-114/data/ib_buffer_pool [Note] InnoDB: Restricted to 2016 pages due to innodb_buf_pool_dump_pct=25 [Note] InnoDB: Buffer pool(s) dump completed at 240612 14:11:11 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" [Note] InnoDB: Shutdown completed; log sequence number 4011132308; transaction id 139 [Note] bin/mariadbd: Shutdown complete

Und wenn man sich die Datei ibdata1 danach auf Platte anschaut, ist sie wieder so klein wie zu Beginn des Experiments:

$ ll ibdata1* -h -rw-rw---- 1 mysql mysql 12M Jun 12 14:11 ibdata1
Literatur

Siehe auch:

UNDO logs in InnoDB system tablespace ibdata1
Shrinking InnoDB system tablespace file ibdata1 PoC
Who else is using my memory - File System Cache analysis
InnoDB system tablespace cannot be shrunk (MDEV-14795)
ibdata1 shrinking (MDEV-31462)
InnoDB System Tablespaces - Decreasing the Size

Taxonomy upgrade extras: innodb tablespace ibdata1

Categories: FromDual all (de), FromDual TechFeed (de)

Verkleinern des InnoDB-System-Tablespaces

Oli Sennhauser - Wed, 2024-06-12 15:27

Ein Feature, das mich im neuen MariaDB 11.4 LTS Release wirklich begeistert hat, ist das Verkleinern bzw. Schrumpfen des System-Tablespaces (ibdata1). Auf dieses Feature habe ich seit ca. 2006 sehnsüchtig gewartet und nun ist es mit MariaDB 11.4 endlich gekommen.
Eigentlich gibt es dieses Feature schon seit dem MariaDB 11.2 IR (Juni 2023).

Leider ist die Ankündigung dieses Features etwas zu kurz gekommen. In den MariaDB Release Notes heisst es lapidar:

The InnoDB system tablespace is now shrunk by reclaiming unused space at startup (MDEV-14795)

Aus den MariaDB 11.2.0 Release Notes.

Die Gründe, warum dieses Datei ins Unermessliche wachsen kann, sind eigentlich schon lange bekannt und die Massnahmen dagegen sind auch klar (siehe Literatur). Nur sehen wir immer wieder MariaDB-Anwender draussen im Feld, die das Problem nicht oder zu spät auf dem Schirm hatten und nun mit einer viel zu grossen ibdata1-Datei da stehen...

Wie kann das Problem provoziert werden?

Das Problem kann provoziert werden, indem man eine Tabelle im System-Tablespace anlegt:

SQL> SET global innodb_file_per_table = off; SQL> CREATE TABLE `test` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `data` varchar(128) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(), PRIMARY KEY (`id`) ) ENGINE=InnoDB;

und diese dann mit Daten befüllt:

SQL> INSERT INTO test SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW() ; SQL> INSERT INTO test SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW() FROM test LIMIT 1000000 ; ...

Während die Tabelle gefüllt wird, kann man auf dem Dateisystem beobachten, wie die Datei ibdata1 anschwillt:

$ while [ 1 ] ; do ll -h ibdata1 ; sleep 5 ; done -rw-rw---- 1 mysql mysql 12M Jun 2 13:57 ibdata1 -rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1 -rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1 -rw-rw---- 1 mysql mysql 140M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 204M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 268M Jun 12 13:58 ibdata1 -rw-rw---- 1 mysql mysql 332M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 396M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 460M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 524M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 588M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 652M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 716M Jun 12 13:59 ibdata1 -rw-rw---- 1 mysql mysql 780M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 844M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 908M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 972M Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.1G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.2G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.3G Jun 12 14:00 ibdata1 -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:00 ibdata1

Wenn die Datei ibdata1 gross genug ist, kann man die Tabelle vom System-Tablespace in einen dedizierten Tablespace verschieben:

SQL> SET global innodb_file_per_table = off; SQL> ALTER TABLE test.test FORCE; Query OK, 0 rows affected (33.764 sec)

Und man sieht, wie die neue Datei aufgebaut wird:

$ ll -h ibdata1 test/* -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:01 ibdata1 -rw-rw---- 1 mysql mysql 1.1K Jun 12 14:01 test/#sql-alter-dca30-12.frm -rw-rw---- 1 mysql mysql 696M Jun 12 14:01 test/#sql-alter-dca30-12.ibd -rw-rw---- 1 mysql mysql 1.1K Jun 12 13:56 test/test.frm -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:01 ibdata1 -rw-rw---- 1 mysql mysql 1.1K Jun 12 14:01 test/test.frm -rw-rw---- 1 mysql mysql 1.4G Jun 12 14:02 test/test.ibd

Wir haben jetzt also einmal die Daten aber doppelte so viel Platz verbraucht.

Und wie kann man den System-Tablespace wieder verkleinern?

Diese Information ist leider etwas versteck und muss aus der Dokumentation und den MariaDB Jira Issues (siehe Literatur) zusammengesucht werden:

SQL> SET GLOBAL innodb_fast_shutdown=0; SQL> SHUTDOWN;

Beim Herunterfahren sieht man die entsprechenden Einträge im MariaDB Error Log:

[Note] bin/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown [Note] InnoDB: FTS optimize thread exiting. [Note] InnoDB: Truncating system tablespace from 90880 to 768 pages [Note] InnoDB: System tablespace truncated successfully [Note] InnoDB: Starting shutdown... [Note] InnoDB: Dumping buffer pool(s) to /home/mysql/database/mariadb-114/data/ib_buffer_pool [Note] InnoDB: Restricted to 2016 pages due to innodb_buf_pool_dump_pct=25 [Note] InnoDB: Buffer pool(s) dump completed at 240612 14:11:11 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" [Note] InnoDB: Shutdown completed; log sequence number 4011132308; transaction id 139 [Note] bin/mariadbd: Shutdown complete

Und wenn man sich die Datei ibdata1 danach auf Platte anschaut, ist sie wieder so klein wie zu Beginn des Experiments:

$ ll ibdata1* -h -rw-rw---- 1 mysql mysql 12M Jun 12 14:11 ibdata1
Literatur

Siehe auch:

UNDO logs in InnoDB system tablespace ibdata1
Shrinking InnoDB system tablespace file ibdata1 PoC
Who else is using my memory - File System Cache analysis
InnoDB system tablespace cannot be shrunk (MDEV-14795)
ibdata1 shrinking (MDEV-31462)
InnoDB System Tablespaces - Decreasing the Size

Taxonomy upgrade extras: innodb tablespace ibdata1

Categories: FromDual all (de), FromDual TechFeed (de)

FromDual Ops Center 1.2.2 for MariaDB and MySQL databases has been released

Shinguz - Mon, 2024-04-29 17:41

FromDual is pleased to announce the release of the new version 1.2.2 of the popular FromDual Ops Center focmm, a Graphical User Interface (GUI) for MariaDB, MySQL and compatible databases.

The FromDual Ops Center for MariaDB and MySQL (focmm) helps DBAs and system administrators to better manage their MariaDB and MySQL databases and Galera Cluster farms. Ops Center makes life easier for DBAs and Admins!

The main purpose of the Ops Center is to help you with your daily MariaDB and MySQL operations. More information about FromDual Ops Center can be found in the General Information section.

Download now

The new FromDual Ops Center for MariaDB and MySQL (focmm) can be downloaded from our download page. How to install and use focmm is documented in the Ops Center User Guide.

In the unlikely event that you find a bug in the FromDual Ops Center for MariaDB and MySQL please report it to the FromDual bug tracker or just send us an email.

Any feedback, comments and testimonials are also welcome! Please send them to feedback@fromdual.com.

Installing Ops Center 1.2.2

Information on how to install FromDual Ops Center can be found in the Ops Center User Guide.

Upgrading from 0.9.x to 1.2.2

The upgrade from 0.9.x to 1.2.2 should be automatic. Please make a backup of your Ops Center database instance before upgrading! See also Upgrading.

Important: In some cases the directory focmm/tmp/start_jobs.lock is missing. In this case jobs will not start. Please check the log file under focmm/log/start_jobs.log and create the folder accordingly. There should also be a file called pid in this folder. Create this file if it does not exist.

Downgrading from 1.2.2

Downgrading from 1.2.2 to 1.2.1 should work.

Changes in Ops Center 1.2.2 Machine

Machine information is automatically gathered after first successful check now.
Machine function calls optimized.
Ssh key check added for machine.
Some tests around machine library added.
Function createPublicKey + page refactored.
Copy of ssh key and ssh command implemented.
Skip machines which are down during gathering resource costs.
Error message in pricing when adding a new machine was fixed.

Instance

Instance operations are now faster when instance is running but not reachable.
Node check improved in cluster and instance, works now also correctly on multi instance environments.
PID file is gathered now.
Stop instance on instance operations improved if a Galera cluster node.
Instance is now checked immediately after start or restart and checks are set to failed after instance stop. So the instance state is more appropriate and in time now.
Link added to read_only variable comment.
Instance is shown orange when ping check was not run yet. This is fixed now.
Error log gathering improved in instance operations.
If mysql_upgrade_info file is missing error is suppressed now.
Gather instance information is done now right after the first successful check.
Check database_ping was removed from general instance tests to avoid noise in the database error log.
Check instance_ping_check removed from instanceTest to avoid error log messages on a regular base, wondering what kind of impact this has...
innodb_log_writer_threads rule added to instance configuration.
Target connect function error logging improved.
Continue button is handled correctly now when stopping, starting or restarting an instance.
Instance operations location of error log is shown now.
Refactored instance operations code for moving Galera buttons out.
Function refreshInstance refactored because we need this information for Galera Cluster.
Unknown array_keys error message fixed in instance.
Foreign MariaDB/MySQL repositories should now be considered as well.
Optimized instance handling.
Instance error log handling can now handle systemd/journalctl.
Instance hardening when creating instance improved.
Create instance improved.
Create instance bug fixed.
Instance create on Rocky 9, fixed.
Create instance for Rocky 8 improved.
Check for restart also removed, so also restart works now.
Check removed to allow a running but not reachable instance to be restarted, restart and start button added.
Non existing configuration was not handled correctly, fixed.
Edit instance placeholder added and default instance name changed from mysqld to mariadbd.
Create instance output made nicer in case of failure.

Cluster

Cluster overview signal lights are also working now for Galera.
Cluster operations returns faster now if machine is not reachable.
Cluster show checks non important information removed.
Cluster checks in menu are now shown correctly.
grastate.dat async check added.
Cluster check made async.
Instance state added to cluster operations and check view.
M/S cluster tests made more robust against strange situations.
Some bugs in M/S cluster fixed.
focmm user passwords should sync in galera.
Bootstrap should distribute focmm user on all nodes.
Bootstrap on Debian fixed again.
Error message for missing grastate.dat file is suppressed now.
Deploy configuration button is now on a new line.
2 little bugs in M/S cluster operations removed.
Instance operations improved with Galera clusters.
Soft bootstrap implemented.
Cluster node bootstrap, start stop fixed.
Function bootstrap implemented.
Async rolling cluster restart added.
Problem with rolling restart fixed.
Cluster check and operations is read if not OK. Failed, same behaviour as before!
Rolling cluster restart and other buttons added to cluster operations.
Variable wsrep_node_name is set to instance name per default.
Galera cluster config deployment made smoother and fixed bug.
Sort order of cluster nodes and title of table clarified.
Page start/stop notices added to cluster.php.
Indention problem in function getClusterChecks fixed.

Load Balancer

Terms load-balancer and loadbalancer replaced by load balancer.
User radmin added for ProxySQL monitoring.
Socket default for HAproxy changed to the new standards.

Virtual IP (VIP)/Floating IP

Status grey image does not exist in vip_show_checks, fixed.

Tools

Crontab: Missing library in check_instances.php added.
Crontab: html tags removed in mail.
Crontab: Error messages and error handling in crontab and start jobs improved, lock file directory is created automatically now.

Configuration

No changes.

Database-as-a-Service (DBaaS)

No changes.

Building and Packaging

Bulid moved to MyEnv project.
Missing Rocky Linux branch added.
Package redhat-lsb-core does not exist on Rocky 9 any more, spec file adapted.
Package build test fixed.

Themes / UI

jquery updated from 3.6.3 to 3.7.1

General

Modification for license key automation added.
Package installation ssh-keygen is only called if key does not exist. So upgrade should work without error/warning.
Some tests fixed, bug in createPublicKey fixed.
Function listRemoteDirectory does not write needlessly error messages any more.
HTML tag placeholder added for add function, add object and save object made more error prone.
Variable tx_isolation replaced by transaction_isolation which was deprecated in MariaDB 11.2 and MySQL 5.7.
Code clean-up.
Some menu items opened a new tab which was not intended. Fixed.
Field naming made consistent.
Error handling and error messages improved.
Copyright year updated from 2023 to 2024.
Message downgraded from ERROR to DEBUG.
Function runRemoteCommand will return an error code again when going_back_to functionality implemented.
Function renderDropDown newline was removed.
Function renderSubmitButton accepts a title now.
Function runRemoteCommand can suppress error logging now.
Command apt-get replaced by apt.
Distribution information refactored.
Function whoami cleaned-up.
PHP 8.0 function calls removed again because of Debian 10/11 support.
User interaction made unique among all 5 different objects.
Redundant DEBUG information removed.
Configuration file was removed again when not needed any more (clean-up).
Command yum replaced by dnf, also apt-get partly.
Command which replaced by type -p to make it Rocky 9 compliant.
Some more lsb_release stuff removed.
myEnv.inc library updated, lsb_release removed.
Dead code message added.

Repository

No changes.

Documentation

Version changed from 1.2.0 to 1.2.1 and TLS conflict on ubuntu 22.04 fixed.
Build automatized.
Documentation for M/S updated.
All text taken over from CMS.
Load balancer documentation added.

Taxonomy upgrade extras: Operations release FromDual Ops Center ops center dbaas focmm

Categories: FromDual all (en), FromDual TechFeed (en), MySQL Tech-Feed (en)

FromDual Ops Center 1.2.2 for MariaDB and MySQL databases has been released

Shinguz - Mon, 2024-04-29 17:41

FromDual is pleased to announce the release of the new version 1.2.2 of the popular FromDual Ops Center focmm, a Graphical User Interface (GUI) for MariaDB, MySQL and compatible databases.

The FromDual Ops Center for MariaDB and MySQL (focmm) helps DBAs and system administrators to better manage their MariaDB and MySQL databases and Galera Cluster farms. Ops Center makes life easier for DBAs and Admins!

The main purpose of the Ops Center is to help you with your daily MariaDB and MySQL operations. More information about FromDual Ops Center can be found in the General Information section.

Download now

The new FromDual Ops Center for MariaDB and MySQL (focmm) can be downloaded from our download page. How to install and use focmm is documented in the Ops Center User Guide.

In the unlikely event that you find a bug in the FromDual Ops Center for MariaDB and MySQL please report it to the FromDual bug tracker or just send us an email.

Any feedback, comments and testimonials are also welcome! Please send them to feedback@fromdual.com.

Installing Ops Center 1.2.2

Information on how to install FromDual Ops Center can be found in the Ops Center User Guide.

Upgrading from 0.9.x to 1.2.2

The upgrade from 0.9.x to 1.2.2 should be automatic. Please make a backup of your Ops Center database instance before upgrading! See also Upgrading.

Important: In some cases the directory focmm/tmp/start_jobs.lock is missing. In this case jobs will not start. Please check the log file under focmm/log/start_jobs.log and create the folder accordingly. There should also be a file called pid in this folder. Create this file if it does not exist.

Downgrading from 1.2.2

Downgrading from 1.2.2 to 1.2.1 should work.

Changes in Ops Center 1.2.2 Machine

Machine information is automatically gathered after first successful check now.
Machine function calls optimized.
Ssh key check added for machine.
Some tests around machine library added.
Function createPublicKey + page refactored.
Copy of ssh key and ssh command implemented.
Skip machines which are down during gathering resource costs.
Error message in pricing when adding a new machine was fixed.

Instance

Instance operations are now faster when instance is running but not reachable.
Node check improved in cluster and instance, works now also correctly on multi instance environments.
PID file is gathered now.
Stop instance on instance operations improved if a Galera cluster node.
Instance is now checked immediately after start or restart and checks are set to failed after instance stop. So the instance state is more appropriate and in time now.
Link added to read_only variable comment.
Instance is shown orange when ping check was not run yet. This is fixed now.
Error log gathering improved in instance operations.
If mysql_upgrade_info file is missing error is suppressed now.
Gather instance information is done now right after the first successful check.
Check database_ping was removed from general instance tests to avoid noise in the database error log.
Check instance_ping_check removed from instanceTest to avoid error log messages on a regular base, wondering what kind of impact this has...
innodb_log_writer_threads rule added to instance configuration.
Target connect function error logging improved.
Continue button is handled correctly now when stopping, starting or restarting an instance.
Instance operations location of error log is shown now.
Refactored instance operations code for moving Galera buttons out.
Function refreshInstance refactored because we need this information for Galera Cluster.
Unknown array_keys error message fixed in instance.
Foreign MariaDB/MySQL repositories should now be considered as well.
Optimized instance handling.
Instance error log handling can now handle systemd/journalctl.
Instance hardening when creating instance improved.
Create instance improved.
Create instance bug fixed.
Instance create on Rocky 9, fixed.
Create instance for Rocky 8 improved.
Check for restart also removed, so also restart works now.
Check removed to allow a running but not reachable instance to be restarted, restart and start button added.
Non existing configuration was not handled correctly, fixed.
Edit instance placeholder added and default instance name changed from mysqld to mariadbd.
Create instance output made nicer in case of failure.

Cluster

Cluster overview signal lights are also working now for Galera.
Cluster operations returns faster now if machine is not reachable.
Cluster show checks non important information removed.
Cluster checks in menu are now shown correctly.
grastate.dat async check added.
Cluster check made async.
Instance state added to cluster operations and check view.
M/S cluster tests made more robust against strange situations.
Some bugs in M/S cluster fixed.
focmm user passwords should sync in galera.
Bootstrap should distribute focmm user on all nodes.
Bootstrap on Debian fixed again.
Error message for missing grastate.dat file is suppressed now.
Deploy configuration button is now on a new line.
2 little bugs in M/S cluster operations removed.
Instance operations improved with Galera clusters.
Soft bootstrap implemented.
Cluster node bootstrap, start stop fixed.
Function bootstrap implemented.
Async rolling cluster restart added.
Problem with rolling restart fixed.
Cluster check and operations is read if not OK. Failed, same behaviour as before!
Rolling cluster restart and other buttons added to cluster operations.
Variable wsrep_node_name is set to instance name per default.
Galera cluster config deployment made smoother and fixed bug.
Sort order of cluster nodes and title of table clarified.
Page start/stop notices added to cluster.php.
Indention problem in function getClusterChecks fixed.

Load Balancer

Terms load-balancer and loadbalancer replaced by load balancer.
User radmin added for ProxySQL monitoring.
Socket default for HAproxy changed to the new standards.

Virtual IP (VIP)/Floating IP

Status grey image does not exist in vip_show_checks, fixed.

Tools

Crontab: Missing library in check_instances.php added.
Crontab: html tags removed in mail.
Crontab: Error messages and error handling in crontab and start jobs improved, lock file directory is created automatically now.

Configuration

No changes.

Database-as-a-Service (DBaaS)

No changes.

Building and Packaging

Bulid moved to MyEnv project.
Missing Rocky Linux branch added.
Package redhat-lsb-core does not exist on Rocky 9 any more, spec file adapted.
Package build test fixed.

Themes / UI

jquery updated from 3.6.3 to 3.7.1

General

Modification for license key automation added.
Package installation ssh-keygen is only called if key does not exist. So upgrade should work without error/warning.
Some tests fixed, bug in createPublicKey fixed.
Function listRemoteDirectory does not write needlessly error messages any more.
HTML tag placeholder added for add function, add object and save object made more error prone.
Variable tx_isolation replaced by transaction_isolation which was deprecated in MariaDB 11.2 and MySQL 5.7.
Code clean-up.
Some menu items opened a new tab which was not intended. Fixed.
Field naming made consistent.
Error handling and error messages improved.
Copyright year updated from 2023 to 2024.
Message downgraded from ERROR to DEBUG.
Function runRemoteCommand will return an error code again when going_back_to functionality implemented.
Function renderDropDown newline was removed.
Function renderSubmitButton accepts a title now.
Function runRemoteCommand can suppress error logging now.
Command apt-get replaced by apt.
Distribution information refactored.
Function whoami cleaned-up.
PHP 8.0 function calls removed again because of Debian 10/11 support.
User interaction made unique among all 5 different objects.
Redundant DEBUG information removed.
Configuration file was removed again when not needed any more (clean-up).
Command yum replaced by dnf, also apt-get partly.
Command which replaced by type -p to make it Rocky 9 compliant.
Some more lsb_release stuff removed.
myEnv.inc library updated, lsb_release removed.
Dead code message added.

Repository

No changes.

Documentation

Version changed from 1.2.0 to 1.2.1 and TLS conflict on ubuntu 22.04 fixed.
Build automatized.
Documentation for M/S updated.
All text taken over from CMS.
Load balancer documentation added.

Taxonomy upgrade extras: Operations release FromDual Ops Center ops center dbaas focmm

Categories: FromDual all (en), FromDual TechFeed (en), MySQL Tech-Feed (en)

dbstat for MariaDB after one month of productive use

Shinguz - Fri, 2024-04-26 14:13

Table of contents

Review
One month later
Size of the tables
Process list
Global variables
Metadata Lock and InnoDB Transaction Lock
Global status

Review

After we introduced dbstat for MariaDB (and MySQL) a good 5 weeks ago, we naturally also rolled it out on our systems to test the behaviour in daily use (eat your own dog food).

This went quite well until we came up with the idea of activating dbstat on the passive dbstat node on our MariaDB active/passive master/master replication cluster (a similar situation would also occur with a Galera cluster). We realised that the design of dbstat still had potential. After this problem was fixed (v0.0.2 and v0.0.3) and the problem of how to activate events on master AND slave was solved (MDEV-33782: Event is always disabled on slave), everything seemed fine at first glance. Unfortunately, we did not realise that the data also had to be adjusted. As a result, our replication came to a complete stop over the Easter holidays, which then led to another problem when catching up (MDEV-33923: MariaDB parallel replication causes Foreign Key errors).

After this minor incident was also resolved, dbstat has been running flawlessly on our MariaDB master/master replication cluster ever since... The product dbstat is open source (GPLv2) and can be downloaded from GitHub.

One month later

Databases should NOT grow over time but only over the number of {customers, products, etc.} once the desired equilibrium (steady state) is reached. In our dbstat installation, we have set this equilibrium state to 30 days. So it is now time that the size of dbstat stabilises and the database stops growing...

It would also be interesting to understand what practical use dbstat has. That is why we have now set to work and are trying to analyse the results of dbstat.

Here is an overview of the 11 current running database events:

Firstly, the growth of dbstat itself is interesting. But of course this evaluation can also be carried out for any other database, table or catalogue (coming in MariaDB 11.7?):

SQL> SET SESSION sql_mode='STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,only_full_group_by'; SQL> SET @machine_name = @@hostname; SQL> SELECT `table_schema`, SUBSTR(`ts`, 1, 10) AS date , ROUND(SUM(`data_length`)/1024/1024, 1) AS data_mb , ROUND(SUM(`index_length`)/1024/1024, 1) AS index_mb , ROUND(SUM(`data_free`)/1024/1024, 1) AS free_mb , ROUND((SUM(`data_length`) + SUM(`index_length`) + SUM(`data_free`))/1024/1024, 1) AS total_mb , ROUND(SUM(`table_rows`)/1000/1000, 1) AS rows_m FROM `table_size` WHERE `machine_name` = @machine_name AND `table_catalog` = 'def' AND `table_schema` = 'dbstat' GROUP BY `table_catalog`, `table_schema`, `date` ORDER BY `table_catalog`, `table_schema`, `date` ASC ; +--------------+------------+---------+----------+---------+----------+--------+ | table_schema | date | data_mb | index_mb | free_mb | total_mb | rows_m | +--------------+------------+---------+----------+---------+----------+--------+ | dbstat | 2024-03-26 | 762.8 | 1128.6 | 18.0 | 1909.4 | 10.9 | | dbstat | 2024-03-27 | 835.8 | 1241.6 | 17.0 | 2094.4 | 11.1 | | dbstat | 2024-03-28 | 837.8 | 1241.6 | 14.0 | 2093.4 | 11.8 | | dbstat | 2024-03-29 | 960.7 | 1443.6 | 18.0 | 2422.4 | 14.2 | | dbstat | 2024-03-30 | 960.7 | 1443.6 | 17.0 | 2421.4 | 15.0 | | dbstat | 2024-03-31 | 1057.7 | 1604.6 | 20.0 | 2682.4 | 16.9 | | dbstat | 2024-04-01 | 1057.7 | 1602.6 | 21.0 | 2681.4 | 17.6 | | dbstat | 2024-04-02 | 1172.7 | 1797.6 | 22.0 | 2992.3 | 17.8 | | dbstat | 2024-04-03 | 1442.8 | 2333.7 | 12.0 | 3788.5 | 22.8 | | dbstat | 2024-04-04 | 1649.8 | 2723.7 | 13.0 | 4386.5 | 24.4 | | dbstat | 2024-04-05 | 1649.8 | 2722.7 | 14.0 | 4386.5 | 26.0 | | dbstat | 2024-04-06 | 1821.8 | 3034.8 | 13.0 | 4869.6 | 24.6 | | dbstat | 2024-04-07 | 1821.8 | 3034.8 | 14.0 | 4870.6 | 26.2 | | dbstat | 2024-04-08 | 1989.9 | 3344.8 | 12.0 | 5346.6 | 29.9 | | dbstat | 2024-04-09 | 1990.9 | 3343.8 | 14.0 | 5348.6 | 31.5 | | dbstat | 2024-04-10 | 2193.9 | 3712.8 | 13.0 | 5919.7 | 31.6 | | dbstat | 2024-04-11 | 2193.9 | 3712.8 | 15.0 | 5921.7 | 31.1 | | dbstat | 2024-04-12 | 2405.8 | 4119.1 | 12.0 | 6537.0 | 34.9 | | dbstat | 2024-04-13 | 2405.8 | 4119.1 | 14.0 | 6538.9 | 35.7 | | dbstat | 2024-04-14 | 2480.8 | 4278.9 | 15.0 | 6774.8 | 36.2 | | dbstat | 2024-04-15 | 2560.8 | 4443.7 | 12.0 | 7016.5 | 37.5 | | dbstat | 2024-04-16 | 2560.8 | 4443.7 | 12.0 | 7016.5 | 38.2 | | dbstat | 2024-04-17 | 2640.8 | 4610.6 | 18.0 | 7269.4 | 38.5 | | dbstat | 2024-04-18 | 2640.9 | 4611.6 | 14.0 | 7266.5 | 39.7 | | dbstat | 2024-04-19 | 2743.9 | 4826.5 | 14.0 | 7584.3 | 36.9 | | dbstat | 2024-04-20 | 2826.9 | 4995.5 | 14.0 | 7836.4 | 38.3 | | dbstat | 2024-04-21 | 2830.9 | 4997.4 | 18.0 | 7846.3 | 39.2 | | dbstat | 2024-04-22 | 2919.9 | 5177.4 | 14.0 | 8111.3 | 43.2 | | dbstat | 2024-04-23 | 2923.0 | 5177.3 | 16.0 | 8116.3 | 44.1 | | dbstat | 2024-04-24 | 3020.0 | 5376.3 | 16.0 | 8412.3 | 41.0 | | dbstat | 2024-04-25 | 3024.0 | 5377.3 | 17.0 | 8418.3 | 40.9 | +--------------+------------+---------+----------+---------+----------+--------+

If you take the disc space in the O/S for comparison:

# du -shc *.ibd 8.6G global_status.ibd 308K global_variables.ibd 692K metadata_lock.ibd 97M processlist.ibd 18M table_size.ibd 212K trx_and_lck.ibd 8.7G total

you can see that the values from the database are approximately correct (5% error)...

Important: The database dbstat reaches a size of approx. 9 Gbyte after approx. one month on a not particularly large database system.

You can also see that the size of the database is only just stabilising:

If you want to know more precisely which tables are responsible for which part of the data volume, you can also zoom in or drill down into the data:

SQL> SELECT `table_name`, SUBSTR(`ts`, 1, 10) AS date , ROUND(`data_length`/1024/1024, 1) AS data_mb , ROUND(`index_length`/1024/1024, 1) AS index_mb , ROUND(`data_free`/1024/1024, 1) AS free_mb , ROUND((`data_length` + `index_length` + `data_free`)/1024/1024, 1) AS total_mb , ROUND((`data_length` + `index_length` + `data_free`)/1024/1024/8418.26*100, 1) AS pct , ROUND(`table_rows`/1000/1000, 1) AS rows_m FROM `table_size` WHERE `machine_name` = @machine_name AND `table_catalog` = 'def' AND `table_schema` = 'dbstat' AND SUBSTR(`ts`, 1, 10) = CURRENT_DATE() ORDER BY rows_m DESC ; +------------------+------------+---------+----------+---------+----------+------+--------+ | table_name | date | data_mb | index_mb | free_mb | total_mb | pct | rows_m | +------------------+------------+---------+----------+---------+----------+------+--------+ | global_status | 2024-04-25 | 2949.9 | 5356.9 | 5.0 | 8311.8 | 98.7 | 40.4 | | processlist | 2024-04-25 | 68.2 | 17.1 | 7.0 | 92.2 | 1.1 | 0.4 | | global_variables | 2024-04-25 | 0.1 | 0.1 | 0.0 | 0.2 | 0.0 | 0.0 | | metadata_lock | 2024-04-25 | 0.4 | 0.2 | 0.0 | 0.6 | 0.0 | 0.0 | | table_size | 2024-04-25 | 5.4 | 3.1 | 5.0 | 13.5 | 0.2 | 0.0 | | trx_and_lck | 2024-04-25 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +------------------+------------+---------+----------+---------+----------+------+--------+

Note: Please excuse the non-use of the window function!

The only real driver for the data volume of this database is the global_status table. This is to be expected (see: see quantity structure of dbstat).

SQL> SELECT SUBSTR(ts, 1, 10) AS date, table_rows/1000/1000 AS k_rows , ROUND(data_length/1024/1024, 1) AS data_mb, ROUND(index_length/1024/1024, 1) AS index_mb, ROUND(data_free/1024/1024, 1) AS free_mb , ROUND((data_length + index_length + data_free)/1024/1024, 1) AS total_mb FROM table_size WHERE `machine_name` = @machine_name AND `table_catalog` = 'def' AND `table_schema` = 'dbstat' AND table_name = 'global_status' AND ts > DATE_SUB(CURRENT_DATE, INTERVAL 10 DAY) ; +------------+-------------+---------+----------+---------+----------+ | date | k_rows | data_mb | index_mb | free_mb | total_mb | +------------+-------------+---------+----------+---------+----------+ | 2024-04-15 | 37.13876300 | 2512.9 | 4433.0 | 4.0 | 6949.9 | | 2024-04-16 | 37.94217200 | 2512.9 | 4433.0 | 4.0 | 6949.9 | + 0M | 2024-04-17 | 38.19867500 | 2592.9 | 4600.0 | 7.0 | 7199.9 | + 250M | 2024-04-18 | 39.39108500 | 2592.9 | 4600.0 | 5.0 | 7197.9 | - 2M | 2024-04-19 | 36.52539600 | 2691.9 | 4813.0 | 5.0 | 7509.8 | + 312M | 2024-04-20 | 37.99073500 | 2770.9 | 4980.9 | 6.0 | 7757.8 | + 248M | 2024-04-21 | 38.79420200 | 2770.9 | 4980.9 | 7.0 | 7758.8 | + 1M | 2024-04-22 | 42.82606200 | 2855.9 | 5158.9 | 6.0 | 8020.8 | + 263M | 2024-04-23 | 43.62953000 | 2855.9 | 5158.9 | 7.0 | 8021.8 | + 1M | 2024-04-24 | 40.54342200 | 2949.9 | 5356.9 | 7.0 | 8313.8 | + 292M | 2024-04-25 | 40.43067700 | 2949.9 | 5356.9 | 5.0 | 8311.8 | - 2M +------------+-------------+---------+----------+---------+----------+

Note: Sorry, I should really familiarise myself with the window functions...

If we analyse the data a bit more closely, we see that the number of rows has slowly stabilised over the last 4 days (note: table_rows is calculated (from the number of blocks and the average row length?) and is not an exact value), but the "amount of data" has continued to increase until yesterday, which is probably due to the fragmentation of the tables and indexes...

The primary key of the global_status table was chosen to optimise the localisation of the data:

PRIMARY KEY (`machine_name`,`variable_name`,`ts`),

The situation should calm down in the next few days. In 2 to 4 weeks we will have to check the situation again.

Summary: I would say that this feature fulfils the requirements and helps to understand the data growth.

List of processes

Since we do not have any serious load issues in our databases, this feature is not that interesting in our case. For example, we can see what a (persistent) connection has done:

SQL> SELECT connection_id, ts, command, time, state, SUBSTR(REGEXP_REPLACE(REPLACE(query, "\n", ' '), '\ +', ' '), 1, 64) FROM processlist WHERE machine_name = @machine_name AND command != 'Sleep' AND connection_id = @connection_id AND state NOT IN ( 'Waiting for next activation' , 'Master has sent all binlog to slave; waiting for more updates' , 'Waiting for master to send event' , 'Slave has read all relay log; waiting for more updates' ) ORDER BY ts ASC ; +---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+ | connection_id | ts | command | time | state | SUBSTR(REGEXP_REPLACE(REPLACE(query, "\n", ' '), '\ +', ' '), 1, 64) | +---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+ | 18 | 2024-04-17 12:30:28 | Query | 0.029 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-17 14:58:28 | Query | 0.009 | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-18 06:24:28 | Query | 0.003 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-18 11:34:28 | Query | 0.030 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-18 16:39:28 | Query | 0.006 | Sending data | select itemid,functionid,name,parameter,triggerid from functions | | 18 | 2024-04-18 19:12:28 | Query | 0.014 | Sending data | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-18 21:49:28 | Query | 0.004 | Writing to net | select i.itemid,i.hostid,i.templateid from items i inner join ho | | 18 | 2024-04-19 00:21:28 | Query | 0.032 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-19 02:59:28 | Query | 0.017 | Writing to net | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-19 05:39:28 | Query | 0.052 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-19 08:19:28 | Query | 0.000 | Statistics | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-19 13:26:28 | Query | 0.075 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-19 15:57:28 | Query | 0.027 | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-19 18:33:28 | Query | 0.010 | Sending data | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-19 21:10:28 | Query | 0.008 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-19 23:50:28 | Query | 0.067 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 02:28:28 | Query | 0.008 | Sending data | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-20 05:08:28 | Query | 0.052 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 07:44:28 | Query | 0.123 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 10:21:28 | Query | 0.144 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 12:55:28 | Query | 0.004 | Sending data | select i.itemid,i.hostid,i.templateid from items i where i.flags | | 18 | 2024-04-20 15:35:28 | Query | 0.092 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 18:12:28 | Query | 0.041 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 20:47:28 | Query | 0.113 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 23:25:28 | Query | 0.101 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 02:03:28 | Query | 0.120 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 04:42:28 | Query | 0.099 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 07:18:28 | Query | 0.015 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 12:32:28 | Query | 0.018 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 15:06:28 | Query | 0.091 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 20:16:28 | Query | 0.012 | Sending data | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-22 06:44:28 | Query | 0.161 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 09:21:28 | Query | 0.000 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 11:54:28 | Query | 0.020 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 14:23:28 | Query | 0.067 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 16:59:28 | Query | 0.128 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 22:05:28 | Query | 0.078 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 00:38:28 | Query | 0.084 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 03:15:28 | Query | 0.098 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 05:52:28 | Query | 0.000 | starting | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 08:27:28 | Query | 0.011 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-23 10:58:28 | Query | 0.000 | Sending data | select i.itemid,i.hostid,i.templateid from items i inner join ho | | 18 | 2024-04-23 13:31:28 | Query | 0.110 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 16:01:28 | Query | 0.023 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 18:35:28 | Query | 0.095 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 21:10:28 | Query | 0.017 | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-23 23:44:28 | Query | 0.014 | Sending data | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-24 02:21:28 | Query | 0.024 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-24 07:33:28 | Query | 0.046 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | +---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+

It is also important that we only see the entries in this report when the thread has done SOMETHING (we have hidden state Sleep). It is also interesting that we do not see this (persistent) connection before 17 April, but at the moment I have NO explanation for this from an operational point of view (restart etc.). Probably the application (Zabbix) has to explain this.

Global variables

The information in the global_variables table is also interesting:

SQL> SELECT variable_name, ts, variable_value FROM global_variables WHERE machine_name = @machine_name AND variable_name IN ( SELECT variable_name FROM global_variables WHERE machine_name = @machine_name GROUP BY variable_name HAVING COUNT(*) > 1 ) ORDER BY ts, variable_name ; +---------------------------+---------------------+----------------+ | variable_name | ts | variable_value | +---------------------------+---------------------+----------------+ | auto_increment_increment | 2024-03-09 22:10:42 | 1 | | auto_increment_offset | 2024-03-09 22:10:42 | 1 | | read_only | 2024-03-09 22:10:42 | OFF | | slave_parallel_max_queued | 2024-03-09 22:10:42 | 131072 | | slave_parallel_threads | 2024-03-09 22:10:42 | 0 | | slave_parallel_workers | 2024-03-09 22:10:42 | 0 | | slave_skip_errors | 2024-03-09 22:10:42 | OFF | | system_time_zone | 2024-03-09 22:10:42 | CET | | read_only | 2024-03-27 09:42:50 | ON | | slave_skip_errors | 2024-03-27 12:33:13 | 1032 | | slave_skip_errors | 2024-03-27 12:35:13 | OFF | | slave_skip_errors | 2024-03-27 12:42:13 | 1032 | | slave_skip_errors | 2024-03-27 12:50:13 | OFF | | slave_parallel_threads | 2024-04-02 10:17:32 | 8 | | slave_parallel_workers | 2024-04-02 10:17:32 | 8 | | slave_parallel_max_queued | 2024-04-02 10:22:32 | 1048576 | | slave_parallel_max_queued | 2024-04-02 10:23:32 | 4194304 | | slave_parallel_max_queued | 2024-04-02 10:25:32 | 16777216 | | slave_parallel_threads | 2024-04-02 10:25:32 | 16 | | slave_parallel_workers | 2024-04-02 10:25:32 | 16 | | slave_parallel_threads | 2024-04-02 10:28:32 | 32 | | slave_parallel_workers | 2024-04-02 10:28:32 | 32 | | auto_increment_increment | 2024-04-02 10:39:32 | 2 | | auto_increment_offset | 2024-04-02 10:39:32 | 2 | | slave_parallel_max_queued | 2024-04-02 10:57:32 | 131072 | | slave_parallel_threads | 2024-04-02 10:57:32 | 0 | | slave_parallel_workers | 2024-04-02 10:57:32 | 0 | | system_time_zone | 2024-04-02 10:57:32 | CEST | | slave_parallel_max_queued | 2024-04-16 14:06:32 | 16777216 | | slave_parallel_threads | 2024-04-16 14:06:32 | 8 | | slave_parallel_workers | 2024-04-16 14:06:32 | 8 | | slave_parallel_max_queued | 2024-04-16 14:26:32 | 131072 | | slave_parallel_threads | 2024-04-16 14:26:32 | 0 | | slave_parallel_workers | 2024-04-16 14:26:32 | 0 | | slave_parallel_max_queued | 2024-04-17 09:03:32 | 16777216 | | slave_parallel_threads | 2024-04-17 09:03:32 | 16 | | slave_parallel_workers | 2024-04-17 09:03:32 | 16 | | slave_parallel_max_queued | 2024-04-24 08:26:32 | 131072 | | slave_parallel_threads | 2024-04-24 08:26:32 | 0 | | slave_parallel_workers | 2024-04-24 08:26:32 | 0 | | read_only | 2024-04-24 08:42:32 | OFF | +---------------------------+---------------------+----------------+

Here you can see very precisely when and what was done to the database:

On 9 March, dbstat was installed for the first time.
Then on 27 March (before Easter) there seem to have been problems with the replication (here the new version of dbstat was installed, which allows simultaneous collection on master and slave. This led to replication errors, which were partially rectified).
On 2 April (after Easter) we then tried to catch up with parallel replication. You can also see that AUTO_INCREMENT_OFFSET and AUTO_INCREMENT_INCREMENT have been changed. Here we have corrected an error in the database configuration...
You can also see that the time zone has changed from CET to CEST (summer time!) Why only on 2 April is not entirely clear to me. (Maybe because it came via replication?)
Then on 16 and 17 April we tried to reproduce a "bug" in the parallel replication. Apparently we did not reset the value. Because only after the restart on 24 April (usual fortnightly maintenance window) was the value reset again.
On 24 April, you can also see that the database has now assumed the role of the active master (read_only = off). A gracefull switchover has therefore taken place...

Conclusion: A very useful feature to see when something has been changed. Although I have followed all these operations closely, I am still amazed at the informative value of this feature. I would like to see it installed in all databases...

Metadata Lock and InnoDB Transaction Lock

Unfortunately, due to the low traffic on our databases, we do not see too much exciting stuff here.

Here are the metadata locks that we have "caught" on the master in the last 24 hours:

+---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+ | connection_id | ts | user | host | table_schema | table_name | state | SUBSTR(REGEXP_REPLACE(REPLACE(query, "\n", ' '), '\ +', ' '), 1, 64) | +---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+ | 18 | 2024-04-23 14:16:47 | zabbix | localhost:51252 | zabbix | triggers | Writing to net | select triggerid,description,expression,error,priority,type,valu | | 1325025 | 2024-04-23 16:01:47 | zabbix | localhost:50150 | | | init for update | delete from history_text where itemid=85477 and clock<1678167661 | | 1325025 | 2024-04-23 16:01:47 | zabbix | localhost:50150 | zabbix | history_text | init for update | delete from history_text where itemid=85477 and clock<1678167661 | | 1365229 | 2024-04-24 02:13:47 | root | localhost:38096 | dbstat | global_status | Writing to net | SELECT /*!40001 SQL_NO_CACHE */ `machine_name`, `variable_name`, | | 18 | 2024-04-24 03:10:47 | zabbix | localhost:51252 | zabbix | item_tag | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 1368524 | 2024-04-24 04:41:47 | zabbix | localhost:38112 | | | | NULL | | 1368524 | 2024-04-24 04:41:47 | zabbix | localhost:38112 | zabbix | history_uint | | NULL | | 18 | 2024-04-24 05:46:47 | zabbix | localhost:51252 | zabbix | item_tag | Sending data | select itemtagid,itemid,tag,value from item_tag | +---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+

We have not found any InnoDB locks in the last 24 hours for 7 days in a row.

It would be interesting to see a system where more is happening...

Global status

If a normal database monitoring such as the FromDual Performance Monitor for MariaDB and MySQL (fpmmm) with Zabbix is used, this feature is not absolutely necessary. However, most of our customers do not have any useful monitoring in use. Therefore, this feature would be very useful for post-mortem analyses...

For example InnoDB Row Lock Waits, minute-granular over the last 30 days (analogue to sar from sysstat):

Here you can see that the database was restarted on 10 April between 08:37 and 08:41. You could also find this out in another way, but unfortunately this is often not possible for various reasons (error log rotated away, etc.).

The trend break around 2 April is also interesting. At this time we were experimenting with parallel replication. It should not have been a failover (see GLOBAL VARIABLES, above).

Although parallel replication was later deactivated again, there were more locks. A similar situation around the 16th/17th of April, here too we played around with parallel replication, which seems to have had an effect on the locking behaviour.

Even with this feature, there are many ways to analyse the database. Unfortunately, our database is relatively boring: Mainly monotonous traffic (which is plentiful due to the monitoring) and very little exceptional traffic.

Remark: This text has been translated with the assistance of DeepL.

Taxonomy upgrade extras: performance monitoring performance monitoring metadata lock locking performance_schema

Categories: FromDual all (en), FromDual TechFeed (en), MySQL Tech-Feed (en)

dbstat for MariaDB after one month of productive use

Shinguz - Fri, 2024-04-26 14:13

Table of contents

Review
One month later
Size of the tables
Process list
Global variables
Metadata Lock and InnoDB Transaction Lock
Global status

Review

After we introduced dbstat for MariaDB (and MySQL) a good 5 weeks ago, we naturally also rolled it out on our systems to test the behaviour in daily use (eat your own dog food).

This went quite well until we came up with the idea of activating dbstat on the passive dbstat node on our MariaDB active/passive master/master replication cluster (a similar situation would also occur with a Galera cluster). We realised that the design of dbstat still had potential. After this problem was fixed (v0.0.2 and v0.0.3) and the problem of how to activate events on master AND slave was solved (MDEV-33782: Event is always disabled on slave), everything seemed fine at first glance. Unfortunately, we did not realise that the data also had to be adjusted. As a result, our replication came to a complete stop over the Easter holidays, which then led to another problem when catching up (MDEV-33923: MariaDB parallel replication causes Foreign Key errors).

After this minor incident was also resolved, dbstat has been running flawlessly on our MariaDB master/master replication cluster ever since... The product dbstat is open source (GPLv2) and can be downloaded from GitHub.

One month later

Databases should NOT grow over time but only over the number of {customers, products, etc.} once the desired equilibrium (steady state) is reached. In our dbstat installation, we have set this equilibrium state to 30 days. So it is now time that the size of dbstat stabilises and the database stops growing...

It would also be interesting to understand what practical use dbstat has. That is why we have now set to work and are trying to analyse the results of dbstat.

Here is an overview of the 11 current running database events:

Firstly, the growth of dbstat itself is interesting. But of course this evaluation can also be carried out for any other database, table or catalogue (coming in MariaDB 11.7?):

SQL> SET SESSION sql_mode='STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,only_full_group_by'; SQL> SET @machine_name = @@hostname; SQL> SELECT `table_schema`, SUBSTR(`ts`, 1, 10) AS date , ROUND(SUM(`data_length`)/1024/1024, 1) AS data_mb , ROUND(SUM(`index_length`)/1024/1024, 1) AS index_mb , ROUND(SUM(`data_free`)/1024/1024, 1) AS free_mb , ROUND((SUM(`data_length`) + SUM(`index_length`) + SUM(`data_free`))/1024/1024, 1) AS total_mb , ROUND(SUM(`table_rows`)/1000/1000, 1) AS rows_m FROM `table_size` WHERE `machine_name` = @machine_name AND `table_catalog` = 'def' AND `table_schema` = 'dbstat' GROUP BY `table_catalog`, `table_schema`, `date` ORDER BY `table_catalog`, `table_schema`, `date` ASC ; +--------------+------------+---------+----------+---------+----------+--------+ | table_schema | date | data_mb | index_mb | free_mb | total_mb | rows_m | +--------------+------------+---------+----------+---------+----------+--------+ | dbstat | 2024-03-26 | 762.8 | 1128.6 | 18.0 | 1909.4 | 10.9 | | dbstat | 2024-03-27 | 835.8 | 1241.6 | 17.0 | 2094.4 | 11.1 | | dbstat | 2024-03-28 | 837.8 | 1241.6 | 14.0 | 2093.4 | 11.8 | | dbstat | 2024-03-29 | 960.7 | 1443.6 | 18.0 | 2422.4 | 14.2 | | dbstat | 2024-03-30 | 960.7 | 1443.6 | 17.0 | 2421.4 | 15.0 | | dbstat | 2024-03-31 | 1057.7 | 1604.6 | 20.0 | 2682.4 | 16.9 | | dbstat | 2024-04-01 | 1057.7 | 1602.6 | 21.0 | 2681.4 | 17.6 | | dbstat | 2024-04-02 | 1172.7 | 1797.6 | 22.0 | 2992.3 | 17.8 | | dbstat | 2024-04-03 | 1442.8 | 2333.7 | 12.0 | 3788.5 | 22.8 | | dbstat | 2024-04-04 | 1649.8 | 2723.7 | 13.0 | 4386.5 | 24.4 | | dbstat | 2024-04-05 | 1649.8 | 2722.7 | 14.0 | 4386.5 | 26.0 | | dbstat | 2024-04-06 | 1821.8 | 3034.8 | 13.0 | 4869.6 | 24.6 | | dbstat | 2024-04-07 | 1821.8 | 3034.8 | 14.0 | 4870.6 | 26.2 | | dbstat | 2024-04-08 | 1989.9 | 3344.8 | 12.0 | 5346.6 | 29.9 | | dbstat | 2024-04-09 | 1990.9 | 3343.8 | 14.0 | 5348.6 | 31.5 | | dbstat | 2024-04-10 | 2193.9 | 3712.8 | 13.0 | 5919.7 | 31.6 | | dbstat | 2024-04-11 | 2193.9 | 3712.8 | 15.0 | 5921.7 | 31.1 | | dbstat | 2024-04-12 | 2405.8 | 4119.1 | 12.0 | 6537.0 | 34.9 | | dbstat | 2024-04-13 | 2405.8 | 4119.1 | 14.0 | 6538.9 | 35.7 | | dbstat | 2024-04-14 | 2480.8 | 4278.9 | 15.0 | 6774.8 | 36.2 | | dbstat | 2024-04-15 | 2560.8 | 4443.7 | 12.0 | 7016.5 | 37.5 | | dbstat | 2024-04-16 | 2560.8 | 4443.7 | 12.0 | 7016.5 | 38.2 | | dbstat | 2024-04-17 | 2640.8 | 4610.6 | 18.0 | 7269.4 | 38.5 | | dbstat | 2024-04-18 | 2640.9 | 4611.6 | 14.0 | 7266.5 | 39.7 | | dbstat | 2024-04-19 | 2743.9 | 4826.5 | 14.0 | 7584.3 | 36.9 | | dbstat | 2024-04-20 | 2826.9 | 4995.5 | 14.0 | 7836.4 | 38.3 | | dbstat | 2024-04-21 | 2830.9 | 4997.4 | 18.0 | 7846.3 | 39.2 | | dbstat | 2024-04-22 | 2919.9 | 5177.4 | 14.0 | 8111.3 | 43.2 | | dbstat | 2024-04-23 | 2923.0 | 5177.3 | 16.0 | 8116.3 | 44.1 | | dbstat | 2024-04-24 | 3020.0 | 5376.3 | 16.0 | 8412.3 | 41.0 | | dbstat | 2024-04-25 | 3024.0 | 5377.3 | 17.0 | 8418.3 | 40.9 | +--------------+------------+---------+----------+---------+----------+--------+

If you take the disc space in the O/S for comparison:

# du -shc *.ibd 8.6G global_status.ibd 308K global_variables.ibd 692K metadata_lock.ibd 97M processlist.ibd 18M table_size.ibd 212K trx_and_lck.ibd 8.7G total

you can see that the values from the database are approximately correct (5% error)...

Important: The database dbstat reaches a size of approx. 9 Gbyte after approx. one month on a not particularly large database system.

You can also see that the size of the database is only just stabilising:

If you want to know more precisely which tables are responsible for which part of the data volume, you can also zoom in or drill down into the data:

SQL> SELECT `table_name`, SUBSTR(`ts`, 1, 10) AS date , ROUND(`data_length`/1024/1024, 1) AS data_mb , ROUND(`index_length`/1024/1024, 1) AS index_mb , ROUND(`data_free`/1024/1024, 1) AS free_mb , ROUND((`data_length` + `index_length` + `data_free`)/1024/1024, 1) AS total_mb , ROUND((`data_length` + `index_length` + `data_free`)/1024/1024/8418.26*100, 1) AS pct , ROUND(`table_rows`/1000/1000, 1) AS rows_m FROM `table_size` WHERE `machine_name` = @machine_name AND `table_catalog` = 'def' AND `table_schema` = 'dbstat' AND SUBSTR(`ts`, 1, 10) = CURRENT_DATE() ORDER BY rows_m DESC ; +------------------+------------+---------+----------+---------+----------+------+--------+ | table_name | date | data_mb | index_mb | free_mb | total_mb | pct | rows_m | +------------------+------------+---------+----------+---------+----------+------+--------+ | global_status | 2024-04-25 | 2949.9 | 5356.9 | 5.0 | 8311.8 | 98.7 | 40.4 | | processlist | 2024-04-25 | 68.2 | 17.1 | 7.0 | 92.2 | 1.1 | 0.4 | | global_variables | 2024-04-25 | 0.1 | 0.1 | 0.0 | 0.2 | 0.0 | 0.0 | | metadata_lock | 2024-04-25 | 0.4 | 0.2 | 0.0 | 0.6 | 0.0 | 0.0 | | table_size | 2024-04-25 | 5.4 | 3.1 | 5.0 | 13.5 | 0.2 | 0.0 | | trx_and_lck | 2024-04-25 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +------------------+------------+---------+----------+---------+----------+------+--------+

Note: Please excuse the non-use of the window function!

The only real driver for the data volume of this database is the global_status table. This is to be expected (see: see quantity structure of dbstat).

SQL> SELECT SUBSTR(ts, 1, 10) AS date, table_rows/1000/1000 AS k_rows , ROUND(data_length/1024/1024, 1) AS data_mb, ROUND(index_length/1024/1024, 1) AS index_mb, ROUND(data_free/1024/1024, 1) AS free_mb , ROUND((data_length + index_length + data_free)/1024/1024, 1) AS total_mb FROM table_size WHERE `machine_name` = @machine_name AND `table_catalog` = 'def' AND `table_schema` = 'dbstat' AND table_name = 'global_status' AND ts > DATE_SUB(CURRENT_DATE, INTERVAL 10 DAY) ; +------------+-------------+---------+----------+---------+----------+ | date | k_rows | data_mb | index_mb | free_mb | total_mb | +------------+-------------+---------+----------+---------+----------+ | 2024-04-15 | 37.13876300 | 2512.9 | 4433.0 | 4.0 | 6949.9 | | 2024-04-16 | 37.94217200 | 2512.9 | 4433.0 | 4.0 | 6949.9 | + 0M | 2024-04-17 | 38.19867500 | 2592.9 | 4600.0 | 7.0 | 7199.9 | + 250M | 2024-04-18 | 39.39108500 | 2592.9 | 4600.0 | 5.0 | 7197.9 | - 2M | 2024-04-19 | 36.52539600 | 2691.9 | 4813.0 | 5.0 | 7509.8 | + 312M | 2024-04-20 | 37.99073500 | 2770.9 | 4980.9 | 6.0 | 7757.8 | + 248M | 2024-04-21 | 38.79420200 | 2770.9 | 4980.9 | 7.0 | 7758.8 | + 1M | 2024-04-22 | 42.82606200 | 2855.9 | 5158.9 | 6.0 | 8020.8 | + 263M | 2024-04-23 | 43.62953000 | 2855.9 | 5158.9 | 7.0 | 8021.8 | + 1M | 2024-04-24 | 40.54342200 | 2949.9 | 5356.9 | 7.0 | 8313.8 | + 292M | 2024-04-25 | 40.43067700 | 2949.9 | 5356.9 | 5.0 | 8311.8 | - 2M +------------+-------------+---------+----------+---------+----------+

Note: Sorry, I should really familiarise myself with the window functions...

If we analyse the data a bit more closely, we see that the number of rows has slowly stabilised over the last 4 days (note: table_rows is calculated (from the number of blocks and the average row length?) and is not an exact value), but the "amount of data" has continued to increase until yesterday, which is probably due to the fragmentation of the tables and indexes...

The primary key of the global_status table was chosen to optimise the localisation of the data:

PRIMARY KEY (`machine_name`,`variable_name`,`ts`),

The situation should calm down in the next few days. In 2 to 4 weeks we will have to check the situation again.

Summary: I would say that this feature fulfils the requirements and helps to understand the data growth.

List of processes

Since we do not have any serious load issues in our databases, this feature is not that interesting in our case. For example, we can see what a (persistent) connection has done:

SQL> SELECT connection_id, ts, command, time, state, SUBSTR(REGEXP_REPLACE(REPLACE(query, "\n", ' '), '\ +', ' '), 1, 64) FROM processlist WHERE machine_name = @machine_name AND command != 'Sleep' AND connection_id = @connection_id AND state NOT IN ( 'Waiting for next activation' , 'Master has sent all binlog to slave; waiting for more updates' , 'Waiting for master to send event' , 'Slave has read all relay log; waiting for more updates' ) ORDER BY ts ASC ; +---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+ | connection_id | ts | command | time | state | SUBSTR(REGEXP_REPLACE(REPLACE(query, "\n", ' '), '\ +', ' '), 1, 64) | +---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+ | 18 | 2024-04-17 12:30:28 | Query | 0.029 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-17 14:58:28 | Query | 0.009 | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-18 06:24:28 | Query | 0.003 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-18 11:34:28 | Query | 0.030 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-18 16:39:28 | Query | 0.006 | Sending data | select itemid,functionid,name,parameter,triggerid from functions | | 18 | 2024-04-18 19:12:28 | Query | 0.014 | Sending data | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-18 21:49:28 | Query | 0.004 | Writing to net | select i.itemid,i.hostid,i.templateid from items i inner join ho | | 18 | 2024-04-19 00:21:28 | Query | 0.032 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-19 02:59:28 | Query | 0.017 | Writing to net | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-19 05:39:28 | Query | 0.052 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-19 08:19:28 | Query | 0.000 | Statistics | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-19 13:26:28 | Query | 0.075 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-19 15:57:28 | Query | 0.027 | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-19 18:33:28 | Query | 0.010 | Sending data | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-19 21:10:28 | Query | 0.008 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-19 23:50:28 | Query | 0.067 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 02:28:28 | Query | 0.008 | Sending data | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-20 05:08:28 | Query | 0.052 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 07:44:28 | Query | 0.123 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 10:21:28 | Query | 0.144 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 12:55:28 | Query | 0.004 | Sending data | select i.itemid,i.hostid,i.templateid from items i where i.flags | | 18 | 2024-04-20 15:35:28 | Query | 0.092 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 18:12:28 | Query | 0.041 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 20:47:28 | Query | 0.113 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 23:25:28 | Query | 0.101 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 02:03:28 | Query | 0.120 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 04:42:28 | Query | 0.099 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 07:18:28 | Query | 0.015 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 12:32:28 | Query | 0.018 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 15:06:28 | Query | 0.091 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 20:16:28 | Query | 0.012 | Sending data | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-22 06:44:28 | Query | 0.161 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 09:21:28 | Query | 0.000 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 11:54:28 | Query | 0.020 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 14:23:28 | Query | 0.067 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 16:59:28 | Query | 0.128 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 22:05:28 | Query | 0.078 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 00:38:28 | Query | 0.084 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 03:15:28 | Query | 0.098 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 05:52:28 | Query | 0.000 | starting | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 08:27:28 | Query | 0.011 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-23 10:58:28 | Query | 0.000 | Sending data | select i.itemid,i.hostid,i.templateid from items i inner join ho | | 18 | 2024-04-23 13:31:28 | Query | 0.110 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 16:01:28 | Query | 0.023 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 18:35:28 | Query | 0.095 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 21:10:28 | Query | 0.017 | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-23 23:44:28 | Query | 0.014 | Sending data | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-24 02:21:28 | Query | 0.024 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-24 07:33:28 | Query | 0.046 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | +---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+

It is also important that we only see the entries in this report when the thread has done SOMETHING (we have hidden state Sleep). It is also interesting that we do not see this (persistent) connection before 17 April, but at the moment I have NO explanation for this from an operational point of view (restart etc.). Probably the application (Zabbix) has to explain this.

Global variables

The information in the global_variables table is also interesting:

SQL> SELECT variable_name, ts, variable_value FROM global_variables WHERE machine_name = @machine_name AND variable_name IN ( SELECT variable_name FROM global_variables WHERE machine_name = @machine_name GROUP BY variable_name HAVING COUNT(*) > 1 ) ORDER BY ts, variable_name ; +---------------------------+---------------------+----------------+ | variable_name | ts | variable_value | +---------------------------+---------------------+----------------+ | auto_increment_increment | 2024-03-09 22:10:42 | 1 | | auto_increment_offset | 2024-03-09 22:10:42 | 1 | | read_only | 2024-03-09 22:10:42 | OFF | | slave_parallel_max_queued | 2024-03-09 22:10:42 | 131072 | | slave_parallel_threads | 2024-03-09 22:10:42 | 0 | | slave_parallel_workers | 2024-03-09 22:10:42 | 0 | | slave_skip_errors | 2024-03-09 22:10:42 | OFF | | system_time_zone | 2024-03-09 22:10:42 | CET | | read_only | 2024-03-27 09:42:50 | ON | | slave_skip_errors | 2024-03-27 12:33:13 | 1032 | | slave_skip_errors | 2024-03-27 12:35:13 | OFF | | slave_skip_errors | 2024-03-27 12:42:13 | 1032 | | slave_skip_errors | 2024-03-27 12:50:13 | OFF | | slave_parallel_threads | 2024-04-02 10:17:32 | 8 | | slave_parallel_workers | 2024-04-02 10:17:32 | 8 | | slave_parallel_max_queued | 2024-04-02 10:22:32 | 1048576 | | slave_parallel_max_queued | 2024-04-02 10:23:32 | 4194304 | | slave_parallel_max_queued | 2024-04-02 10:25:32 | 16777216 | | slave_parallel_threads | 2024-04-02 10:25:32 | 16 | | slave_parallel_workers | 2024-04-02 10:25:32 | 16 | | slave_parallel_threads | 2024-04-02 10:28:32 | 32 | | slave_parallel_workers | 2024-04-02 10:28:32 | 32 | | auto_increment_increment | 2024-04-02 10:39:32 | 2 | | auto_increment_offset | 2024-04-02 10:39:32 | 2 | | slave_parallel_max_queued | 2024-04-02 10:57:32 | 131072 | | slave_parallel_threads | 2024-04-02 10:57:32 | 0 | | slave_parallel_workers | 2024-04-02 10:57:32 | 0 | | system_time_zone | 2024-04-02 10:57:32 | CEST | | slave_parallel_max_queued | 2024-04-16 14:06:32 | 16777216 | | slave_parallel_threads | 2024-04-16 14:06:32 | 8 | | slave_parallel_workers | 2024-04-16 14:06:32 | 8 | | slave_parallel_max_queued | 2024-04-16 14:26:32 | 131072 | | slave_parallel_threads | 2024-04-16 14:26:32 | 0 | | slave_parallel_workers | 2024-04-16 14:26:32 | 0 | | slave_parallel_max_queued | 2024-04-17 09:03:32 | 16777216 | | slave_parallel_threads | 2024-04-17 09:03:32 | 16 | | slave_parallel_workers | 2024-04-17 09:03:32 | 16 | | slave_parallel_max_queued | 2024-04-24 08:26:32 | 131072 | | slave_parallel_threads | 2024-04-24 08:26:32 | 0 | | slave_parallel_workers | 2024-04-24 08:26:32 | 0 | | read_only | 2024-04-24 08:42:32 | OFF | +---------------------------+---------------------+----------------+

Here you can see very precisely when and what was done to the database:

On 9 March, dbstat was installed for the first time.
Then on 27 March (before Easter) there seem to have been problems with the replication (here the new version of dbstat was installed, which allows simultaneous collection on master and slave. This led to replication errors, which were partially rectified).
On 2 April (after Easter) we then tried to catch up with parallel replication. You can also see that AUTO_INCREMENT_OFFSET and AUTO_INCREMENT_INCREMENT have been changed. Here we have corrected an error in the database configuration...
You can also see that the time zone has changed from CET to CEST (summer time!) Why only on 2 April is not entirely clear to me. (Maybe because it came via replication?)
Then on 16 and 17 April we tried to reproduce a "bug" in the parallel replication. Apparently we did not reset the value. Because only after the restart on 24 April (usual fortnightly maintenance window) was the value reset again.
On 24 April, you can also see that the database has now assumed the role of the active master (read_only = off). A gracefull switchover has therefore taken place...

Conclusion: A very useful feature to see when something has been changed. Although I have followed all these operations closely, I am still amazed at the informative value of this feature. I would like to see it installed in all databases...

Metadata Lock and InnoDB Transaction Lock

Unfortunately, due to the low traffic on our databases, we do not see too much exciting stuff here.

Here are the metadata locks that we have "caught" on the master in the last 24 hours:

+---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+ | connection_id | ts | user | host | table_schema | table_name | state | SUBSTR(REGEXP_REPLACE(REPLACE(query, "\n", ' '), '\ +', ' '), 1, 64) | +---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+ | 18 | 2024-04-23 14:16:47 | zabbix | localhost:51252 | zabbix | triggers | Writing to net | select triggerid,description,expression,error,priority,type,valu | | 1325025 | 2024-04-23 16:01:47 | zabbix | localhost:50150 | | | init for update | delete from history_text where itemid=85477 and clock<1678167661 | | 1325025 | 2024-04-23 16:01:47 | zabbix | localhost:50150 | zabbix | history_text | init for update | delete from history_text where itemid=85477 and clock<1678167661 | | 1365229 | 2024-04-24 02:13:47 | root | localhost:38096 | dbstat | global_status | Writing to net | SELECT /*!40001 SQL_NO_CACHE */ `machine_name`, `variable_name`, | | 18 | 2024-04-24 03:10:47 | zabbix | localhost:51252 | zabbix | item_tag | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 1368524 | 2024-04-24 04:41:47 | zabbix | localhost:38112 | | | | NULL | | 1368524 | 2024-04-24 04:41:47 | zabbix | localhost:38112 | zabbix | history_uint | | NULL | | 18 | 2024-04-24 05:46:47 | zabbix | localhost:51252 | zabbix | item_tag | Sending data | select itemtagid,itemid,tag,value from item_tag | +---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+

We have not found any InnoDB locks in the last 24 hours for 7 days in a row.

It would be interesting to see a system where more is happening...

Global status

If a normal database monitoring such as the FromDual Performance Monitor for MariaDB and MySQL (fpmmm) with Zabbix is used, this feature is not absolutely necessary. However, most of our customers do not have any useful monitoring in use. Therefore, this feature would be very useful for post-mortem analyses...

For example InnoDB Row Lock Waits, minute-granular over the last 30 days (analogue to sar from sysstat):

Here you can see that the database was restarted on 10 April between 08:37 and 08:41. You could also find this out in another way, but unfortunately this is often not possible for various reasons (error log rotated away, etc.).

The trend break around 2 April is also interesting. At this time we were experimenting with parallel replication. It should not have been a failover (see GLOBAL VARIABLES, above).

Although parallel replication was later deactivated again, there were more locks. A similar situation around the 16th/17th of April, here too we played around with parallel replication, which seems to have had an effect on the locking behaviour.

Even with this feature, there are many ways to analyse the database. Unfortunately, our database is relatively boring: Mainly monotonous traffic (which is plentiful due to the monitoring) and very little exceptional traffic.

Remark: This text has been translated with the assistance of DeepL.

Taxonomy upgrade extras: performance monitoring performance monitoring metadata lock locking performance_schema

Categories: FromDual all (en), FromDual TechFeed (en), MySQL Tech-Feed (en)

dbstat für MariaDB nach einem Monat produktiver Nutzung

Oli Sennhauser - Thu, 2024-04-25 12:39

Inhaltsverzeichnis

Rückblick
Einen Monat später
Grösse der Tabellen
Prozessliste
Globale Variablen
Metadata Lock und InnoDB Transaction Lock
Globaler Status

Rückblick

Nachdem wir vor gut 5 Wochen dbstat für MariaDB (und MySQL) vorgestellt haben, haben wir es natürlich auch auf unseren Systemen ausgerollt um das Verhalten im täglichen Einsatz zu testen (eat your own dog food).

Das ging soweit ganz gut, bis wir auf unserem MariaDB aktiv/passiv Master/Master Replikationscluster auf die Idee kamen, dbstat auch auf dem passiven dbstat Node zu aktivieren (eine ähnliche Situation hätte man auch bei einem Galera Cluster). Dabei stellten wir fest, dass das Design von dbstat noch Potential hatte. Nachdem dieses Problem behoben war (v0.0.2 und v0.0.3) und auch das Problem gelöst war, wie man Events auf Master UND Slave aktivieren kann (MDEV-33782: Event is always disabled on slave), schien auf den ersten Blick alles in Ordnung. Leider haben wir bei der Korrektur nicht bedacht, dass auch die Daten hätten angepasst werden müssen. Dies hatte zur Folge, dass unsere Replikation über die Osterfeiertage zum Stillstand kam, was dann beim Aufholen zu einem weiteren Problem führte (MDEV-33923: MariaDB parallel replication causes Foreign Key errors).

Nachdem auch dieser kleine Zwischenfall behoben war lief dbstat auf unserem Master/Master Replikationscluster seitdem einwandfrei... Das Produkt dbstat ist Open Source (GPLv2) und kann von GitHub heruntergeladen werden.

Einen Monat später

Datenbanken sollten NICHT mit der Zeit wachsen sondern nur mit der Anzahl {Kunden, Produkte, etc.}, sobald das gewünschte Gleichgewicht (steady state) erreicht ist. In unserer dbstat-Installation haben wir diesen Wert auf 30 Tage gesetzt. Es wird also langsam an der Zeit, dass sich die Grösse von dbstat stabilisiert und nicht weiter wächst...

Ausserdem wäre es spannend zu verstehen, welchen praktischen Nutzen dbstat hat. Deshalb haben wir uns jetzt an die Arbeit gemacht und versuchen, die Ergebnisse von dbstat auszuwerten.

Hier zunächst noch einmal ein Überblick über die 11 laufenden Events:

Zunächst ist das Wachstum von dbstat selbst interessant. Aber natürlich kann diese Auswertung auch für jede andere Datenbank, Tabelle oder Catalog (kommt in MariaDB 11.7?) durchgeführt werden:

SQL> SET SESSION sql_mode='STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,only_full_group_by'; SQL> SET @machine_name = @@hostname; SQL> SELECT `table_schema`, SUBSTR(`ts`, 1, 10) AS date , ROUND(SUM(`data_length`)/1024/1024, 1) AS data_mb , ROUND(SUM(`index_length`)/1024/1024, 1) AS index_mb , ROUND(SUM(`data_free`)/1024/1024, 1) AS free_mb , ROUND((SUM(`data_length`) + SUM(`index_length`) + SUM(`data_free`))/1024/1024, 1) AS total_mb , ROUND(SUM(`table_rows`)/1000/1000, 1) AS rows_m FROM `table_size` WHERE `machine_name` = @machine_name AND `table_catalog` = 'def' AND `table_schema` = 'dbstat' GROUP BY `table_catalog`, `table_schema`, `date` ORDER BY `table_catalog`, `table_schema`, `date` ASC ; +--------------+------------+---------+----------+---------+----------+--------+ | table_schema | date | data_mb | index_mb | free_mb | total_mb | rows_m | +--------------+------------+---------+----------+---------+----------+--------+ | dbstat | 2024-03-26 | 762.8 | 1128.6 | 18.0 | 1909.4 | 10.9 | | dbstat | 2024-03-27 | 835.8 | 1241.6 | 17.0 | 2094.4 | 11.1 | | dbstat | 2024-03-28 | 837.8 | 1241.6 | 14.0 | 2093.4 | 11.8 | | dbstat | 2024-03-29 | 960.7 | 1443.6 | 18.0 | 2422.4 | 14.2 | | dbstat | 2024-03-30 | 960.7 | 1443.6 | 17.0 | 2421.4 | 15.0 | | dbstat | 2024-03-31 | 1057.7 | 1604.6 | 20.0 | 2682.4 | 16.9 | | dbstat | 2024-04-01 | 1057.7 | 1602.6 | 21.0 | 2681.4 | 17.6 | | dbstat | 2024-04-02 | 1172.7 | 1797.6 | 22.0 | 2992.3 | 17.8 | | dbstat | 2024-04-03 | 1442.8 | 2333.7 | 12.0 | 3788.5 | 22.8 | | dbstat | 2024-04-04 | 1649.8 | 2723.7 | 13.0 | 4386.5 | 24.4 | | dbstat | 2024-04-05 | 1649.8 | 2722.7 | 14.0 | 4386.5 | 26.0 | | dbstat | 2024-04-06 | 1821.8 | 3034.8 | 13.0 | 4869.6 | 24.6 | | dbstat | 2024-04-07 | 1821.8 | 3034.8 | 14.0 | 4870.6 | 26.2 | | dbstat | 2024-04-08 | 1989.9 | 3344.8 | 12.0 | 5346.6 | 29.9 | | dbstat | 2024-04-09 | 1990.9 | 3343.8 | 14.0 | 5348.6 | 31.5 | | dbstat | 2024-04-10 | 2193.9 | 3712.8 | 13.0 | 5919.7 | 31.6 | | dbstat | 2024-04-11 | 2193.9 | 3712.8 | 15.0 | 5921.7 | 31.1 | | dbstat | 2024-04-12 | 2405.8 | 4119.1 | 12.0 | 6537.0 | 34.9 | | dbstat | 2024-04-13 | 2405.8 | 4119.1 | 14.0 | 6538.9 | 35.7 | | dbstat | 2024-04-14 | 2480.8 | 4278.9 | 15.0 | 6774.8 | 36.2 | | dbstat | 2024-04-15 | 2560.8 | 4443.7 | 12.0 | 7016.5 | 37.5 | | dbstat | 2024-04-16 | 2560.8 | 4443.7 | 12.0 | 7016.5 | 38.2 | | dbstat | 2024-04-17 | 2640.8 | 4610.6 | 18.0 | 7269.4 | 38.5 | | dbstat | 2024-04-18 | 2640.9 | 4611.6 | 14.0 | 7266.5 | 39.7 | | dbstat | 2024-04-19 | 2743.9 | 4826.5 | 14.0 | 7584.3 | 36.9 | | dbstat | 2024-04-20 | 2826.9 | 4995.5 | 14.0 | 7836.4 | 38.3 | | dbstat | 2024-04-21 | 2830.9 | 4997.4 | 18.0 | 7846.3 | 39.2 | | dbstat | 2024-04-22 | 2919.9 | 5177.4 | 14.0 | 8111.3 | 43.2 | | dbstat | 2024-04-23 | 2923.0 | 5177.3 | 16.0 | 8116.3 | 44.1 | | dbstat | 2024-04-24 | 3020.0 | 5376.3 | 16.0 | 8412.3 | 41.0 | | dbstat | 2024-04-25 | 3024.0 | 5377.3 | 17.0 | 8418.3 | 40.9 | +--------------+------------+---------+----------+---------+----------+--------+

Nimmt man zum Vergleich den Plattenplatz im O/S:

# du -shc *.ibd 8.6G global_status.ibd 308K global_variables.ibd 692K metadata_lock.ibd 97M processlist.ibd 18M table_size.ibd 212K trx_and_lck.ibd 8.7G total

sieht man, dass die Werte aus der Datenbank in etwa stimmen (5% Fehler)...

Wichtig: Die Datenbank dbstat erreicht nach ca. einem Monat eine Grösse von ca. 9 Gbyte auf einem nicht besonders grossen Datenbanksystem.

Man sieht auch, dass sich die Grösse der Datenbank gerade erst stabilisiert.

Wenn man genauer wissen will, welche Tabellen für welchen Teil des Datenvolumens verantwortlich sind, kann man auch in die Daten hineinzoomen bzw. hineindrillen (drill down):

SQL> SELECT `table_name`, SUBSTR(`ts`, 1, 10) AS date , ROUND(`data_length`/1024/1024, 1) AS data_mb , ROUND(`index_length`/1024/1024, 1) AS index_mb , ROUND(`data_free`/1024/1024, 1) AS free_mb , ROUND((`data_length` + `index_length` + `data_free`)/1024/1024, 1) AS total_mb , ROUND((`data_length` + `index_length` + `data_free`)/1024/1024/8418.26*100, 1) AS pct , ROUND(`table_rows`/1000/1000, 1) AS rows_m FROM `table_size` WHERE `machine_name` = @machine_name AND `table_catalog` = 'def' AND `table_schema` = 'dbstat' AND SUBSTR(`ts`, 1, 10) = CURRENT_DATE() ORDER BY rows_m DESC ; +------------------+------------+---------+----------+---------+----------+------+--------+ | table_name | date | data_mb | index_mb | free_mb | total_mb | pct | rows_m | +------------------+------------+---------+----------+---------+----------+------+--------+ | global_status | 2024-04-25 | 2949.9 | 5356.9 | 5.0 | 8311.8 | 98.7 | 40.4 | | processlist | 2024-04-25 | 68.2 | 17.1 | 7.0 | 92.2 | 1.1 | 0.4 | | global_variables | 2024-04-25 | 0.1 | 0.1 | 0.0 | 0.2 | 0.0 | 0.0 | | metadata_lock | 2024-04-25 | 0.4 | 0.2 | 0.0 | 0.6 | 0.0 | 0.0 | | table_size | 2024-04-25 | 5.4 | 3.1 | 5.0 | 13.5 | 0.2 | 0.0 | | trx_and_lck | 2024-04-25 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +------------------+------------+---------+----------+---------+----------+------+--------+

Anmerkung: Bitte entschuldigen Sie die Nichverwendung der Window-Funktion!

Der einzige wirkliche Treiber für die Datenmenge auf diesem System ist die Tabelle global_status. Dies ist auch zu erwarten (siehe Mengengerüst von dbstat).

SQL> SELECT SUBSTR(ts, 1, 10) AS date, table_rows/1000/1000 AS k_rows , ROUND(data_length/1024/1024, 1) AS data_mb, ROUND(index_length/1024/1024, 1) AS index_mb, ROUND(data_free/1024/1024, 1) AS free_mb , ROUND((data_length + index_length + data_free)/1024/1024, 1) AS total_mb FROM table_size WHERE `machine_name` = @machine_name AND `table_catalog` = 'def' AND `table_schema` = 'dbstat' AND table_name = 'global_status' AND ts > DATE_SUB(CURRENT_DATE, INTERVAL 10 DAY) ; +------------+-------------+---------+----------+---------+----------+ | date | k_rows | data_mb | index_mb | free_mb | total_mb | +------------+-------------+---------+----------+---------+----------+ | 2024-04-15 | 37.13876300 | 2512.9 | 4433.0 | 4.0 | 6949.9 | | 2024-04-16 | 37.94217200 | 2512.9 | 4433.0 | 4.0 | 6949.9 | + 0M | 2024-04-17 | 38.19867500 | 2592.9 | 4600.0 | 7.0 | 7199.9 | + 250M | 2024-04-18 | 39.39108500 | 2592.9 | 4600.0 | 5.0 | 7197.9 | - 2M | 2024-04-19 | 36.52539600 | 2691.9 | 4813.0 | 5.0 | 7509.8 | + 312M | 2024-04-20 | 37.99073500 | 2770.9 | 4980.9 | 6.0 | 7757.8 | + 248M | 2024-04-21 | 38.79420200 | 2770.9 | 4980.9 | 7.0 | 7758.8 | + 1M | 2024-04-22 | 42.82606200 | 2855.9 | 5158.9 | 6.0 | 8020.8 | + 263M | 2024-04-23 | 43.62953000 | 2855.9 | 5158.9 | 7.0 | 8021.8 | + 1M | 2024-04-24 | 40.54342200 | 2949.9 | 5356.9 | 7.0 | 8313.8 | + 292M | 2024-04-25 | 40.43067700 | 2949.9 | 5356.9 | 5.0 | 8311.8 | - 2M +------------+-------------+---------+----------+---------+----------+

Anmerkung: Sorry, ich sollte mich wirklich mit den Window-Funktionen vertraut machen...

Wenn wir die Daten etwas genauer analysieren, sehen wir, dass sich die Anzahl der Rows in den letzten 4 Tagen langsam stabilisiert hat (Achtung: table_rows wird berechnet (aus der Anzahl der Blöcke und der durchschnittlichen Zeilenlänge?) und ist kein exakter Wert), aber die "Datenmenge" hat bis gestern weiter zugenommen, was wahrscheinlich auf das "Zerfleddern" der Tabellen und Indizes zurückzuführen ist...

Der Primärschlüssel der Tabelle global_status wurde gewählt, um die Lokalisierung der Daten zu optimieren:

PRIMARY KEY (`machine_name`,`variable_name`,`ts`),

Die Lage sollte sich in den nächsten Tagen beruhigen. In 2 bis 4 Wochen müssen wir die Lage erneut prüfen.

Zusammenfassung: Ich würde sagen, dass dieses Feature die Anforderungen erfüllt und hilft, das Datenwachstum zu verstehen.

Liste der Prozesse

Da wir keine ernsthaften Lastprobleme in unseren Datenbanken haben, ist diese Funktion in unserem Fall nicht so interessant. Wir können zum Beispiel sehen, was eine (persistente) Verbindung gemacht hat:

SQL> SELECT connection_id, ts, command, time, state, SUBSTR(REGEXP_REPLACE(REPLACE(query, "\n", ' '), '\ +', ' '), 1, 64) FROM processlist WHERE machine_name = @machine_name AND command != 'Sleep' AND connection_id = @connection_id AND state NOT IN ( 'Waiting for next activation' , 'Master has sent all binlog to slave; waiting for more updates' , 'Waiting for master to send event' , 'Slave has read all relay log; waiting for more updates' ) ORDER BY ts ASC ; +---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+ | connection_id | ts | command | time | state | SUBSTR(REGEXP_REPLACE(REPLACE(query, "\n", ' '), '\ +', ' '), 1, 64) | +---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+ | 18 | 2024-04-17 12:30:28 | Query | 0.029 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-17 14:58:28 | Query | 0.009 | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-18 06:24:28 | Query | 0.003 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-18 11:34:28 | Query | 0.030 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-18 16:39:28 | Query | 0.006 | Sending data | select itemid,functionid,name,parameter,triggerid from functions | | 18 | 2024-04-18 19:12:28 | Query | 0.014 | Sending data | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-18 21:49:28 | Query | 0.004 | Writing to net | select i.itemid,i.hostid,i.templateid from items i inner join ho | | 18 | 2024-04-19 00:21:28 | Query | 0.032 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-19 02:59:28 | Query | 0.017 | Writing to net | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-19 05:39:28 | Query | 0.052 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-19 08:19:28 | Query | 0.000 | Statistics | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-19 13:26:28 | Query | 0.075 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-19 15:57:28 | Query | 0.027 | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-19 18:33:28 | Query | 0.010 | Sending data | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-19 21:10:28 | Query | 0.008 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-19 23:50:28 | Query | 0.067 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 02:28:28 | Query | 0.008 | Sending data | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-20 05:08:28 | Query | 0.052 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 07:44:28 | Query | 0.123 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 10:21:28 | Query | 0.144 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 12:55:28 | Query | 0.004 | Sending data | select i.itemid,i.hostid,i.templateid from items i where i.flags | | 18 | 2024-04-20 15:35:28 | Query | 0.092 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 18:12:28 | Query | 0.041 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 20:47:28 | Query | 0.113 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-20 23:25:28 | Query | 0.101 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 02:03:28 | Query | 0.120 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 04:42:28 | Query | 0.099 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 07:18:28 | Query | 0.015 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 12:32:28 | Query | 0.018 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 15:06:28 | Query | 0.091 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-21 20:16:28 | Query | 0.012 | Sending data | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-22 06:44:28 | Query | 0.161 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 09:21:28 | Query | 0.000 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 11:54:28 | Query | 0.020 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 14:23:28 | Query | 0.067 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 16:59:28 | Query | 0.128 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-22 22:05:28 | Query | 0.078 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 00:38:28 | Query | 0.084 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 03:15:28 | Query | 0.098 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 05:52:28 | Query | 0.000 | starting | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 08:27:28 | Query | 0.011 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h | | 18 | 2024-04-23 10:58:28 | Query | 0.000 | Sending data | select i.itemid,i.hostid,i.templateid from items i inner join ho | | 18 | 2024-04-23 13:31:28 | Query | 0.110 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 16:01:28 | Query | 0.023 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 18:35:28 | Query | 0.095 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-23 21:10:28 | Query | 0.017 | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 18 | 2024-04-23 23:44:28 | Query | 0.014 | Sending data | select triggerid,description,expression,error,priority,type,valu | | 18 | 2024-04-24 02:21:28 | Query | 0.024 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | | 18 | 2024-04-24 07:33:28 | Query | 0.046 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s | +---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+

Wichtig ist auch, dass wir in dieser Auswertung nur die Einträge sehen, wenn der Thread ETWAS gemacht hat (State Sleep haben wir ausgeblendet). Interessant ist auch, dass wir diese (persistente) Verbindung nicht vor dem 17. April sehen, aber ich habe im Moment KEINE Erklärung dafür aus operativer Sicht (Restart etc.). Wahrscheinlich muss die Anwendung (Zabbix) das erklären.

Globale Variablen

Interessant sind auch die Informationen in der Tabelle global_variables:

SQL> SELECT variable_name, ts, variable_value FROM global_variables WHERE machine_name = @machine_name AND variable_name IN ( SELECT variable_name FROM global_variables WHERE machine_name = @machine_name GROUP BY variable_name HAVING COUNT(*) > 1 ) ORDER BY ts, variable_name ; +---------------------------+---------------------+----------------+ | variable_name | ts | variable_value | +---------------------------+---------------------+----------------+ | auto_increment_increment | 2024-03-09 22:10:42 | 1 | | auto_increment_offset | 2024-03-09 22:10:42 | 1 | | read_only | 2024-03-09 22:10:42 | OFF | | slave_parallel_max_queued | 2024-03-09 22:10:42 | 131072 | | slave_parallel_threads | 2024-03-09 22:10:42 | 0 | | slave_parallel_workers | 2024-03-09 22:10:42 | 0 | | slave_skip_errors | 2024-03-09 22:10:42 | OFF | | system_time_zone | 2024-03-09 22:10:42 | CET | | read_only | 2024-03-27 09:42:50 | ON | | slave_skip_errors | 2024-03-27 12:33:13 | 1032 | | slave_skip_errors | 2024-03-27 12:35:13 | OFF | | slave_skip_errors | 2024-03-27 12:42:13 | 1032 | | slave_skip_errors | 2024-03-27 12:50:13 | OFF | | slave_parallel_threads | 2024-04-02 10:17:32 | 8 | | slave_parallel_workers | 2024-04-02 10:17:32 | 8 | | slave_parallel_max_queued | 2024-04-02 10:22:32 | 1048576 | | slave_parallel_max_queued | 2024-04-02 10:23:32 | 4194304 | | slave_parallel_max_queued | 2024-04-02 10:25:32 | 16777216 | | slave_parallel_threads | 2024-04-02 10:25:32 | 16 | | slave_parallel_workers | 2024-04-02 10:25:32 | 16 | | slave_parallel_threads | 2024-04-02 10:28:32 | 32 | | slave_parallel_workers | 2024-04-02 10:28:32 | 32 | | auto_increment_increment | 2024-04-02 10:39:32 | 2 | | auto_increment_offset | 2024-04-02 10:39:32 | 2 | | slave_parallel_max_queued | 2024-04-02 10:57:32 | 131072 | | slave_parallel_threads | 2024-04-02 10:57:32 | 0 | | slave_parallel_workers | 2024-04-02 10:57:32 | 0 | | system_time_zone | 2024-04-02 10:57:32 | CEST | | slave_parallel_max_queued | 2024-04-16 14:06:32 | 16777216 | | slave_parallel_threads | 2024-04-16 14:06:32 | 8 | | slave_parallel_workers | 2024-04-16 14:06:32 | 8 | | slave_parallel_max_queued | 2024-04-16 14:26:32 | 131072 | | slave_parallel_threads | 2024-04-16 14:26:32 | 0 | | slave_parallel_workers | 2024-04-16 14:26:32 | 0 | | slave_parallel_max_queued | 2024-04-17 09:03:32 | 16777216 | | slave_parallel_threads | 2024-04-17 09:03:32 | 16 | | slave_parallel_workers | 2024-04-17 09:03:32 | 16 | | slave_parallel_max_queued | 2024-04-24 08:26:32 | 131072 | | slave_parallel_threads | 2024-04-24 08:26:32 | 0 | | slave_parallel_workers | 2024-04-24 08:26:32 | 0 | | read_only | 2024-04-24 08:42:32 | OFF | +---------------------------+---------------------+----------------+

Hier sieht man sehr genau, wann und was an der Datenbank gemacht wurde:

Am 9. März wurde dbstat zum ersten Mal installiert.
Dann am 27. März (vor Ostern) scheint es dann Probleme mit der Replikation gegeben zu haben (hier wurde die neue Version von dbstat installiert, die das gleichzeitige Sammeln auf Master und Slave erlaubt. Dies führte zu Replikationsfehlern, die teilweise behoben wurden).
Am 2. April (nach Ostern) haben wir dann versucht, den Rückstand mit der parallelen Replikation aufzuholen. Man sieht auch, dass AUTO_INCREMENT_OFFSET und AUTO_INCREMENT_INCREMENT geändert wurden. Hier haben wir einen Fehler in der Datenbankkonfiguration korrigiert...
Ausserdem ist zu sehen, dass die Zeitzone von CET auf CEST geändert hat (Sommerzeit!) Warum erst am 2. April ist mir nicht ganz klar. (Vielleicht weil das über die Replikation gekommen ist?)
Dann haben wir am 16. und 17. April haben wir versucht einen "Bug" in der parallelen Replikation zu reproduzieren. Anscheinend haben wir den Wert nicht zurückgesetzt. Denn erst nach dem Neustart am 24. April (übliches zweiwöchentliches Wartungsfenster) wurde der Wert wieder zurückgesetzt.
Am 24. April sieht man auch, dass die Datenbank jetzt die Rolle des aktiven Masters übernommen hat (read_only = off). Es hat also ein Gracefull-Switchover stattgefunden...

Fazit: Ein sehr nützliches Feature um zu sehen, wann was geändert wurde. Obwohl ich alle diese Operationen genau verfolgt habe, bin ich doch erstaunt über die Aussagekraft dieses Features. Ich würde mir wünschen, dass es in allen Datenbanken installiert wird...

Metadata Lock und InnoDB Transaction Lock

Aufgrund des geringen Traffics auf unseren Datenbanken sehen wir hier leider nicht allzu viel Spannendes.

Hier sind die Metadata Locks, die wir in den den letzten 24 Stunden auf dem Master "erwischt" haben:

+---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+ | connection_id | ts | user | host | table_schema | table_name | state | SUBSTR(REGEXP_REPLACE(REPLACE(query, "\n", ' '), '\ +', ' '), 1, 64) | +---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+ | 18 | 2024-04-23 14:16:47 | zabbix | localhost:51252 | zabbix | triggers | Writing to net | select triggerid,description,expression,error,priority,type,valu | | 1325025 | 2024-04-23 16:01:47 | zabbix | localhost:50150 | | | init for update | delete from history_text where itemid=85477 and clock<1678167661 | | 1325025 | 2024-04-23 16:01:47 | zabbix | localhost:50150 | zabbix | history_text | init for update | delete from history_text where itemid=85477 and clock<1678167661 | | 1365229 | 2024-04-24 02:13:47 | root | localhost:38096 | dbstat | global_status | Writing to net | SELECT /*!40001 SQL_NO_CACHE */ `machine_name`, `variable_name`, | | 18 | 2024-04-24 03:10:47 | zabbix | localhost:51252 | zabbix | item_tag | Writing to net | select itemtagid,itemid,tag,value from item_tag | | 1368524 | 2024-04-24 04:41:47 | zabbix | localhost:38112 | | | | NULL | | 1368524 | 2024-04-24 04:41:47 | zabbix | localhost:38112 | zabbix | history_uint | | NULL | | 18 | 2024-04-24 05:46:47 | zabbix | localhost:51252 | zabbix | item_tag | Sending data | select itemtagid,itemid,tag,value from item_tag | +---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+

InnoDB-Locks haben wir in den letzten 24 Stunden repektive 7 Tage keine gefunden.

Es wäre interessant, ein System zu sehen, auf dem mehr passiert...

Globaler Status

Wenn ein normales Datenbankmonitoring wie z.B. der FromDual Performance Monitor für MariaDB und MySQL (fpmmm) mit Zabbix verwendet wird, ist dieses Feature nicht unbedingt notwendig. Die meisten unserer Kunden haben jedoch kein brauchbares Monitoring im Einsatz. Daher wäre dieses Feature sehr nützlich für Post-Morten-Analysen...

Zum Beispiel InnoDB-Lock-Waits, minuten-granular über die letzten 30 Tage (analog zu sar aus sysstat):

Hier sieht man, dass die Datenbank am 10. April zwischen 08:37 und 08:41 neu gestartet wurde. Das könnte man zwar auch anders herausfinden, ist aber leider oft aus verschiedenen Gründen nicht möglich (Error Log wegrotiert, etc.).

Interessant ist auch der Trendbruch um den 2. April. Zu diesem Zeitpunkt haben wir mit der parallelen Replikation rumexperimentiert. Es sollte kein Failover gewesen sein (siehe GLOBAL VARIABLES, weiter oben).

Obwohl die parallele Replikation später wieder deaktiviert wurde, gab es mehr Locks. Eine ähnliche Situation um den 16./17. April auch hier haben wir mit der parallelen Replikation herumgespielt, was sich auf das Locking-Verhalten ausgewirkt zu haben scheint.

Auch mit diesem Feature gibt es viele Möglichkeiten die Datenbank zu untersuchen. Leider ist unsere Datenbank relativ langweilig: Hauptsächlich monotoner Traffic (der durch das Monitoring reichlich vorhanden ist) und aussergwöhnlicher Traffic in sehr geringem Umfang.

Anmerkung: Dieser Text wurde mit der Unterstützung von DeepL optimiert.

Taxonomy upgrade extras: performance monitoring performance monitoring metadata lock locking performance_schema

Categories: FromDual all (de), FromDual TechFeed (de)

dbstat nach einem Monat produktiver Nutzung

Oli Sennhauser - Thu, 2024-04-25 12:39

Inhaltsverzeichnis

Rückblick
Einen Monat später
Grösse der Tabellen
Prozessliste
Globale Variablen
Metadata Lock und InnoDB Transaction Lock
Globaler Status