MySQL blank users preventing slave I/O connection?

Hi,
Recently I have encountered a weird simple issue with starting Slave server.

So created user as tried to run change master:

2017-10-30 14:03:59 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e 
'select @@port'
2017-10-30 14:03:59 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e 
"CREATE USER 'repl'@'%' IDENTIFIED BY 'Baku12345'"
2017-10-30 14:03:59 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e 
"GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%'"
2017-10-30 14:03:59 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/sock0.sock --force test -e 
"CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_USER='repl', MASTER_PASSWORD='Baku12345', MASTER_PORT=10023, MASTER_AUTO_POSITION=1"
2017-10-30 14:03:59 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/sock0.sock --force test -e 
'start slave'

After that got the error:

2017-10-30 13:22:05 1314 [ERROR] Slave I/O: error connecting to master 'repl@localhost:10023' - retry-time: 60  retries: 11, Error_code: 2003
2017-10-30 13:22:48 7182 [ERROR] Slave I/O: error connecting to master 'repl@localhost:10023' - retry-time: 60  retries: 106, Error_code: 2003
2017-10-30 13:22:57 28984 [ERROR] Slave I/O: error connecting to master 'repl@localhost:10023' - retry-time: 60  retries: 1316, Error_code: 2003
2017-10-30 13:23:05 1314 [ERROR] Slave I/O: error connecting to master 'repl@localhost:10023' - retry-time: 60  retries: 12, Error_code: 2003
2017-10-30 13:23:48 7182 [ERROR] Slave I/O: error connecting to master 'repl@localhost:10023' - retry-time: 60  retries: 107, Error_code: 2003
2017-10-30 13:23:57 28984 [ERROR] Slave I/O: error connecting to master 'repl@localhost:10023' - retry-time: 60  retries: 1317, Error_code: 1045
2017-10-30 13:24:05 1314 [ERROR] Slave I/O: error connecting to master 'repl@localhost:10023' - retry-time: 60  retries: 13, Error_code: 1045
2017-10-30 13:24:48 7182 [ERROR] Slave I/O: error connecting to master 'repl@localhost:10023' - retry-time: 60  retries: 108, Error_code: 1045

I got the same result with MySQL 5.5 today and remember previous case with MySQL 5.6

2017-11-24 10:14:15 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231117-5.5.58-38.10-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231117-5.5.58-38.10-linux-x86_64-debug/sock1.sock --force test -e 'show slave status\G'
2017-11-24 10:14:15 DEBUG    <pid.PidFile object at 0x7fbe55751cc8> closing pidfile: /tmp/MySQL-AutoXtraBackup/autoxtrabackup.pid
Traceback (most recent call last):
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/bin/autoxtrabackup", line 11, in <module>
    load_entry_point('mysql-autoxtrabackup==1.5.0', 'console_scripts', 'autoxtrabackup')()
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.5.0-py3.5.egg/autoxtrabackup.py", line 209, in all_procedure
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.5.0-py3.5.egg/prepare_env_test_mode/runner_test_mode.py", line 464, in wipe_backup_prepare_copyback
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.5.0-py3.5.egg/prepare_env_test_mode/runner_test_mode.py", line 378, in run_change_master
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.5.0-py3.5.egg/prepare_env_test_mode/runner_test_mode.py", line 152, in check_slave_status
RuntimeError: Slave_IO_Running is not Yes
2017-11-24 10:14:15 DEBUG    <pid.PidFile object at 0x7fbe55751cc8> closing pidfile: /tmp/MySQL-AutoXtraBackup/autoxtrabackup.pid

So after that, I tried to connect manually via replication user:

$ /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -urepl --password='Baku12345' --port=10023 --socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock
Warning: Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'repl'@'localhost' (using password: YES)

Removed blank users from master:

5.6.37>drop user ''@'localhost';
Query OK, 0 rows affected (0.01 sec)

5.6.37>drop user ''@'qaserver-02.ci.percona.com';
Query OK, 0 rows affected (0.00 sec)

Now reconnecting:

$ /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -urepl --password='Baku12345' --port=10023 --socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 623
Server version: 5.6.37-82.2-debug-log MySQL Community Server (GPL)

So it turned out, the blank users were preventing slave connection to master server 🙂

Also there is a nice bug report Slave can’t reconnect after 2003 error when sha256_password is used

XtraBackup vs. OOM killer(usage of mprof)

This small article is about nicely depicting memory usage prior to OOM killers show.

I have started MySQL with:

/home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/bin/mysqld --no-defaults \
--innodb_buffer_pool_size=1G --innodb_log_file_size=1G --innodb_page_size=64K --early-plugin-load=keyring_file.so \ --keyring_file_data=/home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/mysql-keyring/keyring \
--log-bin=mysql-bin --log-slave-updates --server-id=1 --gtid-mode=ON --enforce-gtid-consistency --binlog-format=row --core-file --basedir=/home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64 \
--tmpdir=/home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/data \
--datadir=/home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/data \
--plugin-load-add=tokudb=ha_tokudb.so --tokudb-check-jemalloc=0 --plugin-load-add=rocksdb=ha_rocksdb.so 
--socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/socket.sock --port=17057 \
--log-error=/home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/log/master.err

So basically it is MySQL 5.7.19 with 64K page size.

Trying to take full + incremental backups:

Full is OK:

/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup --defaults-file= \
--user=root --password=''  --target-dir=/home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/full/2017-11-14_11-31-20 \
--backup --socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/socket.sock \
--compress=quicklz --compress-chunk-size=65536 --compress-threads=4 --encrypt=AES256 --encrypt-key=VVTBwgM4UhwkTTV98fhuj+D1zyWoA89K \
--encrypt-threads=4 --encrypt-chunk-size=65536 --no-version-check --core-file --parallel=10 --throttle=40 \
--keyring-file-data=/home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/mysql-keyring/keyring --stream="xbstream" > \
/home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/full/2017-11-14_11-31-20/full_backup.stream

Incremental Failed:

/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup --defaults-file= \
--user=root --password='' --target-dir=/home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/inc/2017-11-14_11-33-51 \
--incremental-basedir=/home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/full/2017-11-14_11-31-20 --backup \
--socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/socket.sock \
--compress=quicklz --compress-chunk-size=65536 --compress-threads=4 --encrypt=AES256 \
--encrypt-key=VVTBwgM4UhwkTTV98fhuj+D1zyWoA89K --encrypt-threads=4 --encrypt-chunk-size=65536 \
--no-version-check --core-file --parallel=10 --throttle=40 \
--keyring-file-data=/home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/mysql-keyring/keyring --stream="xbstream" > \
 /home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/inc/2017-11-14_11-33-51/inc_backup.stream

171114 11:35:05 Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: /home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/socket.sock
Using server version 5.7.19-17-debug-log
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup version 2.4.7 based on MySQL server 5.7.13 Linux (x86_64) (revision id: )
incremental backup from 70156215 is enabled.
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /home/shahriyar.rzaev/XB_TEST/server_dir/PS131117-percona-server-5.7.19-17-linux-x86_64/data/
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup:   innodb_data_home_dir = .
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 2
xtrabackup:   innodb_log_file_size = 1073741824
InnoDB: Number of pools: 1
171114 11:35:06 >> log scanned up to (70156224)
xtrabackup: Generating a list of tablespaces
InnoDB: Allocated tablespace ID 2 for mysql/plugin, old maximum was 0
xtrabackup: Starting 10 threads for parallel data files transfer
171114 11:35:07 >> log scanned up to (70156224)
Killed

I forgot about already reported bug here -> #1582456

So according to this report and this code line:

write_filt.cc#L78

The buffer calculation for incremental backup is:

/* allocate buffer for incremental backup (4096 pages) */
	buf_size = (cursor->page_size / 4 + 1) * cursor->page_size +
UNIV_PAGE_SIZE_MAX;

So with 64K MySQL it is going to allocate:

(64*1024/4+1)*64*1024+64*1024 == 1073872896 == 1024 MB == 1G roughly.

And if you are going to use –parallel=10 as me, it is going to something like 10*1G = 10G. Due to this it is quite fair action by OOM killer to terminate my process.

After that, I was searching for nice simple solution to depict memory usage of executable and found this Python module:

memory_profiler

To install it:

sudo pip3.5 install memory_profiler
sudo pip3.5 install psutil
sudo pip3.5 install matplotlib

You need tkinter as well:

sudo yum -y install python35u-tkinter

You need xauth as well, for plotting:

sudo yum -y install xauth

After that connecting to my Vagrant VM with -X option:

ssh -X vagrant@127.0.0.1 -i /home/shako/REPOS/MySQL-AutoXtraBackup/.vagrant/machines/default/virtualbox/private_key -p 2222

And running following command:

mprof run append_here_the_command_to_run

It will run the given command and create some file mprofile_20171114122250.dat.

Now run following:

[vagrant@dhcppc3 ~]$ mprof plot mprofile_20171114122250.dat

And it will create:

BTW I have reported a bug to mprof as well: #171
Thanks for reading 🙂

Getting started MySQL Group Replication on Ubuntu with MySQL Sandbox

Welcome to next topic dedicated to Group Replication plugin. Today’s topic is to get started with Group Replication. As you remember we set up our environment using MySQL Sandbox on Ubuntu 14.04 and also we have compiled plugin with MySQL 5.7.8-rc on Ubuntu 14.04. Please refer to this topics respectively:

  1. MySQL Sandbox creating test environment
  2. Compiling MySQL Group replication plugin

So we have already our 3 nodes of MySQL 5.7.8-rc + group replication plugin.
Before starting group replication. We need to play with Corosync. Here is very dedicated article to this -> Corosync Guide

I want to talk about some issues where you may see and help to resolve these problems.
First thing is when you try to start group replication, without starting corosync daemon or while corosync is stopped:

mysql> START GROUP_REPLICATION;
ERROR 3092 (HY000): The server is not configured properly to be an active member of the group. Please see more details on error log.

The error indicates that, you will find some input in error log but you won’t 🙂
So just be sure to start corosync daemon before getting your hand with group replication.

Second thing is to save correct user on /etc/corosync/uidgid.d directory.
For eg, if you will start MySQL service with some ‘john’ Linux user, you must put this user into some file. For me, it is ‘sh’ local user on my Ubuntu:

sh@shrzayev:~$ cat /etc/corosync/uidgid.d/users 
uidgid {
 uid: sh
 gid: sh
}

If you will save wrong user you will likely see:

node1 [localhost] {msandbox} ((none)) > start group_replication;
ERROR 3092 (HY000): The server is not configured properly to be an active member of the group. Please see more details on error log.

But again there will be no such input in error log indicating exact issue.
Just be sure to save exact user related to corosync.

These tips came from Pedro Gomes. He is a replication developer from Oracle.
For further discussion please refer to related report -> #78619

Let’s assume you have successfully completed Corosync setup.
Now it is time to config MySQL server for group replication plugin.
For all 3 nodes the my.cnf file should include following settings:

server-id                      = UNIQUE_NUMBER_HERE
binlog-format                  = row
enforce-gtid-consistency       = true
gtid-mode                      = on
log_slave_updates              = 1
master-info-repository         = TABLE
relay-log-info-repository      = TABLE
binlog-checksum                = NONE
transaction-write-set-extraction=MURMUR32

With MySQL Sandbox there are already unique server-ids and some other usefull system variables with my.sandbox.cnf file. Just add non-existing parameters to config file. Start all 3 nodes first time:

sh@shrzayev:~/sandboxes/multi_msb_5_7_8/node[1,2,3]$ ./start
.. sandbox server started

Just be sure that, all servers started successfully:

sh@shrzayev:~/sandboxes/multi_msb_5_7_8/node[1,2,3]$ ./status
node1 on

Now stop all 3 nodes and append –plugin-load=group_replication.so to ./start script:

sh@shrzayev:~/sandboxes/multi_msb_5_7_8/node[1,2,3]$ ./start --plugin-load=group_replication.so
... sandbox server started

You can check plugin status:

node[1,2,3] [localhost] {msandbox} ((none)) > select plugin_name, plugin_status from information_schema.plugins where plugin_name like 'group_%';
+-------------------+---------------+
| plugin_name       | plugin_status |
+-------------------+---------------+
| group_replication | ACTIVE        |
+-------------------+---------------+
1 row in set (0,01 sec)

You have enabled group_replication plugin.
We need to create replication user do it only on node1:

CREATE USER 'rpl_user'@'%' IDENTIFIED BY 'rpl_pass';
GRANT REPLICATION SLAVE ON *.* TO rpl_user@'%';
GRANT SELECT on performance_schema.global_variables TO rpl_user@'%';

Also we need to give a SELECT grant to performance_schema.global_variables table, otherwise you will get an error:

2015-10-08T07:32:26.000461Z 8 [ERROR] Slave I/O for channel 'group_replication_recovery': The slave I/O thread stops because a fatal error is e
ncountered when it try to get the value of SERVER_ID variable from master. Error: SELECT command denied to user 'rpl_user'@'localhost' for tabl
e 'global_variables', Error_code: 1142

Now there are some global variables to change. do it on node1:

SET GLOBAL group_replication_group_name= "8a94f357-aab4-11df-86ab-c80aa9429562";
SET GLOBAL group_replication_recovery_user='rpl_user';
SET GLOBAL group_replication_recovery_password='rpl_pass';
SET GLOBAL group_replication_recovery_retry_count= 2;
SET GLOBAL group_replication_recovery_reconnect_interval= 120;

After all we can start group replication:

node1 [localhost] {msandbox} ((none)) > start group_replication;
Query OK, 0 rows affected (0,16 sec)

In MySQL error log, you will see new entries:

2015-10-08T05:55:22.419952Z 4 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 3307, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 3307, master_log_file='', master_log_pos= 4, master_bind=''.
2015-10-08T05:55:22.578212Z 3 [Note] Plugin group_replication reported: 'Group Replication applier module successfully initialized!'
2015-10-08T05:55:22.578283Z 5 [Note] Slave SQL thread for channel 'group_replication_applier' initialized, starting replication in log 'FIRST' at position 0, relay log './mysql-relay-group_replication_applier.000001' position: 4
2015-10-08T05:55:22.578306Z 3 [Note] Plugin group_replication reported: 'auto_increment_increment is set to 7'
2015-10-08T05:55:22.578359Z 3 [Note] Plugin group_replication reported: 'auto_increment_offset is set to 101'
2015-10-08T05:55:22.579736Z 0 [Note] Plugin group_replication reported: 'Starting group replication recovery with view_id 571734024:1'
2015-10-08T05:55:22.712797Z 6 [Note] Plugin group_replication reported: 'Only one server alive. Declaring this server as online within the replication group'
2015-10-08T05:55:22.713721Z 0 [Note] Plugin group_replication reported: 'This server was declared online within the replication group'

Even you can notice new connection from corosync log file:

Oct 08 10:55:22 [1257] shrzayev corosync debug   [QB    ] IPC credentials authenticated (1293-7202-19)
Oct 08 10:55:22 [1257] shrzayev corosync debug   [QB    ] connecting to client [7202]
Oct 08 10:55:22 [1257] shrzayev corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Oct 08 10:55:22 [1257] shrzayev corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Oct 08 10:55:22 [1257] shrzayev corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Oct 08 10:55:22 [1257] shrzayev corosync debug   [MAIN  ] connection created
Oct 08 10:55:22 [1257] shrzayev corosync debug   [CPG   ] lib_init_fn: conn=0x7fe2da6a4e60, cpd=0x7fe2da6a5454
Oct 08 10:55:22 [1257] shrzayev corosync debug   [CPG   ] got procjoin message from cluster node 2130706433 (r(0) ip(127.0.0.1) ) for pid 7202

Now you can check group replication member status:

node1 [localhost] {msandbox} ((none)) > SELECT * FROM performance_schema.replication_connection_statusG
*************************** 1. row ***************************
             CHANNEL_NAME: group_replication_applier
               GROUP_NAME: 8a94f357-aab4-11df-86ab-c80aa9429562
              SOURCE_UUID: 8a94f357-aab4-11df-86ab-c80aa9429562
                THREAD_ID: NULL
            SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 0
 LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
 RECEIVED_TRANSACTION_SET: 8a94f357-aab4-11df-86ab-c80aa9429562:1-2
        LAST_ERROR_NUMBER: 0
       LAST_ERROR_MESSAGE: 
     LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
1 row in set (0,01 sec)

So there is no ERRORs here, it is working.
You can check also group members:

node1 [localhost] {msandbox} ((none)) > SELECT * FROM performance_schema.replication_group_membersG
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
   MEMBER_ID: 00008379-1111-1111-1111-111111111111
 MEMBER_HOST: shrzayev
 MEMBER_PORT: 8379
MEMBER_STATE: ONLINE
1 row in set (0,01 sec)

At this moment we have only 1 member -> node1.
Let’s add node2 too. Procedure is same set global variables and start group replication:

SET GLOBAL group_replication_group_name= "8a94f357-aab4-11df-86ab-c80aa9429562";
SET GLOBAL group_replication_recovery_user='rpl_user';
SET GLOBAL group_replication_recovery_password='rpl_pass';
SET GLOBAL group_replication_recovery_retry_count= 2;
SET GLOBAL group_replication_recovery_reconnect_interval= 120;

And start group replication:

node2 [localhost] {msandbox} ((none)) > start group_replication;
Query OK, 0 rows affected (0,16 sec)

Same procedure for node3 and we will have 3 nodes on our group:

node1 [localhost] {msandbox} ((none)) > select * from performance_schema.replication_group_membersG
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
   MEMBER_ID: 00008379-1111-1111-1111-111111111111
 MEMBER_HOST: shrzayev
 MEMBER_PORT: 8379
MEMBER_STATE: ONLINE
*************************** 2. row ***************************
CHANNEL_NAME: group_replication_applier
   MEMBER_ID: 00008380-2222-2222-2222-222222222222
 MEMBER_HOST: shrzayev
 MEMBER_PORT: 8380
MEMBER_STATE: ONLINE
*************************** 3. row ***************************
CHANNEL_NAME: group_replication_applier
   MEMBER_ID: 00008381-3333-3333-3333-333333333333
 MEMBER_HOST: shrzayev
 MEMBER_PORT: 8381
MEMBER_STATE: ONLINE
3 rows in set (0,00 sec)

That’s all. Here it is, how we actually can use MySQL Sandbox on Ubuntu 14.04 to create environment and to get started with Group Replication plugin.
As well as from now, you should be able to get start with Group Replication.
If you have your very own test cases, feel free to share or to ask.

MySQL Sandbox, creating test environment for Group Replication plugin

Today’s topic is about how to test Group Replication plugin in easy way, with using MySQL Sandbox
But firstly we must compile MySQL with Group Replication plugin, refer to previous topic -> Compiling Group Replication Plugin

Who did not use Sandbox before, please refer to official link and read a bit more. It is wonderful tool for installing, testing new things with MySQL. Especially if you have new MySQL release to test and want to install 3 or 5 instances, just use MySQL Sandbox and it will do this work for you.

In our condition, we have already compiled MySQL with our plugin. So we have source folder in -> /home/sh/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN

With MySQL Sandbox there is a wonderful tool, named make_sandbox_from_source.
Let’s see options available with this tool:

sh@shrzayev:~$ make_sandbox_from_source --help
    The MySQL Sandbox,  version 3.1.00
    (C) 2006-2015 Giuseppe Maxia

*** sandbox_type missing
Syntax: /usr/local/bin/make_sandbox_from_source source_directory sandbox_type [options]

source directory is where you have successfully
    run ./configure && make

sandbox_type is one of 
    single
    replication
    circular
    multiple

options is anything that is needed by the sandbox
    application of your choice

We want to test Group Replication plugin, so we need multiple option it will install 3 nodes of same version of MySQL by default. So after running we will get on our hand 3 nodes of Group Replication enabled MySQL.
We must just specify the source folder as follows:

sh@shrzayev:~$ make_sandbox_from_source /home/sh/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN multiple
can't find executable script 'make_binary_distribution'

Again with appending BUILD folder:

sh@shrzayev:~$ make_sandbox_from_source /home/sh/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN/BUILD multiple
can't find executable script 'make_binary_distribution'

No success.
If you go on and list BUILD folder:

sh@shrzayev:~/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN/BUILD$ ls
mysql-server  mysql-server.build  mysql-server.inst

There is a mysql-server.build folder which we will append and run again:

sh@shrzayev:~$ make_sandbox_from_source /home/sh/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN/BUILD/mysql-server.build/ multiple
can't find a version in Makefile

Error has changed -> can’t find a version in Makefile
After searching i found a BUG report(LP BUG 853764) related to this issue and found a workaround. We should add MYSQL_NO_DASH_VERSION option in Makefile:

sh@shrzayev:~/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN/BUILD$ cd mysql-server.build/
sh@shrzayev:~/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN/BUILD/mysql-server.build$ vim Makefile

And put MYSQL_NO_DASH_VERSION=5.7.8 (for me) anywhere in Makefile, save and exit. Then rerun:

sh@shrzayev:~$ make_sandbox_from_source /home/sh/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN/BUILD/mysql-server.build/ multiple
no old tarball found
CPack: Create package using TGZ
CPack: Install projects
CPack: - Run preinstall target for: MySQL
CPack: - Install project: MySQL
CPack: Create package
CPack: - package: /home/sh/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN/BUILD/mysql-server.build/mysql-5.7.8-rc-linux-x86_64.tar.gz generated.
installing node 1
installing node 2
installing node 3
group directory installed in $HOME/sandboxes/multi_msb_mysql-5_7_8

That’s it we have 3 nodes and we are ready to test plugin.
Check for node1 for eg:

node1 [localhost] {msandbox} ((none)) > select @@plugin_dir;
+-----------------------------------------------------------------------------------------------------+
| @@plugin_dir                                                                                        |
+-----------------------------------------------------------------------------------------------------+
| /home/sh/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN/BUILD/mysql-server.build/5.7.8/lib/plugin/ |
+-----------------------------------------------------------------------------------------------------+
1 row in set (0,00 sec)

sh@shrzayev:~$ ls /home/sh/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN/BUILD/mysql-server.build/5.7.8/lib/plugin/ | grep group_
group_replication_license
group_replication.so

We have our plugin in plugin directory.
In the subsequent topics i will talk about activating and using group replication plugin. Again, we are using Ubuntu 14.04 with MySQL 5.7.8-rc2 and mysql-group-replication-0.5.0-dmr.

Compiling MySQL Group Replication plugin with MySQL 5.7.8-rc2 on Ubuntu

MySQL Group Replication plugin is in labs.mysql.com and is available for EL6 x86_64 version Linux. But most of us have Ubuntu desktops where it should be easier to test this new thing, especially with MySQL Sandbox. After getting source code we should have compile this plugin with MySQL from source. So let’s begin. Extract both mysql group replication archive and mysql source archive:

sh@shrzayev:~/Sandboxes$ ls -l
total 650732
drwxr-xr-x 34 sh sh      4096 Ä°yl 20 17:25 mysql-5.7.8-rc
-rw-rw-r--  1 sh sh  49762480 Avq 20 16:19 mysql-5.7.8-rc.tar.gz
drwxrwxr-x  3 sh sh      4096 Sen 28 12:08 mysql-group-replication-0.5.0-dmr
-rw-rw-r--  1 sh sh    251687 Sen 28 11:57 mysql-group-replication-0.5.0-labs.tar.gz

You will have 2 directories as above. Then, go to mysql-group-replication folder:

sh@shrzayev:~/Sandboxes$ cd mysql-group-replication-0.5.0-dmr/
sh@shrzayev:~/Sandboxes/mysql-group-replication-0.5.0-dmr$ mkdir BIN
sh@shrzayev:~/Sandboxes/mysql-group-replication-0.5.0-dmr$ cd BIN

Inside BIN folder:

> NOTE: there must be no spaces beetween cmake options(between ";")

You can run following cmake command:

cmake ..  -DMYSQL_SERVER_SRC_DIR="/home/sh/Sandboxes/mysql-5.7.8-rc" -DMYSQL_SERVER_CMAKE_ARGS="-DMYSQL_DATADIR=/opt/mysql-5.7.8-grouprepl/datadir;-DSYSCONFDIR=/opt/mysql-5.7.8-grouprepl;-DWITH_SSL=system;-DMYSQL_TCP_PORT=3307;-DMYSQL_UNIX_ADDR=/opt/mysql-5.7.8-grouprepl/mysqld.sock;-DDEFAULT_CHARSET=utf8;-DDEFAULT_COLLATION=utf8_general_ci;-DWITH_DEBUG=1;-DCOMPILATION_COMMENT='Group Replication enabled MySQL-5.7.8';-DOPTIMIZER_TRACE=1;-DWITH_ZLIB=system;-DWITH_VALGRIND=1;-DCMAKE_C_FLAGS=-DHAVE_purify;-DCMAKE_CXX_FLAGS=-DHAVE_purify;-DDOWNLOAD_BOOST=1;-DWITH_BOOST=/home/sh/Sandboxes"

But before running this command you must install corosync and related packages, if not, you will have success with compilation but will get an empty binary package 🙂 For further reading refer to related BUG report -> #78600

So install dependencies:

sudo apt-get install corosync corosync-dev libcpg-dev

Then run cmake command above and after sucessful cmake run:

make package

If you list BIN folder:

sh@shrzayev:~/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN$ ls
BUILD           CMakeFiles           CPackConfig.cmake  CPackSourceConfig.cmake  mysql-group-replication-0.5.0-dmr-amd64.tar.gz  SOURCE
CMakeCache.txt  cmake_install.cmake  _CPack_Packages    Makefile                 mysql_server-prefix

You will notice that there is, mysql-group-replication-0.5.0-dmr-amd64.tar.gz package where is group_replication.so file located:

sh@shrzayev:~/Sandboxes/mysql-group-replication-0.5.0-dmr/BIN$ tar -tf mysql-group-replication-0.5.0-dmr-amd64.tar.gz | grep group_replication.so
mysql-group-replication-0.5.0-dmr-amd64/lib/plugin/group_replication.so

From now we have mysql group replication plugin for 64 bit Ubuntu 14.04. We will continue our topic about testing group replication plugin in easy ways. There is some other related issues, if you are interested refer to my previous article: Group Replication Plugin issues

MySQL Group Replication plugin issues with 5.7.8-rc2

Well the new things are cool to test and learn. But it is not possible every time, to have smooth testing process. if you can’t get right software on your hand.

So there are great articles about this new plugin here you can follow all of them: Group Replication Topics

I must say that, plugin available for download as source code and as for Oracle Linux 6 x86_64 from -> labs.mysql.com

So if you want to try it with MySQL 5.7.8-rc2 on Ubuntu you should compile it with MySQL. But you will not be able to compile due to well known ‘boost’ dependency -> see BUG #78600

UPDATE to BUG 78600:

Pedro Gomes clarified the reason of failed compiling process. There must not be any “spaces” between cmake options specified with -DMYSQL_SERVER_CMAKE_ARGS=””. So we have working cmake command as follows:

    cmake ..  -DMYSQL_SERVER_SRC_DIR="/home/sh/Sandboxes/mysql-5.7.8-rc" -DMYSQL_SERVER_CMAKE_ARGS="-DMYSQL_DATADIR=/opt/mysql-5.7.8-grouprepl/datadir;-DSYSCONFDIR=/opt/mysql-5.7.8-grouprepl;-DWITH_SSL=system;-DMYSQL_TCP_PORT=3307;-DMYSQL_UNIX_ADDR=/opt/mysql-5.7.8-grouprepl/mysqld.sock;-DDEFAULT_CHARSET=utf8;-DDEFAULT_COLLATION=utf8_general_ci;-DWITH_DEBUG=1;-DCOMPILATION_COMMENT='Group Replication enabled MySQL-5.7.8';-DOPTIMIZER_TRACE=1;-DWITH_ZLIB=system;-DWITH_VALGRIND=1;-DCMAKE_C_FLAGS=-DHAVE_purify;-DCMAKE_CXX_FLAGS=-DHAVE_purify;-DDOWNLOAD_BOOST=1;-DWITH_BOOST=/home/sh/Sandboxes"

But even you have Oracle Linux 6 x86_64 and even you follow topics: 1. Corosync Guide 2. Getting Started with MySQL Group Replication

You will likely get “Segmentation Fault” after running START GROUP_REPLICATION:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc8ef2700 (LWP 3294)]
0x00007ffff625d532 in _int_malloc () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install corosynclib-1.4.7-2.el6.x86_64
(gdb) bt
#0  0x00007ffff625d532 in _int_malloc () from /lib64/libc.so.6
#1  0x00007ffff625e991 in malloc () from /lib64/libc.so.6
#2  0x00007ffff6acf0bd in operator new(unsigned long) () from /usr/lib64/libstdc++.so.6
#3  0x00007fffc8f5b2ab in Certifier::Certifier (this=0x7fffbc051d20)
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/certifier.cc:209
#4  0x00007fffc8f6bb36 in Certification_handler::initialize (this=0x7fffbc0507e0)
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/handlers/certification_handler.cc:32
#5  0x00007fffc8f634d7 in configure_pipeline (pipeline=0x7fffbc051a80, handler_list=0x7fffbc0506f0, num_handlers=3)
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/pipeline_factory.cc:143
#6  0x00007fffc8f6364d in get_pipeline (pipeline_type=<value optimized out>, pipeline=0x7fffbc051a80)
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/pipeline_factory.cc:30
#7  0x00007fffc8f554b0 in Applier_module::setup_applier_module (this=0x7fffbc051920, pipeline_type=STANDARD_GROUP_REPLICATION_PIPELINE, reset_logs=false, 
    stop_timeout=31536000, group_sidno=2)
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/applier.cc:99
#8  0x00007fffc8f63eca in configure_and_start_applier_module ()
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/plugin.cc:761
#9  0x00007fffc8f657df in plugin_group_replication_start ()
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/plugin.cc:289
#10 0x00000000016927c9 in Group_replication_handler::start (this=0x378f2d0) at /home/full_datadir/mysql-5.7.8-rc/sql/rpl_group_replication.cc:56
#11 0x0000000001692b81 in group_replication_start () at /home/full_datadir/mysql-5.7.8-rc/sql/rpl_group_replication.cc:192
#12 0x000000000149739c in mysql_execute_command (thd=0x7fffbc007180) at /home/full_datadir/mysql-5.7.8-rc/sql/sql_parse.cc:3029
#13 0x000000000149e15b in mysql_parse (thd=0x7fffbc007180, parser_state=0x7fffc8ef14a0) at /home/full_datadir/mysql-5.7.8-rc/sql/sql_parse.cc:5255
#14 0x0000000001492dec in dispatch_command (thd=0x7fffbc007180, com_data=0x7fffc8ef1cd0, command=COM_QUERY)
    at /home/full_datadir/mysql-5.7.8-rc/sql/sql_parse.cc:1272
#15 0x00000000014917f3 in do_command (thd=0x7fffbc007180) at /home/full_datadir/mysql-5.7.8-rc/sql/sql_parse.cc:852
#16 0x00000000015c9042 in handle_connection (arg=0x36c1a50) at /home/full_datadir/mysql-5.7.8-rc/sql/conn_handler/connection_handler_per_thread.cc:300
#17 0x0000000001c56c41 in pfs_spawn_thread (arg=0x37d1990) at /home/full_datadir/mysql-5.7.8-rc/storage/perfschema/pfs.cc:2178
#18 0x00007ffff7bc79d1 in start_thread () from /lib64/libpthread.so.0
#19 0x00007ffff62ccb6d in clone () from /lib64/libc.so.6

For further reading refer to related fresh BUG report -> #78627

It is really cool to help community to improve Open Source projects. So go on and test by your own, report interesting things and ofcourse contact with community members. They are all open to discussion 🙂

Getting SIGABRT -> Assertion `is_binlog_empty()’ failed with MySQL 5.7.7-rc

What about inserting data to MySQL 5.7.7-rc-debug with GTID/binary log enabled. It is OK until you hit disk full condition while inserting data. It will crash on the way and likely you will lose MySQL. Error log indicates related problem:

2015-07-29T07:20:05.865708Z 2 [ERROR] InnoDB: posix_fallocate(): Failed to preallocate data for file ./sls2/sales.ibd, desired size 4194304 bytes. Operating system error number 28. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html
2015-07-29T07:20:05.895529Z 2 [Warning] InnoDB: 1048576 bytes should have been written. Only 557056 bytes written. Retrying again to write the remaining bytes.
2015-07-29T07:20:05.905149Z 2 [Warning] InnoDB: 491520 bytes should have been written. Only 106496 bytes written. Retrying again to write the remaining bytes.
2015-07-29T07:20:05.906292Z 2 [Warning] InnoDB: 385024 bytes should have been written. Only 4096 bytes written. Retrying again to write the remaining bytes.
2015-07-29T07:20:05.907309Z 2 [Warning] InnoDB: 380928 bytes should have been written. Only 4096 bytes written. Retrying again to write the remaining bytes.
2015-07-29T07:20:05.908075Z 2 [Warning] InnoDB: 376832 bytes should have been written. Only 4096 bytes written. Retrying again to write the remaining bytes.
2015-07-29T07:20:05.908727Z 2 [Warning] InnoDB: 372736 bytes should have been written. Only 4096 bytes written. Retrying again to write the remaining bytes.
2015-07-29T07:20:05.909397Z 2 [Warning] InnoDB: 368640 bytes should have been written. Only 4096 bytes written. Retrying again to write the remaining bytes.
2015-07-29T07:20:05.910056Z 2 [ERROR] InnoDB: Error in system call pwrite(). The operating system error number is 28.
2015-07-29T07:20:05.910135Z 2 [ERROR] InnoDB: Error number 28 means 'No space left on device'.
2015-07-29T07:20:05.910156Z 2 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html
2015-07-29T07:20:05.910172Z 2 [Warning] InnoDB: Retry attempts for writing partial data failed.
2015-07-29T07:20:05.910190Z 2 [ERROR] InnoDB: Write to file ./sls2/sales.ibd failed at offset 49283072. 1048576 bytes should have been written, only 684032 were written. Check that your OS and file system support files of  this size. Check also that the disk is not full or a disk quota  exceeded.
2015-07-29T07:20:05.910220Z 2 [Warning] InnoDB: Error while writing 4194304 zeroes to ./sls2/sales.ibd starting at offset 46137344
2015-07-29T07:20:06.001777Z 2 [ERROR] /opt/mysql/bin/mysqld: The table 'sales' is full
mysqld: /root/mysql-5.7.7-rc/sql/binlog.cc:389: virtual void binlog_cache_data::reset(): Assertion `is_binlog_empty()' failed.

Getting: mysqld: /root/mysql-5.7.7-rc/sql/binlog.cc:389: virtual void binlog_cache_data::reset(): Assertion `is_binlog_empty()’ failed.

From GDB:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffde31f700 (LWP 2816)]
0x00007ffff61675d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56    return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);

The next step is to examine backtrace:

(gdb) bt
#0  0x00007ffff61675d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff6168cc8 in __GI_abort () at abort.c:90
#2  0x00007ffff6160546 in __assert_fail_base (fmt=0x7ffff62b0128 "%s%s%s:%u: %s%sAssertion `%s' failed.n%n", 
    assertion=assertion@entry=0x20fdb01 "is_binlog_empty()", file=file@entry=0x20fd990 "/root/mysql-5.7.7-rc/sql/binlog.cc", line=line@entry=389, 
    function=function@entry=0x21030c0 <binlog_cache_data::reset()::__PRETTY_FUNCTION__> "virtual void binlog_cache_data::reset()") at assert.c:92
#3  0x00007ffff61605f2 in __GI___assert_fail (assertion=0x20fdb01 "is_binlog_empty()", file=0x20fd990 "/root/mysql-5.7.7-rc/sql/binlog.cc", line=389, 
    function=0x21030c0 <binlog_cache_data::reset()::__PRETTY_FUNCTION__> "virtual void binlog_cache_data::reset()") at assert.c:101
#4  0x000000000181ed71 in binlog_cache_data::reset (this=0x7fffb400da68) at /root/mysql-5.7.7-rc/sql/binlog.cc:389
#5  0x000000000181f091 in binlog_trx_cache_data::reset (this=0x7fffb400da68) at /root/mysql-5.7.7-rc/sql/binlog.cc:599
#6  0x000000000180776a in binlog_trx_cache_data::truncate (this=0x7fffb400da68, thd=0x7fffb4000c80, all=false) at /root/mysql-5.7.7-rc/sql/binlog.cc:1457
#7  0x00000000018089fa in MYSQL_BIN_LOG::rollback (this=0x2bd78c0 <mysql_bin_log>, thd=0x7fffb4000c80, all=false) at /root/mysql-5.7.7-rc/sql/binlog.cc:1994
#8  0x0000000000f84dfa in ha_rollback_trans (thd=0x7fffb4000c80, all=false) at /root/mysql-5.7.7-rc/sql/handler.cc:1854
#9  0x0000000001682449 in trans_rollback_stmt (thd=0x7fffb4000c80) at /root/mysql-5.7.7-rc/sql/transaction.cc:442
#10 0x000000000158e778 in mysql_execute_command (thd=0x7fffb4000c80) at /root/mysql-5.7.7-rc/sql/sql_parse.cc:4586
#11 0x000000000158ff62 in mysql_parse (thd=0x7fffb4000c80, parser_state=0x7fffde31e5a0) at /root/mysql-5.7.7-rc/sql/sql_parse.cc:5159
#12 0x0000000001585cf1 in dispatch_command (command=COM_QUERY, thd=0x7fffb4000c80, 
    packet=0x7fffb49fd901 "INSERT INTO `sales` VALUES (408505,'1155','40','2001-06-30 14:28:05','253','6399','34','597','0','A','','0'),(408506,'827','5','2005-05-31 14:28:05','378','1169','46','707','0','A','','0'),(408507,'16"..., packet_length=1034768) at /root/mysql-5.7.7-rc/sql/sql_parse.cc:1249
#13 0x0000000001584b21 in do_command (thd=0x7fffb4000c80) at /root/mysql-5.7.7-rc/sql/sql_parse.cc:835
#14 0x00000000016adcff in handle_connection (arg=0x3cbd6b0) at /root/mysql-5.7.7-rc/sql/conn_handler/connection_handler_per_thread.cc:298
#15 0x0000000001cd9905 in pfs_spawn_thread (arg=0x3bfb1f0) at /root/mysql-5.7.7-rc/storage/perfschema/pfs.cc:2147
#16 0x00007ffff7bc6df5 in start_thread (arg=0x7fffde31f700) at pthread_create.c:308
#17 0x00007ffff62281ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

For further reading and finding bt full output refer to related BUG report -> #76825