Hunting the core

Core files under Linux

When dealing with MySQL crashes it is very useful to get the core files for further debugging. I have collected all the informations I found about it and wrote it together here:

Find core files

# find $HOME -name "core*"
/home/oli/core.6440

# file core
core:      ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style

See who caused the core file:

# strings core.6440 | head
CORE
CORE
mysqld
/home/mysql/product/mysql-5.1.30/bin/mysqld --defaults-file=/home/mysql/product
...

Soft and hard limit of core files size

(in blocks of 512 byte? -> seems to be 1k blocks!)

# ulimit -Sc
# ulimit -Hc
# ulimit -c unlimited

Getting an setting core file pattern

# cat /proc/sys/kernel/core_pattern
core
# cat /proc/sys/kernel/core_uses_pid
0
# echo "1" > /proc/sys/kernel/core_uses_pid
# echo "/tmp/corefiles/core" > /proc/sys/kernel/core_pattern

Provoke a core dump

# kill -s SIGSEGV $$

or

# kill -11 <pid>

Dump more information

# cat /proc/<pid>/coredump_filter
00000003

The following 4 memory types are supported:

• (bit 0) anonymous private memory
• (bit 1) anonymous shared memory
• (bit 2) file-backed private memory
• (bit 3) file-backed shared memory

# echo 0xF > /proc/<pid>/coredump_filter

Finding the limits of a process

# egrep "Units|core" /proc/<pid>/limits
Limit                     Soft Limit           Hard Limit           Units
Max core file size        1024000              1024000              bytes

Processes switching the user

(for example MySQL)

# cat /proc/sys/fs/suid_dumpable
0

0. (default) This provides the traditional (pre-Linux 2.6.13) behaviour. A core dump will NOT be produced for a process which has changed credentials (by calling seteuid or similar) or whose binary does not have read permission enabled.
1. ("debug") All processes dump core when possible.
2. ("suidsafe") Any binary which normally would not be dumped (see "0" above) is dumped readable by root only.

# echo 2 > /proc/sys/fs/suid_dumpable

# cat /proc/sys/fs/core_setuid_ok
# cat /proc/sys/kernel/core_setuid_ok

Dumping core files with MySQL

# my.cnf
[mysqld]
core-file
user = root

[mysqld_safe]
core-file-size = unlimited

A demo program creating a core dump:

/* create_core.c */

int a (int *p);

int main (void)
{
  int *p = 0;   /* null pointer */
  return a (p);
}

int a (int *p)
{
  int y = *p;
  return y;
}

And how to compile and run it:

# gcc -Wall -g create_core.c -o create_core
# ./create_core
Segmentation fault (core dumped)

Debugging a core with dbg

# gdb create_core core
(gdb) print p
(gdb) backtrace
#0  0x080483ed in a (p=0x0) at null.c:13
#1  0x080483d9 in main () at null.c:7

A suid test case

// testcase.c

#include
#include

int main(int args, char *argv[])
{
  seteuid(getuid());
  struct rlimit rlp;
  struct rlimit newRlp = {RLIM_INFINITY, RLIM_INFINITY};
  getrlimit(RLIMIT_CORE, &rlp);
  setrlimit(RLIMIT_CORE, &newRlp);

  int *badPtr = NULL;
  *badPtr = 5; //forces segfault

  return 0; //shouldn't happen
}

And how to compile an run it:

# g++ -g -Wall testcase.c -o testcase
# sudo su
# chown root testcase
# chmod u+s testcase
# ./testcase
Segmentation fault (core dumped)
# exit
# ./testcase
Segmentation fault   <-- no core was dumped

Why do I NOT get a core dump?

• The core would have been larger than the current limit.
• You don't have the necessary permissions to dump core (directory and file). Notice that core dumps are placed in the dumping process' current directory which could be different from the parent process.
• Verify that the file system is writeable and have sufficient free space.
• If a sub directory named core exist in the working directory no core will be dumped.
• If a file named core already exist but has multiple hard links the kernel will not dump core.
• Verify the permissions on the executable, if the executable has the suid or sgid bit enabled core dumps will by default be disabled. The same will be the case if you have execute permissions but no read permissions on the file.
• Verify that the process has not changed working directory, core size limit, or dumpable flag.
• Some kernel versions cannot dump processes with shared address space (AKA threads). Newer kernel versions can dump such processes but will append the pid to the file name.
• The executable could be in a non-standard format not supporting core dumps. Each executable format must implement a core dump routine.
• The segmentation fault could actually be a kernel Oops, check the system logs for any Oops messages.
• The application called exit() instead of using the core dump handler. (Thanks Shane!)

How does a stack trace and a core dump looks like in MySQL

...
/home/mysql/product/mysql-5.1.30/bin/mysqld(my_print_stacktrace+0x21)[0x84c64f1]
/home/mysql/product/mysql-5.1.30/bin/mysqld(handle_segfault+0x381)[0x81f6261]
[0xb7fe2420]
/home/mysql/product/mysql-5.1.30/bin/mysqld(main+0x668)[0x81fb2d8]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe0)[0xb7e08450]
/home/mysql/product/mysql-5.1.30/bin/mysqld(__fxstat64+0x91)[0x812fc61]
081230 14:44:30 - mysqld got signal 11 ;
...
thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x30000
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
Segmentation fault (core dumped)   <-- MySQL dumped the core.

...
Writing a core file
Segmentation fault   <-- MySQL was not able to dump the core this is empty.

Instead of running MySQL as root, this helped for me:

# echo 2 > /proc/sys/fs/suid_dumpable
# mkdir /tmp/corefiles
# chmod 777 /tmp/corefiles
# echo "/tmp/corefiles/core" > /proc/sys/kernel/core_pattern
# echo 1 > /proc/sys/kernel/core_uses_pid
# mysqld_safe ...

Literature

Open items, more to investigate

I am playing around with making it permanent:

# /etc/sysctl.conf

fs.suid_dumpable = 2
kernel.core_pattern = /tmp/core

Possibly it is more standard compliant to use the sysctl command instead of echo/cat?

If you need some more help on MySQL consider our MySQL consulting services!