If you have some very specific problem, you can always try to debug MySQL. To do this you must configure MySQL with the --with-debug or the --with-debug=full option. You can check whether MySQL was compiled with debugging by doing: mysqld --help. If the --debug flag is listed with the options then you have debugging enabled. mysqladmin ver also lists the mysqld version as mysql ... --debug in this case.

If you are using gcc or egcs, the recommended configure line is:

CC=gcc CFLAGS="-O2" CXX=gcc CXXFLAGS="-O2 -felide-constructors \
   -fno-exceptions -fno-rtti" ./configure --prefix=/usr/local/mysql \
   --with-debug --with-extra-charsets=complex

This avoids problems with the libstdc++ library and with C++ exceptions (many compilers have problems with C++ exceptions in threaded code) and compile a MySQL version with support for all character sets.

If you suspect a memory overrun error, you can configure MySQL with --with-debug=full, which installs a memory allocation (SAFEMALLOC) checker. However, running with SAFEMALLOC is quite slow, so if you get performance problems you should start mysqld with the --skip-safemalloc option. This disables the memory overrun checks for each call to malloc() and free().

If mysqld stops crashing when you compile it with --with-debug, you probably have found a compiler bug or a timing bug within MySQL. In this case, you can try to add -g to the CFLAGS and CXXFLAGS variables above and not use --with-debug. If mysqld dies, you can at least attach to it with gdb or use gdb on the core file to find out what happened.

When you configure MySQL for debugging you automatically enable a lot of extra safety check functions that monitor the health of mysqld. If they find something “unexpected,” an entry is written to stderr, which safe_mysqld directs to the error log! This also means that if you are having some unexpected problems with MySQL and are using a source distribution, the first thing you should do is to configure MySQL for debugging! (The second thing is to send mail to a MySQL mailing list and ask for help. See Section 1.7.1.1, “The MySQL Mailing Lists”. Please use the mysqlbug script for all bug reports or questions regarding the MySQL version you are using!

In the Windows MySQL distribution, mysqld.exe is by default compiled with support for trace files.

If the mysqld server doesn't start or if you can cause it to crash quickly, you can try to create a trace file to find the problem.

To do this, you must have a mysqld that has been compiled with debugging support. You can check this by executing mysqld -V. If the version number ends with -debug, it's compiled with support for trace files. (On Windows, the debugging server is named mysqld-debug rather than mysqld as of MySQL 4.1.)

Start the mysqld server with a trace log in /tmp/mysqld.trace on Unix or C:\mysqld.trace on Windows:

shell> mysqld --debug

On Windows, you should also use the --standalone flag to not start mysqld as a service. In a console window, use this command:

C:\> mysqld-debug --debug --standalone

After this, you can use the mysql.exe command-line tool in a second console window to reproduce the problem. You can stop the mysqld server with mysqladmin shutdown.

Note that the trace file become very big! If you want to generate a smaller trace file, you can use debugging options something like this:

mysqld --debug=d,info,error,query,general,where:O,/tmp/mysqld.trace

This only prints information with the most interesting tags to the trace file.

If you make a bug report about this, please only send the lines from the trace file to the appropriate mailing list where something seems to go wrong! If you can't locate the wrong place, you can ftp the trace file, together with a full bug report, to ftp://ftp.mysql.com/pub/mysql/upload/ so that a MySQL developer can take a look a this.

The trace file is made with the DBUG package by Fred Fish. See Section E.3, “The DBUG Package”.

On most systems you can also start mysqld from gdb to get more information if mysqld crashes.

With some older gdb versions on Linux you must use run --one-thread if you want to be able to debug mysqld threads. In this case, you can only have one thread active at a time. We recommend you to upgrade to gdb 5.1 ASAP as thread debugging works much better with this version!

NTPL threads (the new thread library on Linux) may cause problems while running mysqld under gdb. Some symptoms are:

In this case you should set the following environment variable in the shell before starting gdb:

LD_ASSUME_KERNEL=2.4.1
export LD_ASSUME_KERNEL

When running mysqld under gdb, you should disable the stack trace with --skip-stack-trace to be able to catch segfaults within gdb.

In MySQL 4.0.14 and above you should use the --gdb option to mysqld. This installs an interrupt handler for SIGINT (needed to stop mysqld with ^C to set breakpoints) and disable stack tracing and core file handling.

It's very hard to debug MySQL under gdb if you do a lot of new connections the whole time as gdb doesn't free the memory for old threads. You can avoid this problem by starting mysqld with -O thread_cache_size= 'max_connections +1'. In most cases just using -O thread_cache_size=5' helps a lot!

If you want to get a core dump on Linux if mysqld dies with a SIGSEGV signal, you can start mysqld with the --core-file option. This core file can be used to make a backtrace that may help you find out why mysqld died:

shell> gdb mysqld core
gdb>   backtrace full
gdb>   exit

See Section A.4.2, “What to Do If MySQL Keeps Crashing”.

If you are using gdb 4.17.x or above on Linux, you should install a .gdb file, with the following information, in your current directory:

set print sevenbit off
handle SIGUSR1 nostop noprint
handle SIGUSR2 nostop noprint
handle SIGWAITING nostop noprint
handle SIGLWP nostop noprint
handle SIGPIPE nostop
handle SIGALRM nostop
handle SIGHUP nostop
handle SIGTERM nostop noprint

If you have problems debugging threads with gdb, you should download gdb 5.x and try this instead. The new gdb version has very improved thread handling!

Here is an example how to debug mysqld:

shell> gdb /usr/local/libexec/mysqld
gdb> run
...
backtrace full # Do this when mysqld crashes

Include the above output in a mail generated with mysqlbug and mail this to the general MySQL mailing list. See Section 1.7.1.1, “The MySQL Mailing Lists”.

If mysqld hangs you can try to use some system tools like strace or /usr/proc/bin/pstack to examine where mysqld has hung.

strace /tmp/log libexec/mysqld

If you are using the Perl DBI interface, you can turn on debugging information by using the trace method or by setting the DBI_TRACE environment variable.

On some operating systems, the error log contains a stack trace if mysqld dies unexpectedly. You can use this to find out where (and maybe why) mysqld died. See Section 5.10.1, “The Error Log”. To get a stack trace, you must not compile mysqld with the -fomit-frame-pointer option to gcc. See Section E.1.1, “Compiling MySQL for Debugging”.

If the error file contains something like the following:

mysqld got signal 11;
The manual section 'Debugging a MySQL server' tells you how to use a
stack trace and/or the core file to produce a readable backtrace that may
help in finding out why mysqld died
Attempting backtrace. You can use the following information to find out
where mysqld died.  If you see no messages after this, something went
terribly wrong...
stack range sanity check, ok, backtrace follows
0x40077552
0x81281a0
0x8128f47
0x8127be0
0x8127995
0x8104947
0x80ff28f
0x810131b
0x80ee4bc
0x80c3c91
0x80c6b43
0x80c1fd9
0x80c1686

you can find where mysqld died by doing the following:

Note that before starting mysqld with --log you should check all your tables with myisamchk. See Chapter 5, Database Administration.

If mysqld dies or hangs, you should start mysqld with --log. When mysqld dies again, you can examine the end of the log file for the query that killed mysqld.

If you are using --log without a file name, the log is stored in the database directory as host_name.log In most cases it is the last query in the log file that killed mysqld, but if possible you should verify this by restarting mysqld and executing the found query from the mysql command-line tools. If this works, you should also test all complicated queries that didn't complete.

You can also try the command EXPLAIN on all SELECT statements that takes a long time to ensure that mysqld is using indexes properly. See Section 7.2.1, “EXPLAIN Syntax (Get Information About a SELECT)”.

You can find the queries that take a long time to execute by starting mysqld with --log-slow-queries. See Section 5.10.5, “The Slow Query Log”.

If you find the text mysqld restarted in the error log file (normally named hostname.err) you probably have found a query that causes mysqld to fail. If this happens, you should check all your tables with myisamchk (see Chapter 5, Database Administration), and test the queries in the MySQL log files to see if one doesn't work. If you find such a query, try first upgrading to the newest MySQL version. If this doesn't help and you can't find anything in the mysql mail archive, you should report the bug to a MySQL mailing list. The mailing lists are described at http://lists.mysql.com/, which also has links to online list archives.

If you have started mysqld with myisam-recover, MySQL automatically checks and tries to repair MyISAM tables if they are marked as 'not closed properly' or 'crashed'. If this happens, MySQL writes an entry in the hostname.err file 'Warning: Checking table ...' which is followed by Warning: Repairing table if the table needs to be repaired. If you get a lot of these errors, without mysqld having died unexpectedly just before, then something is wrong and needs to be investigated further. See Section 5.2.1, “mysqld Command-Line Options”.

It's not a good sign if mysqld did die unexpectedly, but in this case one shouldn't investigate the Checking table... messages but instead try to find out why mysqld died.

If you get corrupted tables or if mysqld always fails after some update commands, you can test whether this bug is reproducible by doing the following:

You can also use the script mysql_find_rows to just execute some of the update statements if you want to narrow down the problem.