Improved support for character set handling was added to MySQL in
version 4.1. The features described here are as implemented in MySQL
4.1.1. (MySQL 4.1.0 has some but not all of these features, and some
of them are implemented differently.)
This chapter discusses the following topics:
What are character sets and collations?
The multiple-level default system
New syntax in MySQL 4.1
Affected functions and operations
Unicode support
The meaning of each individual character set and collation
Character set support currently is included in the
MyISAM, MEMORY
(HEAP), and (as of MySQL 4.1.2)
InnoDB storage engines. The
ISAM storage engine does not include character
set support; there are no plans to change this, because
ISAM is deprecated.
10.1. Character Sets and Collations in General
A character set is a set of symbols and
encodings. A collation is a set of rules
for comparing characters in a character set. Let's make the
distinction clear with an example of an imaginary character set.
Suppose that we have an alphabet with four letters:
‘A’,
‘B’,
‘a’,
‘b’. We give each letter a number:
‘A’ = 0,
‘B’ = 1,
‘a’ = 2,
‘b’ = 3. The letter
‘A’ is a symbol, the number 0 is
the encoding for
‘A’, and the combination of all
four letters and their encodings is a
character set.
Suppose that we want to compare two string values,
‘A’ and
‘B’. The simplest way to do this is
to look at the encodings: 0 for ‘A’
and 1 for ‘B’. Because 0 is less
than 1, we say ‘A’ is less than
‘B’. What we've just done is apply
a collation to our character set. The collation is a set of rules
(only one rule in this case): “compare the
encodings.” We call this simplest of all possible
collations a binary collation.
But what if we want to say that the lowercase and uppercase
letters are equivalent? Then we would have at least two rules: (1)
treat the lowercase letters ‘a’ and
‘b’ as equivalent to
‘A’ and
‘B’; (2) then compare the
encodings. We call this a case-insensitive
collation. It's a little more complex than a binary collation.
In real life, most character sets have many characters: not just
‘A’ and
‘B’ but whole alphabets, sometimes
multiple alphabets or eastern writing systems with thousands of
characters, along with many special symbols and punctuation marks.
Also in real life, most collations have many rules: not just case
insensitivity but also accent insensitivity (an
“accent” is a mark attached to a character as in
German ‘Ö’) and multiple-character
mappings (such as the rule that
‘Ö’ =
‘OE’ in one of the two German
collations).
MySQL 4.1 can do these things for you:
Store strings using a variety of character sets
Compare strings using a variety of collations
Mix strings with different character sets or collations in the
same server, the same database, or even the same table
Allow specification of character set and collation at any
level
In these respects, not only is MySQL 4.1 far more flexible than
MySQL 4.0, it also is far ahead of other DBMSs. However, to use
the new features effectively, you need to learn what character
sets and collations are available, how to change their defaults,
and what the various string operators do with them.
10.2. Character Sets and Collations in MySQL
The MySQL server can support multiple character sets. To list the
available character sets, use the SHOW CHARACTER
SET statement:
mysql> SHOW CHARACTER SET;
+----------+-----------------------------+---------------------+--------+
| Charset | Description | Default collation | Maxlen |
+----------+-----------------------------+---------------------+--------+
| big5 | Big5 Traditional Chinese | big5_chinese_ci | 2 |
| dec8 | DEC West European | dec8_swedish_ci | 1 |
| cp850 | DOS West European | cp850_general_ci | 1 |
| hp8 | HP West European | hp8_english_ci | 1 |
| koi8r | KOI8-R Relcom Russian | koi8r_general_ci | 1 |
| latin1 | cp1252 West European | latin1_swedish_ci | 1 |
| latin2 | ISO 8859-2 Central European | latin2_general_ci | 1 |
| swe7 | 7bit Swedish | swe7_swedish_ci | 1 |
| ascii | US ASCII | ascii_general_ci | 1 |
| ujis | EUC-JP Japanese | ujis_japanese_ci | 3 |
| sjis | Shift-JIS Japanese | sjis_japanese_ci | 2 |
| hebrew | ISO 8859-8 Hebrew | hebrew_general_ci | 1 |
| tis620 | TIS620 Thai | tis620_thai_ci | 1 |
| euckr | EUC-KR Korean | euckr_korean_ci | 2 |
...
Any given character set always has at least one collation. It may
have several collations.
To list the collations for a character set, use the SHOW
COLLATION statement. For example, to see the collations
for the latin1 (“cp1252 West
European”) character set, use this statement to find those
collation names that begin with latin1:
The latin1 collations have the following
meanings:
Collation
Meaning
latin1_german1_ci
German DIN-1
latin1_swedish_ci
Swedish/Finnish
latin1_danish_ci
Danish/Norwegian
latin1_german2_ci
German DIN-2
latin1_bin
Binary according to latin1 encoding
latin1_general_ci
Multilingual (Western European)
latin1_general_cs
Multilingual (ISO Western European), case sensitive
latin1_spanish_ci
Modern Spanish
Collations have these general characteristics:
Two different character sets cannot have the same collation.
Each character set has one collation that is the
default collation. For example, the
default collation for latin1 is
latin1_swedish_ci.
There is a convention for collation names: They start with the
name of the character set with which they are associated, they
usually include a language name, and they end with
_ci (case insensitive),
_cs (case sensitive), or
_bin (binary).
10.3. Determining the Default Character Set and Collation
There are default settings for character sets and collations at
four levels: server, database, table, and connection. The
following description may appear complex, but it has been found in
practice that multiple-level defaulting leads to natural and
obvious results.
10.3.1. Server Character Set and Collation
The MySQL Server has a server character set and a server
collation, neither of which which may be null.
MySQL determines the server character set and server collation
as follows:
According to the option settings in effect when the server
starts
According to the values set at runtime
At the server level, the decision is simple. The server
character set and collation depend initially on the options that
you use when you start mysqld. You can use
--default-character-set for the character set,
and along with it you can add
--default-collation for the collation. If you
don't specify a character set, that is the same as saying
--default-character-set=latin1. If you specify
only a character set (for example, latin1)
but not a collation, that is the same as saying
--default-charset=latin1--default-collation=latin1_swedish_ci because
latin1_swedish_ci is the default collation
for latin1. Therefore, the following three
commands all have the same effect:
One way to change the settings is by recompiling. If you want to
change the default server character set and collation when
building from sources, use: --with-charset and
--with-collation as arguments for
configure. For example:
Both mysqld and configure
verify that the character set/collation combination is valid. If
not, each program displays an error message and terminates.
The current server character set and collation are available as
the values of the character_set_server and
collation_server system variables. These
variables can be changed at runtime.
10.3.2. Database Character Set and Collation
Every database has a database character set and a database
collation, which may not be null. The CREATE
DATABASE and ALTER DATABASE
statements have optional clauses for specifying the database
character set and collation:
CREATE DATABASE db_name
[[DEFAULT] CHARACTER SET charset_name]
[[DEFAULT] COLLATE collation_name]
ALTER DATABASE db_name
[[DEFAULT] CHARACTER SET charset_name]
[[DEFAULT] COLLATE collation_name]
Example:
CREATE DATABASE db_name
DEFAULT CHARACTER SET latin1 COLLATE latin1_swedish_ci;
MySQL chooses the database character set and database collation
thus:
If both CHARACTER SET
X and COLLATE
Y were specified, then
character set X and collation
Y.
If CHARACTER SET
X was specified without
COLLATE, then character set
X and its default collation.
Otherwise, the server character set and server collation.
MySQL's CREATE DATABASE ... DEFAULT CHARACTER SET
... syntax is analogous to the standard SQL
CREATE SCHEMA ... CHARACTER SET ... syntax.
Because of this, it is possible to create databases with
different character sets and collations on the same MySQL
server.
The database character set and collation are used as default
values if the table character set and collation are not
specified in CREATE TABLE statements. They
have no other purpose.
The character set and collation for the default database are
available as the values of the
character_set_database and
collation_database system variables. The
server sets these variables whenever the default database
changes. If there is no default database, the variables have the
same value as the corresponding server-level variables,
character_set_server and
collation_server.
10.3.3. Table Character Set and Collation
Every table has a table character set and a table collation,
which may not be null. The CREATE TABLE and
ALTER TABLE statements have optional clauses
for specifying the table character set and collation:
CREATE TABLE tbl_name (column_list)
[DEFAULT CHARACTER SET charset_name [COLLATE collation_name]]
ALTER TABLE tbl_name
[DEFAULT CHARACTER SET charset_name] [COLLATE collation_name]
Example:
CREATE TABLE t1 ( ... )
DEFAULT CHARACTER SET latin1 COLLATE latin1_danish_ci;
MySQL chooses the table character set and collation in the
following manner:
If both CHARACTER SET
X and COLLATE
Y were specified, then
character set X and collation
Y.
If CHARACTER SET
X was specified without
COLLATE, then character set
X and its default collation.
Otherwise, the database character set and collation.
The table character set and collation are used as default values
if the column character set and collation are not specified in
individual column definitions. The table character set and
collation are MySQL extensions; there are no such things in
standard SQL.
10.3.4. Column Character Set and Collation
Every “character” column (that is, a column of type
CHAR, VARCHAR, or
TEXT) has a column character set and a column
collation, which may not be null. Column definition syntax has
optional clauses for specifying the column character set and
collation:
CREATE TABLE Table1
(
column1 VARCHAR(5) CHARACTER SET latin1 COLLATE latin1_german1_ci
);
MySQL chooses the column character set and collation in the
following manner:
If both CHARACTER SET
X and COLLATE
Y were specified, then
character set X and collation
Y are used.
If CHARACTER SET
X was specified without
COLLATE, then character set
X and its default collation are
used.
Otherwise, the table character set and collation are used.
The CHARACTER SET and
COLLATE clauses are standard SQL.
10.3.5. Examples of Character Set and Collation Assignment
The following examples show how MySQL determines default
character set and collation values.
Example 1: Table and Column
Definition
CREATE TABLE t1
(
c1 CHAR(10) CHARACTER SET latin1 COLLATE latin1_german1_ci
) DEFAULT CHARACTER SET latin2 COLLATE latin2_bin;
Here we have a column with a latin1 character
set and a latin1_german1_ci collation. The
definition is explicit, so that's straightforward. Notice that
there is no problem with storing a latin1
column in a latin2 table.
Example 2: Table and Column
Definition
CREATE TABLE t1
(
c1 CHAR(10) CHARACTER SET latin1
) DEFAULT CHARACTER SET latin1 COLLATE latin1_danish_ci;
This time we have a column with a latin1
character set and a default collation. Although it might seem
natural, the default collation is not taken from the table
level. Instead, because the default collation for
latin1 is always
latin1_swedish_ci, column
c1 has a collation of
latin1_swedish_ci (not
latin1_danish_ci).
Example 3: Table and Column
Definition
CREATE TABLE t1
(
c1 CHAR(10)
) DEFAULT CHARACTER SET latin1 COLLATE latin1_danish_ci;
We have a column with a default character set and a default
collation. In this circumstance, MySQL looks up to the table
level for inspiration in determining the column character set
and collation. So, the character set for column
c1 is latin1 and its
collation is latin1_danish_ci.
Example 4: Database, Table, and Column
Definition
CREATE DATABASE d1
DEFAULT CHARACTER SET latin2 COLLATE latin2_czech_ci;
USE d1;
CREATE TABLE t1
(
c1 CHAR(10)
);
We create a column without specifying its character set and
collation. We're also not specifying a character set and a
collation at the table level. In this circumstance, MySQL looks
up to the database level for inspiration. (The database's
settings become the table's settings, and thereafter become the
column's setting.) So, the character set for column
c1 is latin2 and its
collation is latin2_czech_ci.
10.3.6. Connection Character Sets and Collations
Several character set and collation system variables relate to a
client's interaction with the server. Some of these have been
mentioned in earlier sections:
The server character set and collation are available as the
values of the character_set_server and
collation_server variables.
The character set and collation of the default database are
available as the values of the
character_set_database and
collation_database variables.
Additional character set and collation variables are involved in
handling traffic for the connection between a client and the
server. Every client has connection-related character set and
collation variables.
Consider what a “connection” is: It's what you make
when you connect to the server. The client sends SQL statements,
such as queries, over the connection to the server. The server
sends responses, such as result sets, over the connection back
to the client. This leads to several questions about character
set and collation handling for client connections, each of which
can be answered in terms of system variables:
What character set is the query in when it leaves the
client?
The server takes the character_set_client
variable to be the character set in which queries are sent
by the client.
What character set should the server translate a query to
after receiving it?
For this, character_set_connection and
collation_connection are used by the
server. It converts queries sent by the client from
character_set_client to
character_set_connection (except for
string literals that have an introducer such as
_latin1 or _utf8).
collation_connection is important for
comparisons of literal strings. For comparisons of strings
with column values, it does not matter because columns have
a higher collation precedence.
What character set should the server translate to before
shipping result sets or error messages back to the client?
The character_set_results variable
indicates the character set in which the server returns
query results to the client. This includes result data such
as column values, and result metadata such as column names.
You can fine-tune the settings for these variables, or you can
depend on the defaults (in which case, you can skip this
section).
There are two statements that affect the connection character
sets:
SET NAMES 'charset_name'
SET CHARACTER SET charset_name
SET NAMES indicates what is in the SQL
statements that the client sends. Thus, SET NAMES
'cp1251' tells the server “future incoming
messages from this client are in character set
cp1251.” It also specifies the
character set for results that the server sends back to the
client. (For example, it indicates what character set column
values are if you use a SELECT statement.)
A SET NAMES 'x'
statement is equivalent to these three statements:
mysql> SET character_set_client = x;
mysql> SET character_set_results = x;
mysql> SET character_set_connection = x;
Setting character_set_connection to
x also sets
collation_connection to the default collation
for x.
SET CHARACTER SET is similar but sets the
connection character set and collation to be those of the
default database. A SET CHARACTER SET x
statement is equivalent to these three statements:
mysql> SET character_set_client = x;
mysql> SET character_set_results = x;
mysql> SET collation_connection = @@collation_database;
When a client connects, it sends to the server the name of the
character set that it wants to use. The server sets the
character_set_client,
character_set_results, and
character_set_connection variables to that
character set. (In effect, the server performs a SET
NAMES operation using the character set.)
With the mysql client, it is not necessary to
execute SET NAMES every time you start up if
you want to use a character set different from the default. You
can add the --default-character-set option
setting to your mysql statement line, or in
your option file. For example, the following option file setting
changes the three character set variables set to
koi8r each time you run
mysql:
[mysql]
default-character-set=koi8r
Example: Suppose that column1 is defined as
CHAR(5) CHARACTER SET latin2. If you do not
say SET NAMES or SET CHARACTER
SET, then for SELECT column1 FROM
t, the server sends back all the values for
column1 using the character set that the
client specified when it connected. On the other hand, if you
say SET NAMES 'latin1' or SET
CHARACTER SET latin1, then just before sending results
back, the server converts the latin2 values
to latin1. Conversion may be lossy if there
are characters that are not in both character sets.
If you do not want the server to perform any conversion, set
character_set_results to
NULL:
mysql> SET character_set_results = NULL;
10.3.7. Character String Literal Character Set and Collation
Every character string literal has a character set and a
collation, which may not be null.
A character string literal may have an optional character set
introducer and COLLATE clause:
For the simple statement SELECT
'string', the string has
the character set and collation defined by the
character_set_connection and
collation_connection system variables.
The _charset_name
expression is formally called an
introducer. It tells the parser, “the
string that is about to follow uses character set
X.” Because this has confused
people in the past, we emphasize that an introducer does not
cause any conversion; it is strictly a signal that does not
change the string's value. An introducer is also legal before
standard hex literal and numeric hex literal notation
(x'literal' and
0xnnnn), and
before ? (parameter substitution when using
prepared statements within a programming language interface).
MySQL determines a literal's character set and collation thus:
If both _X and COLLATE
Y were specified, then
character set X and collation
Y are used.
If _X is specified but
COLLATE is not specified, then character
set X and its default collation
are used.
Otherwise, the character set and collation given by the
character_set_connection and
collation_connection system variables are
used.
Examples:
A string with latin1 character set and
latin1_german1_ci collation:
SELECT _latin1'Müller' COLLATE latin1_german1_ci;
A string with latin1 character set and
its default collation (that is,
latin1_swedish_ci):
SELECT _latin1'Müller';
A string with the connection default character set and
collation:
SELECT 'Müller';
Character set introducers and the COLLATE
clause are implemented according to standard SQL specifications.
10.3.8. Using COLLATE in SQL Statements
With the COLLATE clause, you can override
whatever the default collation is for a comparison.
COLLATE may be used in various parts of SQL
statements. Here are some examples:
With ORDER BY:
SELECT k
FROM t1
ORDER BY k COLLATE latin1_german2_ci;
With AS:
SELECT k COLLATE latin1_german2_ci AS k1
FROM t1
ORDER BY k1;
With GROUP BY:
SELECT k
FROM t1
GROUP BY k COLLATE latin1_german2_ci;
With aggregate functions:
SELECT MAX(k COLLATE latin1_german2_ci)
FROM t1;
With DISTINCT:
SELECT DISTINCT k COLLATE latin1_german2_ci
FROM t1;
With WHERE:
SELECT *
FROM t1
WHERE _latin1 'Müller' COLLATE latin1_german2_ci = k;
SELECT *
FROM t1
WHERE k LIKE _latin1 'Müller' COLLATE latin1_german2_ci;
With HAVING:
SELECT k
FROM t1
GROUP BY k
HAVING k = _latin1 'Müller' COLLATE latin1_german2_ci;
10.3.9. COLLATE Clause Precedence
The COLLATE clause has high precedence
(higher than ||), so the following two
expressions are equivalent:
x || y COLLATE z
x || (y COLLATE z)
10.3.10. BINARY Operator
The BINARY operator is a shorthand for a
COLLATE clause. BINARY
'x' is equivalent to
'x' COLLATE
y, where
y is the name of the binary collation
for the character set of 'x'. Every
character set has a binary collation. For example, the binary
collation for the latin1 character set is
latin1_bin, so if the column
a is of character set
latin1, the following two statements have the
same effect:
SELECT * FROM t1 ORDER BY BINARY a;
SELECT * FROM t1 ORDER BY a COLLATE latin1_bin;
10.3.11. Some Special Cases Where the Collation Determination Is Tricky
In the great majority of queries, it is obvious what collation
MySQL uses to resolve a comparison operation. For example, in
the following cases, it should be clear that the collation is
“the column collation of column
x”:
SELECT x FROM T ORDER BY x;
SELECT x FROM T WHERE x = x;
SELECT DISTINCT x FROM T;
However, when multiple operands are involved, there can be
ambiguity. For example:
SELECT x FROM T WHERE x = 'Y';
Should this query use the collation of the column
x, or of the string literal
'Y'?
Standard SQL resolves such questions using what used to be
called “coercibility” rules. Basically, this means:
Since both x and 'Y' have
collations, whose collation takes precedence? This can be
difficult to resolve, but the following rules take care of most
situations:
An explicit COLLATE clause has a
coercibility of 0. (Not coercible at all.)
The concatenation of two strings with different collations
has a coercibility of 1.
A column's collation has a coercibility of 2.
A “system constant” (the string returned by
functions such as USER() or
VERSION()) has a coercibility of 3.
A literal's collation has a coercibility of 4.
NULL or an expression that is derived
from NULL has a coercibility of 5.
The preceding coercibility values are current as of MySQL
4.1.11. See the note later in this section for additional
version-related information.
Those rules resolve ambiguities thus:
Use the collation with the lowest coercibility value.
If both sides have the same coercibility, then it is an
error if the collations aren't the same.
Examples:
column1 = 'A'
Use collation of column1
column1 = 'A' COLLATE x
Use collation of 'A'
column1 COLLATE x = 'A' COLLATE y
Error
The COERCIBILITY() function can be used to
determine the coercibility of a string expression:
Before MySQL 4.1.11, there is no system constant or ignorable
coercibility. Functions such as USER() have a
coercibility of 2 rather than 3, and literals have a
coercibility of 3 rather than 4.
10.3.12. Collations Must Be for the Right Character Set
Recall that each character set has one or more collations, and
each collation is associated with one and only one character
set. Therefore, the following statement causes an error message
because the latin2_bin collation is not legal
with the latin1 character set:
mysql> SELECT _latin1 'x' COLLATE latin2_bin;
ERROR 1251: COLLATION 'latin2_bin' is not valid
for CHARACTER SET 'latin1'
In some cases, expressions that worked before MySQL 4.1 fail in
early versions of MySQL 4.1 if you do not take character set and
collation into account. For example, before 4.1, this statement
works as is:
The statement also works as is in MySQL 4.1 as of 4.1.8: In
MySQL 4.1, usernames are stored using the
utf8 character set (see
Section 10.6, “UTF8 for Metadata”). The literal string
'@' has the server character set
(latin1 by default). Although the character
sets are different, MySQL can coerce the
latin1 string to the character set (and
collation) of USER() without data loss. It
does so, performs the substring operation, and returns a result
that has a character set of utf8.
However, in versions of MySQL 4.1 before 4.1.8, the statement
fails:
mysql> SELECT SUBSTRING_INDEX(USER(),'@',1);
ERROR 1267 (HY000): Illegal mix of collations
(utf8_general_ci,IMPLICIT) and (latin1_swedish_ci,COERCIBLE)
for operation 'substr_index'
This happens because the automatic character set conversion of
'@' does not occur and the string operands
have different character sets (and thus different collations):
Another way is to change the connection character set and
collation to utf8. You can do that with
SET NAMES 'utf8' or by setting the
character_set_connection and
collation_connection system variables
directly.
10.3.13. An Example of the Effect of Collation
Suppose that column X in table
T has these latin1 column
values:
Muffler
Müller
MX Systems
MySQL
And suppose that the column values are retrieved using the
following statement:
SELECT X FROM T ORDER BY X COLLATE collation_name;
The resulting order of the values for different collations is
shown in this table:
latin1_swedish_ci
latin1_german1_ci
latin1_german2_ci
Muffler
Muffler
Müller
MX Systems
Müller
Muffler
Müller
MX Systems
MX Systems
MySQL
MySQL
MySQL
The table is an example that shows what the effect would be if
we used different collations in an ORDER BY
clause. The character that causes the different sort orders in
this example is the U with two dots over it
(ü), which the Germans call "U-umlaut".
The first column shows the result of the
SELECT using the Swedish/Finnish
collating rule, which says that U-umlaut sorts with Y.
The second column shows the result of the
SELECT using the German DIN-1 rule, which
says that U-umlaut sorts with U.
The third column shows the result of the
SELECT using the German DIN-2 rule, which
says that U-umlaut sorts with UE.
10.4. Operations Affected by Character Set Support
This section describes operations that take character set
information into account as of MySQL 4.1.
10.4.1. Result Strings
MySQL has many operators and functions that return a string.
This section answers the question: What is the character set and
collation of such a string?
For simple functions that take string input and return a string
result as output, the output's character set and collation are
the same as those of the principal input value. For example,
UPPER(X) returns a
string whose character string and collation are the same as that
of X. The same applies for
INSTR(), LCASE(),
LOWER(), LTRIM(),
MID(), REPEAT(),
REPLACE(), REVERSE(),
RIGHT(), RPAD(),
RTRIM(), SOUNDEX(),
SUBSTRING(), TRIM(),
UCASE(), and UPPER().
(Also note: The REPLACE() function, unlike
all other functions, always ignores the collation of the string
input and performs a case-sensitive comparison.)
For operations that combine multiple string inputs and return a
single string output, the “aggregation rules” of
standard SQL apply:
If an explicit COLLATE
X occurs, then use
X.
If explicit COLLATE
X and COLLATE
Y occur, then raise an
error.
Otherwise, if all collations are
X, then use
X.
Otherwise, the result has no collation.
For example, with CASE ... WHEN a THEN b WHEN b THEN c
COLLATE X END, the
resulting collation is X. The same
applies for CASE, UNION,
||, CONCAT(),
ELT(), GREATEST(),
IF(), and LEAST().
For operations that convert to character data, the character set
and collation of the strings that result from the operations are
defined by the character_set_connection and
collation_connection system variables. This
applies to CAST(), CHAR(),
CONV(), FORMAT(),
HEX(), and SPACE().
10.4.2. CONVERT()
CONVERT() provides a way to convert data
between different character sets. The syntax is:
CONVERT(expr USING transcoding_name)
In MySQL, transcoding names are the same as the corresponding
character set names.
Examples:
SELECT CONVERT(_latin1'Müller' USING utf8);
INSERT INTO utf8table (utf8column)
SELECT CONVERT(latin1field USING utf8) FROM latin1table;
CONVERT(... USING ...) is implemented
according to the standard SQL specification.
10.4.3. CAST()
You may also use CAST() to convert a string
to a different character set. The syntax is:
CAST(character_string AS character_data_type CHARACTER SET charset_name)
Example:
SELECT CAST(_latin1'test' AS CHAR CHARACTER SET utf8);
If you use CAST() without specifying
CHARACTER SET, the resulting character set
and collation are defined by the
character_set_connection and
collation_connection system variables. If you
use CAST() with CHARACTER SET
X, then the resulting character set and collation are
X and the default collation of
X.
You may not use a COLLATE clause inside a
CAST(), but you may use it outside. That is,
CAST(... COLLATE ...) is illegal, but
CAST(...) COLLATE ... is legal.
Example:
SELECT CAST(_latin1'test' AS CHAR CHARACTER SET utf8) COLLATE utf8_bin;
10.4.4. SHOW Statements
Several SHOW statements are added or modified
in MySQL 4.1 to provide additional character set information.
SHOW CHARACTER SET, SHOW
COLLATION, and SHOW CREATE DATABASE
are new. SHOW CREATE TABLE and SHOW
COLUMNS are modified.
The SHOW CHARACTER SET command shows all
available character sets. It takes an optional
LIKE clause that indicates which character
set names to match. For example:
mysql> SHOW CHARACTER SET LIKE 'latin%';
+---------+-----------------------------+-------------------+--------+
| Charset | Description | Default collation | Maxlen |
+---------+-----------------------------+-------------------+--------+
| latin1 | cp1252 West European | latin1_swedish_ci | 1 |
| latin2 | ISO 8859-2 Central European | latin2_general_ci | 1 |
| latin5 | ISO 8859-9 Turkish | latin5_turkish_ci | 1 |
| latin7 | ISO 8859-13 Baltic | latin7_general_ci | 1 |
+---------+-----------------------------+-------------------+--------+
The output from SHOW COLLATION includes all
available character sets. It takes an optional
LIKE clause that indicates which collation
names to match. For example:
SHOW CREATE DATABASE displays the
CREATE DATABASE statement that creates a
given database. The result includes all database options.
DEFAULT CHARACTER SET and
COLLATE are supported. All database options
are stored in a text file named db.opt that
can be found in the database directory.
mysql> SHOW CREATE DATABASE test;
+----------+-----------------------------------------------------------------+
| Database | Create Database |
+----------+-----------------------------------------------------------------+
| test | CREATE DATABASE `test` /*!40100 DEFAULT CHARACTER SET latin1 */ |
+----------+-----------------------------------------------------------------+
SHOW CREATE TABLE is similar, but displays
the CREATE TABLE statement to create a given
table. The column definitions indicate any character set
specifications, and the table options include character set
information.
The SHOW COLUMNS statement displays the
collations of a table's columns when invoked as SHOW
FULL COLUMNS. Columns with CHAR,
VARCHAR, or TEXT data
types have non-NULL collations. Numeric and
other non-character types have NULL
collations. For example:
mysql> SHOW FULL COLUMNS FROM person\G
*************************** 1. row ***************************
Field: id
Type: smallint(5) unsigned
Collation: NULL
Null: NO
Key: PRI
Default: NULL
Extra: auto_increment
Privileges: select,insert,update,references
Comment:
*************************** 2. row ***************************
Field: name
Type: char(60)
Collation: latin1_swedish_ci
Null: NO
Key:
Default:
Extra:
Privileges: select,insert,update,references
Comment:
The character set is not part of the display. (The character set
name is implied by the collation name.)
As of MySQL version 4.1, there are two new character sets for
storing Unicode data:
ucs2, the UCS-2 Unicode character set.
utf8, the UTF8 encoding of the Unicode
character set.
In UCS-2 (binary Unicode representation), every character is
represented by a two-byte Unicode code with the most significant
byte first. For example: "LATIN CAPITAL LETTER A" has the code
0x0041 and it's stored as a two-byte sequence: 0x00 0x41.
"CYRILLIC SMALL LETTER YERU" (Unicode 0x044B) is stored as a
two-byte sequence: 0x04 0x4B. For Unicode characters and their
codes, please refer to the
Unicode Home Page.
A restriction in MySQL 4.1 is that UCS-2 cannot be used as a
client character set. That means that SET NAMES
'ucs2' does not work.
The UTF8 character set (transform Unicode representation) is an
alternative way to store Unicode data. It is implemented according
to RFC 3629. The idea of the UTF8 character set is that various
Unicode characters are encoded using byte sequences of different
lengths:
Basic Latin letters, digits, and punctuation signs use one
byte.
Most European and Middle East script letters fit into a
two-byte sequence: extended Latin letters (with tilde, macron,
acute, grave and other accents), Cyrillic, Greek, Armenian,
Hebrew, Arabic, Syriac, and others.
Korean, Chinese, and Japanese ideographs use three-byte
sequences.
RFC 3629 describes encoding sequences that take from one to four
bytes. Currently, MySQL UTF8 support does not include four-byte
sequences. (An older standard for UTF8 encoding is given by RFC
2279, which describes UTF8 sequences that take from one to six
bytes. RFC 3629 renders RFC 2279 obsolete; for this reason,
sequences with five and six bytes are no longer used.)
Tip: To save space with UTF8, use
VARCHAR instead of CHAR.
Otherwise, MySQL has to reserve 30 bytes for a CHAR(10)
CHARACTER SET utf8 column, because this is the maximum
possible length.
10.6. UTF8 for Metadata
Metadata is “the data about the
data”. Anything that describes the
database — as opposed to being the
contents of the database — is metadata.
Thus column names, database names, usernames, version names, and
most of the string results from SHOW are
metadata.
Representation of metadata must satisfy these requirements:
All metadata must be in the same character set. Otherwise,
SHOW wouldn't work properly because
different rows in the same column would be in different
character sets.
Metadata must include all characters in all languages.
Otherwise, users wouldn't be able to name columns and tables
in their own languages.
In order to satisfy both requirements, MySQL stores metadata in a
Unicode character set, namely UTF8. This does not cause any
disruption if you never use accented characters. But if you do,
you should be aware that metadata is in UTF8.
This means that the return values of the
USER(), CURRENT_USER(),
DATABASE(), and VERSION()
functions have the UTF8 character set by default, as do synonyms
such as SESSION_USER() and
SYSTEM_USER().
The server sets the character_set_system system
variable to the name of the metadata character set:
mysql> SHOW VARIABLES LIKE 'character_set_system';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| character_set_system | utf8 |
+----------------------+-------+
Storage of metadata using Unicode does not
mean that the headers of columns and the results of
DESCRIBE functions are in the
character_set_system character set by default.
When you use SELECT column1 FROM t, the name
column1 itself is returned from the server to
the client in the character set as determined by the SET
NAMES statement. More specifically, the character set
used is determined by the value of the
character_set_results system variable. If this
variable is set to NULL, no conversion is
performed and the server returns metadata using its original
character set (the set indicated by
character_set_system).
If you want the server to pass metadata results back in a non-UTF8
character set, then use SET NAMES to force the
server to perform character set conversion (see
Section 10.3.6, “Connection Character Sets and Collations”), or else set the client to
do the conversion. It is always more efficient to set the client
to do the conversion, but this option is not available for many
clients until late in the MySQL 4.x product cycle.
If you are using (for example) the USER()
function for comparison or assignment within a single statement,
don't worry. MySQL performs some automatic conversion for you.
SELECT * FROM Table1 WHERE USER() = latin1_column;
This works because the contents of
latin1_column are automatically converted to
UTF8 before the comparison.
INSERT INTO Table1 (latin1_column) SELECT USER();
This works because the contents of USER() are
automatically converted to latin1 before the
assignment. Automatic conversion is not fully implemented yet, but
should work correctly in a later version.
Although automatic conversion is not in the SQL standard, the SQL
standard document does say that every character set is (in terms
of supported characters) a “subset” of Unicode. Since
it is a well-known principle that “what applies to a
superset can apply to a subset”, we believe that a
collation for Unicode can apply for comparisons with non-Unicode
strings.
Note: Beginning with MySQL 4.1.1,
the errmsg.txt files all use UTF8. Conversion
to the client character set is automatic, as with metadata.
10.7. Compatibility with Other DBMSs
For MaxDB compatibility these two statements are the same:
CREATE TABLE t1 (f1 CHAR(n) UNICODE);
CREATE TABLE t1 (f1 CHAR(n) CHARACTER SET ucs2);
10.8. New Character Set Configuration File Format
In MySQL 4.1, character set configuration is stored in XML files,
one file per character set. In previous versions, this information
was stored in .conf files.
10.9. National Character Set
Before MySQL 4.1, NCHAR and
CHAR were synonymous. ANSI SQL defines
NCHAR or NATIONAL CHAR as a
way to indicate that a CHAR column should use
some predefined character set. MySQL 4.1 and up uses
utf8 as that predefined character set. For
example, these column type declarations are equivalent:
CHAR(10) CHARACTER SET utf8
NATIONAL CHARACTER(10)
NCHAR(10)
As are these:
VARCHAR(10) CHARACTER SET utf8
NATIONAL VARCHAR(10)
NCHAR VARCHAR(10)
NATIONAL CHARACTER VARYING(10)
NATIONAL CHAR VARYING(10)
You can use
N'literal' to create
a string in the national character set. These two statements are
equivalent:
What about upgrading from older versions of MySQL? MySQL 4.1 is
almost upward compatible with MySQL 4.0 and earlier for the simple
reason that almost all the features are new, so there's nothing in
earlier versions to conflict with. However, there are some
differences and a few things to be aware of.
It is important to note that the “MySQL 4.0 character
set” contains both character set and collation information
in one single entity. Beginning in MySQL 4.1, character sets and
collations are separate entities. Though each collation
corresponds to a particular character set, the two are not bundled
together.
There is a special treatment of national character sets in MySQL
4.1. NCHAR is not the same as
CHAR, and N'...' literals
are not the same as '...' literals.
Finally, there is a different file format for storing information
about character sets and collations. Make sure that you have
reinstalled the /share/mysql/charsets/
directory containing the new configuration files.
If you want to start mysqld from a 4.1.x
distribution with data created by MySQL 4.0, you should start the
server with the same character set and collation. In this case,
you won't need to reindex your data.
10.10.1. 4.0 Character Sets and Corresponding 4.1 Character Set/Collation Pairs
ID
4.0 Character Set
4.1 Character Set
4.1 Collation
1
big5
big5
big5_chinese_ci
2
czech
latin2
latin2_czech_ci
3
dec8
dec8
dec8_swedish_ci
4
dos
cp850
cp850_general_ci
5
german1
latin1
latin1_german1_ci
6
hp8
hp8
hp8_english_ci
7
koi8_ru
koi8r
koi8r_general_ci
8
latin1
latin1
latin1_swedish_ci
9
latin2
latin2
latin2_general_ci
10
swe7
swe7
swe7_swedish_ci
11
usa7
ascii
ascii_general_ci
12
ujis
ujis
ujis_japanese_ci
13
sjis
sjis
sjis_japanese_ci
14
cp1251
cp1251
cp1251_bulgarian_ci
15
danish
latin1
latin1_danish_ci
16
hebrew
hebrew
hebrew_general_ci
17
win1251
(removed)
(removed)
18
tis620
tis620
tis620_thai_ci
19
euc_kr
euckr
euckr_korean_ci
20
estonia
latin7
latin7_estonian_ci
21
hungarian
latin2
latin2_hungarian_ci
22
koi8_ukr
koi8u
koi8u_ukrainian_ci
23
win1251ukr
cp1251
cp1251_ukrainian_ci
24
gb2312
gb2312
gb2312_chinese_ci
25
greek
greek
greek_general_ci
26
win1250
cp1250
cp1250_general_ci
27
croat
latin2
latin2_croatian_ci
28
gbk
gbk
gbk_chinese_ci
29
cp1257
cp1257
cp1257_lithuanian_ci
30
latin5
latin5
latin5_turkish_ci
31
latin1_de
latin1
latin1_german2_ci
10.10.2. Converting 4.0 Character Columns to 4.1 Format
Normally, the server runs using the latin1
character set by default. If you have been storing column data
that actually is in some other character set that the 4.1 server
supports directly, you can convert the column. However, you
should avoid trying to convert directly from
latin1 to the "real" character set. This may
result in data loss. Instead, convert the column to a binary
column type, and then from the binary type to a non-binary type
with the desired character set. Conversion to and from binary
involves no attempt at character value conversion and preserves
your data intact. For example, suppose that you have a 4.0 table
with three columns that are used to store values represented in
latin1, latin2, and
utf8:
For MySQL 4.1, you want to convert this table to leave
latin1_col alone but change the
latin2_col and utf8_col
columns to have character sets of latin2 and
utf8. Before upgrading to 4.1, back up your
table, then convert the columns as follows:
ALTER TABLE t MODIFY latin2_col BINARY(100);
ALTER TABLE t MODIFY utf8_col BINARY(150);
Then, after upgrading to 4.1, complete the conversion by issuing
these statements:
ALTER TABLE t MODIFY latin2_col CHAR(100) CHARACTER SET latin2;
ALTER TABLE t MODIFY utf8_col CHAR(150) CHARACTER SET utf8;
The first two statements “remove” the character set
information from the latin2_col and
utf8_col columns. The second two statements
assign the proper character sets to the two columns.
If you like, you can combine the to-binary conversions and
from-binary conversions into single statements. In MySQL 4.0, do
this:
ALTER TABLE t
MODIFY latin2_col BINARY(100),
MODIFY utf8_col BINARY(150);
After upgrading to 4.1, do this:
ALTER TABLE t
MODIFY latin2_col CHAR(100) CHARACTER SET latin2,
MODIFY utf8_col CHAR(150) CHARACTER SET utf8;
If you can ensure that the tables will not otherwise be modified
before you perform the character set conversion, you can issue
all of the ALTER TABLE statements after
upgrading to MySQL 4.1.
If you specified attributes when creating a column initially,
you should also specify them when altering the table with
ALTER TABLE. For example, if you specified
NOT NULL and an explicit
DEFAULT value, you should also provide them
in the ALTER TABLE statement. Otherwise, the
resulting column definition will not include those attributes.
10.11. Character Sets and Collations That MySQL Supports
MySQL supports 70+ collations for 30+ character sets. The
character sets and their default collations are displayed by the
SHOW CHARACTER SET statement:
mysql> SHOW CHARACTER SET;
+----------+-----------------------------+---------------------+
| Charset | Description | Default collation |
+----------+-----------------------------+---------------------+
| big5 | Big5 Traditional Chinese | big5_chinese_ci |
| dec8 | DEC West European | dec8_swedish_ci |
| cp850 | DOS West European | cp850_general_ci |
| hp8 | HP West European | hp8_english_ci |
| koi8r | KOI8-R Relcom Russian | koi8r_general_ci |
| latin1 | cp1252 West European | latin1_swedish_ci |
| latin2 | ISO 8859-2 Central European | latin2_general_ci |
| swe7 | 7bit Swedish | swe7_swedish_ci |
| ascii | US ASCII | ascii_general_ci |
| ujis | EUC-JP Japanese | ujis_japanese_ci |
| sjis | Shift-JIS Japanese | sjis_japanese_ci |
| hebrew | ISO 8859-8 Hebrew | hebrew_general_ci |
| tis620 | TIS620 Thai | tis620_thai_ci |
| euckr | EUC-KR Korean | euckr_korean_ci |
| koi8u | KOI8-U Ukrainian | koi8u_general_ci |
| gb2312 | GB2312 Simplified Chinese | gb2312_chinese_ci |
| greek | ISO 8859-7 Greek | greek_general_ci |
| cp1250 | Windows Central European | cp1250_general_ci |
| gbk | GBK Simplified Chinese | gbk_chinese_ci |
| latin5 | ISO 8859-9 Turkish | latin5_turkish_ci |
| armscii8 | ARMSCII-8 Armenian | armscii8_general_ci |
| utf8 | UTF-8 Unicode | utf8_general_ci |
| ucs2 | UCS-2 Unicode | ucs2_general_ci |
| cp866 | DOS Russian | cp866_general_ci |
| keybcs2 | DOS Kamenicky Czech-Slovak | keybcs2_general_ci |
| macce | Mac Central European | macce_general_ci |
| macroman | Mac West European | macroman_general_ci |
| cp852 | DOS Central European | cp852_general_ci |
| latin7 | ISO 8859-13 Baltic | latin7_general_ci |
| cp1251 | Windows Cyrillic | cp1251_general_ci |
| cp1256 | Windows Arabic | cp1256_general_ci |
| cp1257 | Windows Baltic | cp1257_general_ci |
| binary | Binary pseudo charset | binary |
| geostd8 | GEOSTD8 Georgian | geostd8_general_ci |
| cp932 | SJIS for Windows Japanese | cp932_japanese_ci |
| eucjpms | UJIS for Windows Japanese | eucjpms_japanese_ci |
+----------+-----------------------------+---------------------+
10.11.1. Unicode Character Sets
MySQL has two Unicode character sets. You can store text in
about 650 languages using these character sets.
ucs2 (UCS-2 Unicode) collations:
ucs2_bin
ucs2_czech_ci
ucs2_danish_ci
ucs2_estonian_ci
ucs2_general_ci (default)
ucs2_icelandic_ci
ucs2_latvian_ci
ucs2_lithuanian_ci
ucs2_persian_ci
ucs2_polish_ci
ucs2_roman_ci
ucs2_romanian_ci
ucs2_slovak_ci
ucs2_slovenian_ci
ucs2_spanish2_ci
ucs2_spanish_ci
ucs2_swedish_ci
ucs2_turkish_ci
ucs2_unicode_ci
Note that there's a limitation in MySQL 4.1 that results in
two characters not being correctly handled when a user tries
to change their case using LOWER() or
UPPER():
LATIN SMALL LETTER DOTLESS i
LATIN CAPITAL LETTER I WITH DOT ABOVE
Here are two workarounds for MySQL 4.1:
Use UCS2 if you have Turkish data.
Use these function calls:
CONVERT(LOWER(CONVERT(col USING ucs2)) USING utf8)
Currently, the utf8_unicode_ci collation has
only partial support for the Unicode Collation Algorithm. Some
characters are not supported yet. Also, combining marks are not
fully supported. This affects primarily Vietnamese and some
minority languages in Russia such as Udmurt, Tatar, Bashkir, and
Mari.
The most significant feature in
utf8_unicode_ci is that it supports
expansions, that is, when one character compares as equal to
combinations of other characters. For example, in German and
some other languages ‘ß’ is
equal to ‘ss’.
utf8_general_ci is a legacy collation that
does not support expansions. It can make only one-to-one
comparisons between characters. This means that comparisons for
the utf8_general_ci collation are faster, but
slightly less correct, than comparisons for
utf8_unicode_ci).
For example, the following equalities hold in both
utf8_general_ci and
utf8_unicode_ci:
Ä = A
Ö = O
Ü = U
A difference between the collations is that this is true for
utf8_general_ci:
ß = s
Whereas this is true for utf8_unicode_ci:
ß = ss
Language-specific collations for the utf8
character set are implemented only if the ordering with
utf8_unicode_ci does not work well for a
language. For example, utf8_unicode_ci works
fine for German and French, so there is no need to create
special utf8 collations for these two
languages.
utf8_general_ci is also satisfactory for both
German and French, except that
‘ß’ is equal to
‘s’, and not to
‘ss’. If this is acceptable for
your application, then you should use
utf8_general_ci because it is faster.
Otherwise, use utf8_unicode_ci because it is
more accurate.
utf8_swedish_ci, like other
utf8 language-specific collations, is derived
from utf8_unicode_ci with additional language
rules. For example, in Swedish, the following relationship
holds, which is not something expected by a German or French
speaker:
Ü = Y < Ö
The utf8_spanish_ci and
utf8_spanish2_ci collations correspond to
modern Spanish and traditional Spanish, respectively. In both
collations, ‘ñ’ (n-tilde) is a
separate letter between ‘n’ and
‘o’. In addition, for traditional
Spanish, ‘ch’ is a separate
letter between ‘c’ and
d, and ‘ll’ is
a separate letter between ‘l’ and
‘m’
10.11.2. West European Character Sets
Western European character sets cover most West European
languages, such as French, Spanish, Catalan, Basque, Portuguese,
Italian, Albanian, Dutch, German, Danish, Swedish, Norwegian,
Finnish, Faroese, Icelandic, Irish, Scottish, and English.
ascii (US ASCII) collations:
ascii_bin
ascii_general_ci (default)
cp850 (DOS West European) collations:
cp850_bin
cp850_general_ci (default)
dec8 (DEC Western European) collations:
dec8_bin
dec8_swedish_ci (default)
hp8 (HP Western European) collations:
hp8_bin
hp8_english_ci (default)
latin1 (cp1252 West European) collations
(see the annotations about “latin1” below):
latin1_bin
latin1_danish_ci
latin1_general_ci
latin1_general_cs
latin1_german1_ci
latin1_german2_ci
latin1_spanish_ci
latin1_swedish_ci (default)
latin1 is the default character set. The
latin1_swedish_ci collation is the
default that probably is used by the majority of MySQL
customers. While it is frequently said that it is based on
the Swedish/Finnish collation rules, there are Swedes and
Finns who disagree with this statement.
The latin1_german1_ci and
latin1_german2_ci collations are based on
the DIN-1 and DIN-2 standards, where DIN stands for
Deutsches Institut für
Normung (the German equivalent of ANSI).
DIN-1 is called the “dictionary collation” and
DIN-2 is called the “phone book collation”.
latin1_german1_ci (dictionary) rules:
Ä = A
Ö = O
Ü = U
ß = s
latin1_german2_ci (phone-book) rules:
Ä = AE
Ö = OE
Ü = UE
ß = ss
In the latin1_spanish_ci collation,
‘ñ’ (n-tilde) is a separate
letter between ‘n’ and
‘o’.
macroman (Mac West European) collations:
macroman_bin
macroman_general_ci (default)
swe7 (7bit Swedish) collations:
swe7_bin
swe7_swedish_ci (default)
Annotations regarding
latin1: MySQL's
latin1 is the same as the Windows
cp1252 character set. This means it is the
same as the official ISO 8859-1 or IANA
(Internet Assigned Numbers Authority) latin1,
but IANA latin1 treats the code points
between 0x80 and 0x9f as “undefined”, while
cp1252, and therefore MySQL's
latin1, assign characters for those
positions. For example, 0x80 is the Euro sign. For the
“undefined” entries in cp1252,
MySQL translates 0x81 to Unicode
0x0081, 0x8d to
0x008d, 0x8f to
0x008f, 0x90 to
0x0090, and 0x9d to
0x009d.
10.11.3. Central European Character Sets
We also provide some support for character sets used in the
Czech Republic, Slovakia, Hungary, Romania, Slovenia, Croatia,
and Poland.
cp1250 (Windows Central European)
collations:
cp1250_bin
cp1250_croatian_ci
cp1250_czech_cs
cp1250_general_ci (default)
cp852 (DOS Central European) collations:
cp852_bin
cp852_general_ci (default)
keybcs2 (DOS Kamenicky Czech-Slovak)
collations:
keybcs2_bin
keybcs2_general_ci (default)
latin2 (ISO 8859-2 Central European)
collations:
latin2_bin
latin2_croatian_ci
latin2_czech_cs
latin2_general_ci (default)
latin2_hungarian_ci
macce (Mac Central European) collations:
macce_bin
macce_general_ci (default)
10.11.4. South European and Middle East Character Sets
South European and Middle Eastern character sets supported by
MySQL include Armenian, Arabic, Georgian, Greek, Hebrew, and
Turkish:
armscii8 (ARMSCII-8 Armenian) collations:
armscii8_bin
armscii8_general_ci (default)
cp1256 (Windows Arabic) collations:
cp1256_bin
cp1256_general_ci (default)
geostd8 (GEOSTD8 Georgian) collations:
geostd8_bin
geostd8_general_ci (default)
greek (ISO 8859-7 Greek) collations:
greek_bin
greek_general_ci (default)
hebrew (ISO 8859-8 Hebrew) collations:
hebrew_bin
hebrew_general_ci (default)
latin5 (ISO 8859-9 Turkish) collations:
latin5_bin
latin5_turkish_ci (default)
10.11.5. Baltic Character Sets
The Baltic character sets cover Estonian, Latvian, and
Lithuanian languages. There are two Baltic character sets
currently supported:
cp1257 (Windows Baltic) collations:
cp1257_bin
cp1257_general_ci (default)
cp1257_lithuanian_ci
latin7 (ISO 8859-13 Baltic) collations:
latin7_bin
latin7_estonian_cs
latin7_general_ci (default)
latin7_general_cs
10.11.6. Cyrillic Character Sets
Here are the Cyrillic character sets and collations for use with
Belarusian, Bulgarian, Russian, and Ukrainian languages.