UTF-8, a transformation format of Unicode and ISO 10646
UTF8，一种Unicode与ISO 10646的转换格式
Status of this Memo
本备忘的状态
This memo provides information for the Internet community. This memo
does not specify an Internet standard of any kind. Distribution of
this memo is unlimited.
这份备忘录为网络社区提供信息。本备忘录不指定任何一种Internet标准。不对该备忘录的发布做出任何限制。
Abstract
摘要
The Unicode Standard, version 1.1, and ISO/IEC 10646-1:1993 jointly
define a 16 bit character set which encompasses most of the world's
writing systems. 16-bit characters, however, are not compatible with
many current applications and protocols, and this has led to the
development of a few so-called UCS transformation formats (UTF), each
with different characteristics. UTF-8, the object of this memo, has
the characteristic of preserving the full US-ASCII range: US-ASCII
characters are encoded in one octet having the usual US-ASCII value,
and any octet with such a value can only be an US-ASCII character.
This provides compatibility with file systems, parsers and other
software that rely on US-ASCII values but are transparent to other
values.
Unicode标准(V1.1)和ISO/IEC 10646-1:1993共同定义了一个16位的字符集，它几乎包含了世界上所有的文字体系。
然而，16位的字符集与目前很多的应用以及协议不能兼容。这导致了一些具有不同特点的UCS 转换格式(UTF)的出现。
UTF-8，本备忘的主要内容，完整地保留了US-ASCII的排列特点。使用8位编码字符的与US-ASCII一致，任何一个符合
这种格式8位编码对应于唯一的US-ASCII字符。这种方法使得依赖US-ASCII的文件系统、解析器及其其他软件具备更好的兼容性。
1. Introduction
介绍
The Unicode Standard, version 1.1 [UNICODE], and ISO/IEC 10646-1:1993
[ISO-10646] jointly define a 16 bit character set, UCS-2, which
encompasses most of the world's writing systems. ISO 10646 further
defines a 31-bit character set, UCS-4, with currently no assignments
outside of the region corresponding to UCS-2 (the Basic Multilingual
Plane, BMP). The UCS-2 and UCS-4 encodings, however, are hard to use
in many current applications and protocols that assume 8 or even 7
bit characters. Even newer systems able to deal with 16 bit
characters cannot process UCS-4 data. This situation has led to the
development of so-called UCS transformation formats (UTF), each with
different characteristics.
Unicode标准(V1.1)和ISO/IEC 10646-1:1993共同定义了一个16位的字符集，它几乎包含了世界上所有的文字体系。
ISO 10646又定义了一种31位的字符集，UCS-4，目前还没有在此区域以外分配UCS-2（BMP）。
这种UCS-2和UCS-4的编码方法很难被目前仅支持8或者7位字符集的设备兼容。甚至一些较新的支持16位
字符集的系统也无法支持UCS-4编码。这种状况导致了多种UTF编码方式的发展。
UTF-1 has only historical interest, having been removed from ISO
10646. UTF-7 has the quality of encoding the full Unicode repertoire
using only octets with the high-order bit clear (7 bit US-ASCII
values, [US-ASCII]), and is thus deemed a mail-safe encoding
([RFC1642]). UTF-8, the object of this memo, uses all bits of an
octet, but has the quality of preserving the full US-ASCII range:
UTF-1仅仅是出于历史的兴趣，已经从ISO 10646中删除。UTF-7能够完整地进行编码，
使用清晰的高序位只有7位。因此为视为一种邮件安全的编码方式。UTF-8，使用1个字节的所有8位，
但依然能够保留US-ASCII码的排列次序。
Yergeau Informational [Page 1]
RFC 2044 UTF-8 October 1996
US-ASCII characters are encoded in one octet having the normal US-
ASCII value, and any octet with such a value can only stand for an
US-ASCII character, and nothing else.
8位代表一个US-ASCII码。
UTF-16 is a scheme for transforming a subset of the UCS-4 repertoire
into a pair of UCS-2 values from a reserved range. UTF-16 impacts
UTF-8 in that UCS-2 values from the reserved range must be treated
specially in the UTF-8 transformation.
UTF16是一项将UCS -4转换成2个反序的UCS-2的计划。UTF-16影响了UTF-8，
因此反序的UCS-2码在UTF-8中要被妥善地处理。
UTF-8 encodes UCS-2 or UCS-4 characters as a varying number of
octets, where the number of octets, and the value of each, depend on
the integer value assigned to the character in ISO 10646. This
transformation format has the following characteristics (all values
are in hexadecimal):
- Character values from 0000 0000 to 0000 007F (US-ASCII repertoire)
correspond to octets 00 to 7F (7 bit US-ASCII values).
- US-ASCII values do not appear otherwise in a UTF-8 encoded charac-
ter stream. This provides compatibility with file systems or
other software (e.g. the printf() function in C libraries) that
parse based on US-ASCII values but are transparent to other val-
ues.
- Round-trip conversion is easy between UTF-8 and either of UCS-4,
UCS-2 or Unicode.
- The first octet of a multi-octet sequence indicates the number of
octets in the sequence.
- Character boundaries are easily found from anywhere in an octet
stream.
- The lexicographic sorting order of UCS-4 strings is preserved. Of
course this is of limited interest since the sort order is not
culturally valid in either case.
- The octet values FE and FF never appear.
UTF-8 was originally a project of the X/Open Joint
Internationalization Group XOJIG with the objective to specify a File
System Safe UCS Transformation Format [FSS-UTF] that is compatible
with UNIX systems, supporting multilingual text in a single encoding.
The original authors were Gary Miller, Greger Leijonhufvud and John
Entenmann. Later, Ken Thompson and Rob Pike did significant work for
the formal UTF-8.
Yergeau Informational [Page 2]
RFC 2044 UTF-8 October 1996
A description can also be found in Unicode Technical Report #4 [UNI-
CODE]. The definitive reference, including provisions for UTF-16
data within UTF-8, is Annex R of ISO/IEC 10646-1 [ISO-10646].
2. UTF-8 definition
UTF-8的定义
In UTF-8, characters are encoded using sequences of 1 to 6 octets.
The only octet of a "sequence" of one has the higher-order bit set to
0, the remaining 7 bits being used to encode the character value. In
a sequence of n octets, n>1, the initial octet has the n higher-order
bits set to 1, followed by a bit set to 0. The remaining bit(s) of
that octet contain bits from the value of the character to be
encoded. The following octet(s) all have the higher-order bit set to
1 and the following bit set to 0, leaving 6 bits in each to contain
bits from the character to be encoded.
在UTF-8中，字符通过一个1到6个8位的序列编码。
只有1个8位的序列最高位编码为0，其余的7位用该编码这个字符的值。
当序列中字节数大于1的时候，第一个字节的最高位被设置为1，紧接着的一位为0，其余的6位来表示这个字符的编码。
The table below summarizes the format of these different octet types.
The letter x indicates bits available for encoding bits of the UCS-4
character value.
下面的这张表总结了这些不同的编码方式，字母x代表UCS-4编码位的字符值。
UCS-4 range (hex.) UTF-8 octet sequence (binary)
0000 0000-0000 007F 0xxxxxxx
0000 0080-0000 07FF 110xxxxx 10xxxxxx
0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10xxxxxx
0001 0000-001F FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
0020 0000-03FF FFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
0400 0000-7FFF FFFF 1111110x 10xxxxxx ... 10xxxxxx
Encoding from UCS-4 to UTF-8 proceeds as follows:
UCS-4到UTF-8的编码转换方法：
1) Determine the number of octets required from the character value
and the first column of the table above.
1）确定转换后的编码占多少位，以及上述表格的第一列。
2) Prepare the high-order bits of the octets as per the second column
of the table.
2）为表的第二列准备高8位。
3) Fill in the bits marked x from the bits of the character value,
starting from the lower-order bits of the character value and
putting them first in the last octet of the sequence, then the
next to last, etc. until all x bits are filled in.
3）从低8位开始，计算并填写对应的x值。
Yergeau Informational [Page 3]
RFC 2044 UTF-8 October 1996
The algorithm for encoding UCS-2 (or Unicode) to UTF-8 can be
obtained from the above, in principle, by simply extending each
UCS-2 character with two zero-valued octets. However, UCS-2 val-
ues between D800 and DFFF, being actually UCS-4 characters trans-
formed through UTF-16, need special treatment: the UTF-16 trans-
formation must be undone, yielding a UCS-4 character that is then
transformed as above.
Decoding from UTF-8 to UCS-4 proceeds as follows:
1) Initialize the 4 octets of the UCS-4 character with all bits set
to 0.
2) Determine which bits encode the character value from the number of
octets in the sequence and the second column of the table above
(the bits marked x).
3) Distribute the bits from the sequence to the UCS-4 character,
first the lower-order bits from the last octet of the sequence and
proceeding to the left until no x bits are left.
If the UTF-8 sequence is no more than three octets long, decoding
can proceed directly to UCS-2 (or equivalently Unicode).
A more detailed algorithm and formulae can be found in [FSS_UTF],
[UNICODE] or Annex R to [ISO-10646].
3. Examples
The Unicode sequence "A<NOT IDENTICAL TO><ALPHA>." (0041, 2262, 0391,
002E) may be encoded as follows:
41 E2 89 A2 CE 91 2E
The Unicode sequence "Hi Mom <WHITE SMILING FACE>!" (0048, 0069,
0020, 004D, 006F, 006D, 0020, 263A, 0021) may be encoded as follows:
48 69 20 4D 6F 6D 20 E2 98 BA 21
The Unicode sequence representing the Han characters for the Japanese
word "nihongo" (65E5, 672C, 8A9E) may be encoded as follows:
E6 97 A5 E6 9C AC E8 AA 9E
Yergeau Informational [Page 4]
RFC 2044 UTF-8 October 1996
MIME registrations
This memo is meant to serve as the basis for registration of a MIME
character encoding (charset) as per [RFC1521]. The proposed charset
parameter value is "UTF-8". This string would label media types
containing text consisting of characters from the repertoire of ISO
10646-1 encoded to a sequence of octets using the encoding scheme
outlined above.
Security Considerations
Security issues are not discussed in this memo.
Acknowledgments
The following have participated in the drafting and discussion of
this memo:
James E. Agenbroad Andries Brouwer
Martin J. D|rst David Goldsmith
Edwin F. Hart Kent Karlsson
Markus Kuhn Michael Kung
Alain LaBonte Murray Sargent
Keld Simonsen Arnold Winkler
Bibliography
[FSS_UTF] X/Open CAE Specification C501 ISBN 1-85912-082-2 28cm.
22p. pbk. 172g. 4/95, X/Open Company Ltd., "File Sys-
tem Safe UCS Transformation Format (FSS_UTF)", X/Open
Preleminary Specification, Document Number P316. Also
published in Unicode Technical Report #4.
[ISO-10646] ISO/IEC 10646-1:1993. International Standard -- Infor-
mation technology -- Universal Multiple-Octet Coded
Character Set (UCS) -- Part 1: Architecture and Basic
Multilingual Plane. UTF-8 is described in Annex R,
adopted but not yet published. UTF-16 is described in
Annex Q, adopted but not yet published.
[RFC1521] Borenstein, N., and N. Freed, "MIME (Multipurpose
Internet Mail Extensions) Part One: Mechanisms for
Specifying and Describing the Format of Internet Mes-
sage Bodies", RFC 1521, Bellcore, Innosoft, September
1993.
[RFC1641] Goldsmith, D., and M. Davis, "Using Unicode with
MIME", RFC 1641, Taligent inc., July 1994.
Yergeau Informational [Page 5]
RFC 2044 UTF-8 October 1996
[RFC1642] Goldsmith, D., and M. Davis, "UTF-7: A Mail-safe
Transformation Format of Unicode", RFC 1642,
Taligent, Inc., July 1994.
[UNICODE] The Unicode Consortium, "The Unicode Standard --
Worldwide Character Encoding -- Version 1.0", Addison-
Wesley, Volume 1, 1991, Volume 2, 1992. UTF-8 is
described in Unicode Technical Report #4.
[US-ASCII] Coded Character Set--7-bit American Standard Code for
Information Interchange, ANSI X3.4-1986.
在windows下打开记事本，写入“AB汉”三个字，通过16进制编辑器打开，发现：
Unicode Little-endian FF FE 41 00 42 00 49 6C
UTF-8: EF BB BF 41 42 E6 B1 89
ANSI: 41 42 BA BA
UNICODE BIG ENDIAN: FE FF 00 41 00 42 6C 49
发现在windows下使用记事本记录数据时，
采用ANSI编码没有文件头，即为默认编码方式；
采用Unicode编码时，文件头为 FF FE；
采用UTF-8编码时，文件头为 EF BB；
采用UNICODE BIG ENDIAN编码时，文件头为： FE FF；
根据上面提供的转换方法：“汉”字的UNICODE BIG ENDIAN码为 6C 49，转换为UTF-8码需要3个字节。
0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10xxxxxx
即 0000 6C 49 (0110 1100 0100 1001) 被转换为 11100110 10110001 10001001(E6 B1 89)
posted on
