UTF8 与 UTF16 编码
Unicode 的发展,英文好的直接去 unicode.org 上去看吧,不好的可以移步到这里 看dengyunze的总结:《关于UTF8,UTF16,UTF32,UTF16-LE,UTF16-BE 》 。此文讲的清除明白:为了能把世界上的所有字符都表示,理论上需要用 UTF-16,但是由于“大部分”(当然这是欧美那边技术宅男拍脑袋想出来的大部分啦~)的字符只需要 1 个字节就搞定了,用 UTF16 实在太浪费啦,于是他们就用了 UTF8. 对于那些个“少数”(比如中日韩)的字符,就通过一个 UTF8-UTF16 的转换来表示。
UTF8 和 UTF16 都是变长表示的,为啥欧美技术宅会觉得太浪费了咧?因为欧美字符 0x0000 - 0x00FF 就搞定了,UTF8 最小变长是 1 个字节,而 UTF16 变长是 2 个字节,所以……(↓看下图中 code unit size)
注意:上面这个图中,UTF-16 和 UTF-16LE 是一样的,因为…… UTF16 默认就是 UTF-16LE
那么,UTF8是如何表示 的咧?↓看下图
↓↓ 举例
表示的方法跟上上个图对应,第一个字节中,从左往右第一个 10 前面的 “1” 的个数表示后面还有这么多个的字节在表示这个字符。UTF8 最多可以表示 31 bit 的字符。
UTF16 编码的过程
v = 0x64321 v′ = v - 0x10000 = 0x54321 = 0101 0100 0011 0010 0001 vh = v′ >> 10 = 01 0101 0000 // higher 10 bits of v′ vl = v′ & 0x3FF = 11 0010 0001 // lower 10 bits of v′ w1 = 0xD800 + vh = 1101 1000 0000 0000 + 01 0101 0000 = 1101 1001 0101 0000 = 0xD950 // first code unit of UTF-16 encoding w2 = 0xDC00 + vl = 1101 1100 0000 0000 + 11 0010 0001 = 1101 1111 0010 0001 = 0xDF21 // second code unit of UTF-16 encoding
附一段 java 版本的 UTF8 与 UTF16 的相互转换,代码来源于 Lucene3.6
/** * Interprets the given byte array as UTF-8 and converts to UTF-16. The * {@link CharsRef} will be extended if it doesn't provide enough space to * hold the worst case of each byte becoming a UTF-16 codepoint. * <p> * NOTE: Full characters are read, even if this reads past the length passed * (and can result in an ArrayOutOfBoundsException if invalid UTF-8 is * passed). Explicit checks for valid UTF-8 are not performed. */ // TODO: broken if chars.offset != 0 public static void UTF8toUTF16(byte[] utf8, int offset, int length, CharsRef chars) { int out_offset = chars.offset = 0; final char[] out = chars.chars = ArrayUtil.grow(chars.chars, length); final int limit = offset + length; while (offset < limit) { int b = utf8[offset++] & 0xff; if (b < 0xc0) { assert b < 0x80; out[out_offset++] = (char) b; } else if (b < 0xe0) { out[out_offset++] = (char) (((b & 0x1f) << 6) + (utf8[offset++] & 0x3f)); } else if (b < 0xf0) { out[out_offset++] = (char) (((b & 0xf) << 12) + ((utf8[offset] & 0x3f) << 6) + (utf8[offset + 1] & 0x3f)); offset += 2; } else { assert b < 0xf8 : "b=" + b; int ch = ((b & 0x7) << 18) + ((utf8[offset] & 0x3f) << 12) + ((utf8[offset + 1] & 0x3f) << 6) + (utf8[offset + 2] & 0x3f); offset += 3; if (ch < UNI_MAX_BMP) { out[out_offset++] = (char) ch; } else { int chHalf = ch - 0x0010000; out[out_offset++] = (char) ((chHalf >> 10) + 0xD800); out[out_offset++] = (char) ((chHalf & HALF_MASK) + 0xDC00); } } } chars.length = out_offset - chars.offset; } /** Encode characters from a char[] source, starting at * offset for length chars. After encoding, result.offset will always be 0. */ public static void UTF16toUTF8(final char[] source, final int offset, final int length, BytesRef result) { int upto = 0; int i = offset; final int end = offset + length; byte[] out = result.bytes; // Pre-allocate for worst case 4-for-1 final int maxLen = length * 4; if (out.length < maxLen) out = result.bytes = new byte[maxLen]; result.offset = 0; while(i < end) { final int code = (int) source[i++]; if (code < 0x80) out[upto++] = (byte) code; else if (code < 0x800) { out[upto++] = (byte) (0xC0 | (code >> 6)); out[upto++] = (byte)(0x80 | (code & 0x3F)); } else if (code < 0xD800 || code > 0xDFFF) { out[upto++] = (byte)(0xE0 | (code >> 12)); out[upto++] = (byte)(0x80 | ((code >> 6) & 0x3F)); out[upto++] = (byte)(0x80 | (code & 0x3F)); } else { // surrogate pair // confirm valid high surrogate if (code < 0xDC00 && i < end) { int utf32 = (int) source[i]; // confirm valid low surrogate and write pair if (utf32 >= 0xDC00 && utf32 <= 0xDFFF) { utf32 = (code << 10) + utf32 + SURROGATE_OFFSET; i++; out[upto++] = (byte)(0xF0 | (utf32 >> 18)); out[upto++] = (byte)(0x80 | ((utf32 >> 12) & 0x3F)); out[upto++] = (byte)(0x80 | ((utf32 >> 6) & 0x3F)); out[upto++] = (byte)(0x80 | (utf32 & 0x3F)); continue; } } // replace unpaired surrogate or out-of-order low surrogate // with substitution character out[upto++] = (byte) 0xEF; out[upto++] = (byte) 0xBF; out[upto++] = (byte) 0xBD; } } //assert matches(source, offset, length, out, upto); result.length = upto; }
TRUST WITHOUT WAVERING