UTF-8 的BOM带来的麻烦- -

转自 http://blog.sina.com.cn/s/blog_3e9d2b350100as0b.html

我是用java 写文件遇到的这个问题， 文件的头3个字节写入utf-8 的bom, 即     EF BB BF 3个字节，乱码问题解决

 

工作需要我用程序生成一个html文件。
由于服务器端使用apache+Tomcat来执行html和jsp文件。
开始生成html文件放在apache目录下，页面无法默认正常识别我页面设置的编码。

必须手动在浏览器上选择Encoding->简体中文（GB2312）才可以正常显示。
这样当然是不行了。
由于我们原来有一个页面是可以正常显示中文的，查看了一下，是UTF－8的格式，于是我也修改程序。
a.修改了页面的编码声明：

b.修改了写字节流的一个方法：
public void htmlWrite(String charsetName) {
        try {
            out = new BufferedWriter(new OutputStreamWriter(
                        new FileOutputStream(outFileName), "UTF-8"));
            out.write(res);
            out.flush();

            if (out != null) {
                out.close();
            }
        } catch (Exception e) {
            try {
                if (out != null) {
                    out.close();
                }
            } catch (IOException e1) {
                System.out.print("write errors!" + e);
            }

            System.out.print("write errors!" + e);
        }
    }
这样，我又生成了一个html，放在服务器下面，可问题又来了，还是无法正常显示，即浏览器无法默认识别为UTF-8的编码方式。奇怪，使用EmEditor打开，和好用的那个页面对比。没有任何问题。唯一的区别在于：
    我生成的那个html文件被EmEditor认为UTF-8 with Signature。而好用的那个html文件被EmEditor认为UTF-8 without Signature.
    对于这两种UTF－8格式的转换，我查看了网上信息，点击记事本，EmEditor等文本编辑器的另存为，当选择了UTF-8的编码格式时，Add a Unicode Signature(BOM)这个选项被激活，只要选择上，我的文件就可以存为UTF-8 with Signature的格式。可是，问题就在于，我用java怎么让我的文件直接生成为 UTF-8 with Signature的格式。
    开始上google搜索UTF-8 with Signature,BOM,Add a Unicode Signature等关键字。
http://www.unicode.org/unicode/faq/utf_bom.html#BOM
我大致了解了他们两个的区别。
Q: What is a BOM?

A: A byte order mark (BOM) consists of the character code U+FEFF at the beginning of a data stream, where it can be used as a signature defining the byte order and encoding form, primarily of unmarked plaintext files. Under some higher level protocols, use of a BOM may be mandatory (or prohibited) in the Unicode data stream defined in that protocol.
http://mindprod.com/jgloss/bom.html
BOM
Byte Order Marks are special characters at the beginning of a Unicode file to indicate whether it is big or little endian, in other words does the high or low order byte come first. These codes also tell whether the encoding is 8, 16 or 32 bit. You can recognise Unicode files by their starting byte order marks, and by the way Unicode-16 files are half zeroes and Unicode-32 files are three-quarters zeros. Unicode Endian Markers
Byte-order mark Description
EF BB BF UTF-8
FF FE UTF-16 aka UCS-2, little endian
FE FF UTF-16 aka UCS-2, big endian
00 00 FF FE UTF-32 aka UCS-4, little endian.
00 00 FE FF UTF-32 aka UCS-4, big-endian.
There are also variants of these encodings that have an implied endian marker.
Unfortunately, often applications, even Javac.exe, choke on these byte order marks. Java Readers don't automatically filter them out. There is not much you can do but manually remove them.

http://cache.baidu.com/c?word=java%2Cbom&url=http%3A//tgdem530%2Eblogchina%2Ecom/&b=0&a=1&user=baidu
c、UTF的字节序和BOM
UTF-8以字节为编码单元，没有字节序的问题。UTF-16以两个字节为编码单元，在解释一个UTF-16文本前，首先要弄清楚每个编码单元的字节序。例如收到一个“奎”的Unicode编码是594E，“乙”的Unicode编码是4E59。如果我们收到UTF-16字节流“594E”，那么这是“奎”还是“乙”？

Unicode规范中推荐的标记字节顺序的方法是BOM。BOM不是“Bill Of Material”的BOM表，而是Byte Order Mark。BOM是一个有点小聪明的想法：

在UCS编码中有一个叫做"ZERO WIDTH NO-BREAK SPACE"的字符，它的编码是FEFF。而FFFE在UCS中是不存在的字符，所以不应该出现在实际传输中。UCS规范建议我们在传输字节流前，先传输字符"ZERO WIDTH NO-BREAK SPACE"。

这样如果接收者收到FEFF，就表明这个字节流是Big-Endian的；如果收到FFFE，就表明这个字节流是Little-Endian的。因此字符"ZERO WIDTH NO-BREAK SPACE"又被称作BOM。

UTF-8不需要BOM来表明字节顺序，但可以用BOM来表明编码方式。字符"ZERO WIDTH NO-BREAK SPACE"的UTF-8编码是EF BB BF（读者可以用我们前面介绍的编码方法验证一下）。所以如果接收者收到以EF BB BF开头的字节流，就知道这是UTF-8编码了。

Windows就是使用BOM来标记文本文件的编码方式的。

原来BOM是在文件的开始加了几个字节作为标记。有了这个标记，一些协议和系统才能识别。好，看看怎么加上这写字节。
终于在这里找到了
http://mindprod.com/jgloss/encoding.html 
UTF-8 
8-bit encoded Unicode. neé UTF8. Optional marker on front of file: EF BB BF for reading. Unfortunately, OutputStreamWriter does not automatically insert the marker on writing. Notepad can't read the file without this marker. Now the question is, how do you get that marker in there? You can't just emit the bytes EF BB BF since they will be encoded and changed. However, the solution is quite simple. prw.write( '\ufeff' ); at the head of the file. This will be encoded as EF BB BF.
DataOutputStreams have a binary length count in front of each string. Endianness does not apply to 8-bit encodings. Java DataOutputStream and ObjectOutputStream uses a slight variant of kosher UTF-8. To aid with compatibility with C in JNI, the null byte '\u0000' is encoded in 2-byte format rather than 1-byte, so that the encoded strings never have embedded nulls. Only the 1-byte, 2-byte, and 3-byte formats are used. Supplementary characters, (above 0xffff), are represented in the form of surrogate pairs (a pair of encoded 16 bit characters in a special range), rather than directly encoding the character.
 
prw.write( '\ufeff' );就是这个。
于是我的代码变为：
public void htmlWrite(String charsetName) {
        try {
            out = new BufferedWriter(new OutputStreamWriter(
                        new FileOutputStream(outFileName), "UTF-8"));
            out.write('\ufeff');
            out.write(res);
            out.flush();

            if (out != null) {
                out.close();
            }
        } catch (Exception e) {
            try {
                if (out != null) {
                    out.close();
                }
            } catch (IOException e1) {
                System.out.print("write errors!" + e);
            }

            System.out.print("write errors!" + e);
        }
    }
问题解决。