诗歌rails之encoding in ruby and rails

    项目中有一个csv文件因为是latin1 encoded,所以解析就会出现错误。

 

    解决方案:可以通过Iconv.iconv("UTF-8", "latin1", file.read)把stream转化成UTF-8格式(rails的默认编码是UTF-8)。

 

    局限:当然此种方法的局限是限定了文件的编码只能是latin1的时候才能解析正确。当然,如果没有特殊字符,其他的一些ASCII字符(single byte character)在两种编码之间会转换正确。但一当文件中有其它编码的特殊字符,那就歇菜了。

 

    更进一步:使用gussing工具猜编码吧,比如下面这段

 

    File.open(tmp_file, 'w') do |f|
      input   = file.read
      charset = CMess::GuessEncoding::Automatic.guess(input)
      f.write(Iconv.iconv('UTF-8', charset, input))
    end

 

    以上问题详见此讨论

 

    顺便说下ruby的encoding

 

    1. In ruby <= 1.8, strings were effectively just byte streams. Those bytes would often contain text in one encoding or another, but there was no formal way to record exactly which one (if any; binary data has none). All the default methods assumed single byte encoding (such as US-ASCII), so behaviour was odd when the encoding was multibyte (such as UTF8).

 

    2. In ruby 1.9:

 

        str.encode!("UTF-8")

        File.open("file", "r:UTF-8") {|f| ...........}

posted @ 2009-08-25 14:21  麦飞  阅读(588)  评论(0编辑  收藏  举报