linux文本编码格式转化 字幕处理

在处理字幕的时候，linux的编码格式转换很烦。

步骤： 用python先判断 其编码，再用iconv 转编码，再用awk处理格式。

file不能判断吗？file有时不准。

1.python判断编码

$ cat t1.py 
# -*- coding:utf8 -*- 
import sys
#f1=open(sys.argv[2],'w')
with open(sys.argv[1], 'rb') as f:
    for line in f:
        # 转码，因为文件内的编码不一致
        try:
            line = line.decode('utf-8')
        except:
            try:
                line = line.decode('GB2312')  #right
                print('hehe')
            except:
                try:
                    line = line.decode('gbk')
                    print('hehe1')
                except:
                    try:
                        line = line.decode('GB18030')
                        print('hehe2')
                    except:
                        try:
                            line = line.decode('iso-8859-1')  #wrong
                        except:
                            continue

        line = line.strip()  # 去除首尾的空格tab回车换行
        print(line)
        #f1.write(line)

也是试出来的。

如果用file判断：   file -b --mime-encoding  text

2.iconv 转码:        iconv -f "GB2312" -t "utf-8" Ep._20:Valar_Morghulis.ass >  Ep._20:Valar_Morghulis.txt

参考  http://kjetilvalle.com/posts/text-file-encodings.html

综合：

$ cat readme.sh
#!/bin/sh
TO='utf-8'
for i in *ass
do
    FROM=$(file -b --mime-encoding $i)
    p=`basename $i .ass`
    [ $FROM != "iso-8859-1" ] && iconv -f $FROM -t $TO $i > ${p}.txt
    [ $FROM = "iso-8859-1" ] && iconv -f "GB2312" -t $TO $i > ${p}.txt
    awk -F',,' '/Dialogue.*正文/{split($0,arr,",正文,,");split($3,brr,"N");split($3,crr,"{");print "\n"arr[1]"\n" brr[1]"\n"crr[length(crr)-1]}' ${p}.txt |sed -e 's/.*}//g' -e 's/\\$//g'  > ${p}.norm
done