基于COCA词频表的文本词汇分布测试工具v0.1

美国语言协会对美国人日常使用的英语单词做了一份详细的统计，按照日常使用的频率做成了一张表，称为COCA词频表。排名越低的单词使用频率越高，该表可以用来统计词汇量。

如果你的词汇量约为6000，那么这张表频率6000以下的单词你应该基本都认识。（不过国内教育平时学的单词未必就是他们常用的，只能说大部分重合）

 

 

我一直有个想法，要是能用COCA词频表统计一本小说中所有的词汇都是什么等级的，然后根据自己的词汇量，就能大致确定这本小说是什么难度，自己能不能读了。

学习了C++的容器和标准库算法后，我发现这个想法很容易实现，于是写了个能够统计文本词频的小工具。现在只实现了基本功能，写得比较糙。还有很多要改进的地方，随着以后的学习，感觉这个工具还是可以再深挖的。

 

v0.1

1.可逐个分析文本中的每一个单词并得到其COCA词频，生成统计报告。
2.仅支持txt。

 

实现的重点主要分三块。

（1）COCA词典的建立

首先建立出COCA词典。我使用了恶魔的奶爸公众号提供的txt格式的COCA词频表，这张表是经过处理的，每行只有一个单词和对应的频率，所以读取起来很方便。

 

读取文本后，用map容器来建立“单词——频率”的一对一映射。

要注意的是有些单词因为使用意思和词性的不同，统计表里会出现两个频率，也就是熟词的偏僻意思。我只取了较低的那个为准，因为你读文本时，虽然可能不是真正的意思但起码这个单词是认识的，而且这种情况总体来说比较少见，所以影响不大。

存储时我用了26组map，一个首字母存一组，单词查询时按首字母查找。可能有更有效率的办法。

 

（2）单词的获取和处理

使用get()方法逐个获取文本的字符。

使用vector容器存储读到的文本内容。如果是字母或连字符就push进去，如果不是，就意味着一个单词读完了，接着开始处理，把读到的文本内容assign到一个字符串中组成完整的单词，然后根据其首字母查找词频。

每个单词查到词频后，使用自定义的frequency_classify()区分它在什么范围，再用word_frequency_analyze()统计在不同等级的分布。

最后报告结果，显示出每个等级的百分比。

 

（3）单词的修正

首先是分析过的单词不会再分析第二遍，因为一般原文中像I，he，she，it，a，this，that这种特别常见的词还是很多的，如果每个都统计进去，会导致不同文章看到的结果都是高频词占到80%，90%。应该以单词的种类为准，而不是数量。

其次有很多单词的变形COCA词频表是不统计的，像许多单词的过去式，复数，进行时等，还有was，were，has，had等。所以我单独加了word_usual_recheck()和word_form_recheck()两个函数来检查是否有这些情况，如果有变形的话，就变为原型再尝试查找一次。但是这样做相当于手动修正，不能覆盖到所有情况，比如有些单词的变形并不遵循传统。

 

我先用了乔布斯斯坦福大学的演讲稿测试。测试结果：

 

总共分析了869个单词，77.45%的单词都找到了，且分布在4000以下，可见演讲稿使用的绝大多数的单词都是极其常用的单词，有4000以上的词汇量就基本能读懂。

 

我再测试了一下《小王子》的原文。

 

约70%的单词都找到了，且分布在4000以下，再加上4000-8000之间的，总共有接近80%的单词是六七千以下的级别。

 

为了对比，我再随便找了一篇纽约客上的文章测试。

 

可以看到，高频词的比例明显降低了一些，约有10%的单词是8000以上的级别，还有更多的单词找不到了，说明出现了许多单词的变形，当然也可能包括人名，毕竟是讨论政治的文章，还有小部分可能是超出20000。

 

以下是代码。

TextVolcabularyAnalyzer.cpp

#include <iostream>
#include <fstream>
#include <string>
#include <algorithm>
#include <array>
#include <vector>
#include <iterator>
#include <map>
#include <numeric>
#include <iomanip>
#include <ctime>

#define COCA_WORDS_NUM       20201U
#define WORDS_HEAD_NUM       26U

#define WORDS_HEAD_A         0U
#define WORDS_HEAD_B         1U
#define WORDS_HEAD_C         2U
#define WORDS_HEAD_D         3U
#define WORDS_HEAD_E         4U
#define WORDS_HEAD_F         5U
#define WORDS_HEAD_G         6U
#define WORDS_HEAD_H         7U
#define WORDS_HEAD_I         8U
#define WORDS_HEAD_J         9U
#define WORDS_HEAD_K         10U
#define WORDS_HEAD_L         11U
#define WORDS_HEAD_M         12U
#define WORDS_HEAD_N         13U
#define WORDS_HEAD_O         14U
#define WORDS_HEAD_P         15U
#define WORDS_HEAD_Q         16U
#define WORDS_HEAD_R         17U
#define WORDS_HEAD_S         18U
#define WORDS_HEAD_T         19U
#define WORDS_HEAD_U         20U
#define WORDS_HEAD_V         21U
#define WORDS_HEAD_W         22U
#define WORDS_HEAD_X         23U
#define WORDS_HEAD_Y         24U
#define WORDS_HEAD_Z         25U

#define USUAL_WORD_NUM       17U

using namespace std;

typedef enum WordFrequencyType
{
    WORD_UNDER_4000 = 0,
    WORD_4000_6000,
    WORD_6000_8000,
    WORD_8000_10000,
    WORD_10000_12000,
    WORD_12000_14000,
    WORD_14000_16000,
    WORD_OVER_16000,
    WORD_NOT_FOUND_COCA,
    WORD_LEVEL_NUM
}TagWordFrequencyType;

const string alphabet_str = "abcdefghijklmnopqrstuvwxyz";

const string report_str[WORD_LEVEL_NUM] = {
    "UNDER 4000: ",
    "4000-6000: ",
    "6000-8000: ",
    "8000-10000: ",
    "10000-12000: ",
    "12000-14000: ",
    "14000-16000: ",
    "16000-20000+: ",
    "\nNot found in COCA:"
};

//for usual words not included in COCA
const string usual_w_out_of_COCA_str[USUAL_WORD_NUM] =
{
    "s","is","are","re","was","were",
    "an","won","t","has","had","been",
    "did","does","cannot","got","men"
};

bool word_usual_recheck(const string &ws)
{
    bool RetVal = false;
    for (int i = 0; i < USUAL_WORD_NUM;i++)
    {
        if (ws == usual_w_out_of_COCA_str[i])
        {
            RetVal = true;
        }
        else
        {
            //do nothing
        }
    }
    return RetVal;
}

bool word_form_recheck(string& ws)
{
    bool RetVal = false;
    if (ws.length() > 3)
    {
        char e1, e2,e3;
        e3 = ws[ws.length() - 3];    //last but two letter
        e2 = ws[ws.length() - 2];    //last but one letter
        e1 = ws[ws.length() - 1];    //last letter

        if (e1 == 's')
        {
            ws.erase(ws.length() - 1);
            RetVal = true;
        }
        else if (e2 == 'e' && e1 == 'd')
        {
            ws.erase(ws.length() - 1);
            ws.erase(ws.length() - 1);
            RetVal = true;
        }
        else if (e3 == 'i' && e2 == 'n' && e1 == 'g')
        {
            ws.erase(ws.length() - 1);
            ws.erase(ws.length() - 1);
            ws.erase(ws.length() - 1);
            RetVal = true;
        }
        else
        {
            //do nothing
        }
    }
    
    return RetVal;
}

TagWordFrequencyType frequency_classify(const int wfrq)
{
    if (wfrq == 0)
    {
        return WORD_NOT_FOUND_COCA;
    }
    else if (wfrq > 0 && wfrq <= 4000)
    {
        return WORD_UNDER_4000;
    }
    else if (wfrq > 4000 && wfrq <= 6000)
    {
        return WORD_4000_6000;
    }
    else if (wfrq > 6000 && wfrq <= 8000)
    {
        return WORD_6000_8000;
    }
    else if (wfrq > 8000 && wfrq <= 10000)
    {
        return WORD_8000_10000;
    }
    else if (wfrq > 10000 && wfrq <= 12000)
    {
        return WORD_10000_12000;
    }
    else if (wfrq > 12000 && wfrq <= 14000)
    {
        return WORD_12000_14000;
    }
    else if (wfrq > 14000 && wfrq <= 16000)
    {
        return WORD_14000_16000;
    }
    else
    {
        return WORD_OVER_16000;
    }
}

void word_frequency_analyze(array<int, WORD_LEVEL_NUM> & wfrq_array, TagWordFrequencyType wfrq_tag)
{
    switch (wfrq_tag)
    {
        case WORD_UNDER_4000:
        {
            wfrq_array[WORD_UNDER_4000] += 1;
            break;
        }
        case WORD_4000_6000:
        {
            wfrq_array[WORD_4000_6000] += 1;
            break;
        }
        case WORD_6000_8000:
        {
            wfrq_array[WORD_6000_8000] += 1;
            break;
        }
        case WORD_8000_10000:
        {
            wfrq_array[WORD_8000_10000] += 1;
            break;
        }
        case WORD_10000_12000:
        {
            wfrq_array[WORD_10000_12000] += 1;
            break;
        }
        case WORD_12000_14000:
        {
            wfrq_array[WORD_12000_14000] += 1;
            break;
        }
        case WORD_14000_16000:
        {
            wfrq_array[WORD_14000_16000] += 1;
            break;
        }
        case WORD_OVER_16000:
        {
            wfrq_array[WORD_OVER_16000] += 1;
            break;
        }
        default:
        {
            wfrq_array[WORD_NOT_FOUND_COCA] += 1;
            break;
        }
    }
}

bool isaletter(const char& c)
{
    if ((c >= 'a' && c <= 'z') || (c >= 'A' && c<= 'Z'))
    {
        return true;
    }
    else
    {
        return false;
    }
}

int main()
{
    //file init
    ifstream COCA_txt("D:\\COCA.txt");
    ifstream USER_txt("D:\\test.txt");

    //time init
    clock_t startTime, endTime;
    double build_map_time = 0;
    double process_time = 0;

    startTime = clock();    //build time start

    //build COCA words map
    map<string, int> COCA_WordsList[WORDS_HEAD_NUM];
    int readlines = 0;

    while (readlines < COCA_WORDS_NUM)
    {
        int frequency = 0; string word = "";
        COCA_txt >> frequency;
        COCA_txt >> word;

        //transform to lower uniformly 
        transform(word.begin(), word.end(), word.begin(), tolower);

        //import every word
        for (int whead = WORDS_HEAD_A; whead < WORDS_HEAD_NUM; whead++)
        {
            //check word head 
            if (word[0] == alphabet_str[whead])
            {
                //if a word already exists, only load its lower frequency
                if (COCA_WordsList[whead].find(word) == COCA_WordsList[whead].end())
                {
                    COCA_WordsList[whead].insert(make_pair(word, frequency));
                }
                else
                {
                    COCA_WordsList[whead][word] = frequency < COCA_WordsList[whead][word] ? frequency : COCA_WordsList[whead][word];
                }
            }
            else
            {
                // do nothing
            }
        }
        readlines++;
    }

    endTime = clock();    //build time stop
    build_map_time = (double)(endTime - startTime) / CLOCKS_PER_SEC;

    //user prompt
    cout << "COCA words list imported.\nPress any key to start frequency analysis...\n";
    cin.get();

    startTime = clock();    //process time start

    //find text words
    vector<char> content_read;
    string word_readed;
    vector<int> frequecy_processed = { 0 };
    array<int, WORD_LEVEL_NUM> words_analysis_array{ 0 };
    char char_read = ' ';

    //get text char one by one
    while (USER_txt.get(char_read))
    {
        //only letters and '-' between letters will be received
        if (isaletter(char_read) || char_read == '-')
        {
            content_read.push_back(char_read);
        }
        else
        {
            //char which is not a letter marks the end of a word
            if (!content_read.empty())    //skip single letter 
            {
                int current_word_frequency = 0;

                //assign letters to make the word
                word_readed.assign(content_read.begin(), content_read.end());
                transform(word_readed.begin(), word_readed.end(), word_readed.begin(), tolower);

                cout << "Finding word \"" << word_readed << "\" \t";
                cout << "Frequency:";

                //check the word's head and find its frequency in COCA list
                for (int whead = WORDS_HEAD_A; whead < WORDS_HEAD_NUM; whead++)
                {
                    if (word_readed[0] == alphabet_str[whead])
                    {
                        cout << COCA_WordsList[whead][word_readed];
                        current_word_frequency = COCA_WordsList[whead][word_readed];

                        //check if the word has been processed
                        if (current_word_frequency == 0)
                        {
                            //addtional check
                            if (word_usual_recheck(word_readed))
                            {
                                word_frequency_analyze(words_analysis_array, WORD_UNDER_4000);
                            }
                            else if (word_form_recheck(word_readed))
                            {
                                current_word_frequency = COCA_WordsList[whead][word_readed];    //try again
                                if (current_word_frequency > 0)
                                {
                                    frequecy_processed.push_back(current_word_frequency);
                                    word_frequency_analyze(words_analysis_array, frequency_classify(current_word_frequency));
                                }
                                else
                                {
                                    // do nothing
                                }
                            }
                            else
                            {
                                word_frequency_analyze(words_analysis_array, WORD_NOT_FOUND_COCA);
                            }
                        }
                        else if (find(frequecy_processed.begin(), frequecy_processed.end(), current_word_frequency)
                            == frequecy_processed.end())
                        {
                            //classify this word and make statistics
                            frequecy_processed.push_back(current_word_frequency);
                            word_frequency_analyze(words_analysis_array, frequency_classify(current_word_frequency));
                        }
                        else
                        {
                            // do nothing
                        }
                    }
                    else
                    {
                        //do nothing
                    }
                }
                cout << endl;

                //classify this word and make statistics
                //word_frequency_analyze(words_analysis_array, frequency_classify(current_word_frequency));
                content_read.clear();
            }
            else
            {
                //do nothing
            }
        }
    }

    endTime = clock();    //process time stop
    process_time = (double)(endTime - startTime) / CLOCKS_PER_SEC;

    //calc whole words processed
    int whole_words_analyzed = 0;
    whole_words_analyzed = accumulate(words_analysis_array.begin(), words_analysis_array.end(), 0);

    //report result
    cout << "\n////////// Report ////////// \n";
    for (int i = 0;i< words_analysis_array.size();i++)
    {
        cout << report_str[i] <<"\t"<< words_analysis_array[i] << " (";
        cout<<fixed<<setprecision(2)<<(float)words_analysis_array[i] * 100 / whole_words_analyzed << "%)" << endl;
    }
    cout << "\nWords totally analyzed: " << whole_words_analyzed << endl;

    //show run time
    cout << "Map build time: " << build_map_time*1000 << "ms.\n";
    cout << "Process time: " << process_time*1000 << "ms.\n";
    cout << "////////////////////////////" << endl;

    //close file
    COCA_txt.close();
    USER_txt.close();

    return 0;
}