Homework3 -Academic Search Engines: Pros and Cons

Academic Search Engines: Pros and Cons

——Microsoft Academic Search vs Google Scholar vs Arnetminer

         In this article, we explore three famous academic search engines - Microsoft Academic Search, Google Scholar and Arnetminer. Comparison between these three engines is conducted in terms of the following features that users of academic search mainly concern:

         1. Publication

         2. Author

         3. Social Network

　　　4. Journal

         5. Conference

         6. Organizations

　　　7. Other features

         The following table shows the summary in terms of the above mentioned features of the investigated search engines. For the sake of brevity, only a subset of the features is included. More detailed introduction to the engines will be discussed later.

Publication

         As the core functionality of an academic search engine, the publication search module must be considered in the first priority. Aimed at mining social relations at one domain, Arnetminer does not provide the function of downloading publication. Thus we will discuss Google Scholar and Microsoft Academic Search more in this part. In testing the performance of a publication search module, we take emphasis on the database coverage.

         We believe that Google Scholar has a larger database and coverage on publication search than Microsoft Academic Search. According to a former investigation (http://www.cnblogs.com/ustc_msra_ase/archive/2010/11/28/1890410.html), Microsoft Academic Search missed a paper accepted to a top-ranked conference while users of Google Scholar can successfully find out this entry. Although this problem appeared to be fixed when we searched for the same paper again, we could easily find out similar cases by checking the citations of a specific publication. For instance, when we enter key words “Improving web search ranking by incorporating user behavior information”, the results show that there are 335 citations in Google Scholar and 114 in Microsoft Academic Search as shown in the following figures. For ArnetMiner, even though it shows there are 332 citations in total, there are only about 120 of the results listed, which is similar to the performance of Microsoft Academic Search. 

The citation numbers, if they are calculated correctly, show that Google Scholar has a significantly larger database than Microsoft Academic Search and ArnetMiner. We selected an item, “Improving Information Retrieval Precision by Finding Related Queries with Similar Information Need Using Information Scent” for example, from the citation list in Google Scholar, and found that it is not indexed in neither Microsoft Academic Search nor ArnetMiner while we can find it in Google Scholar, as shown in the following screen capture.

　　　In short, users may choose Google Scholar and Microsoft Academic Search when they want to download the publications they wanted.Compared to Google Scholar, Microsoft Academic Search has a smaller coverage on publications. Users who only want to follow the general trend of a domain may be satisfied with Microsoft Academic Search. However, researchers are likely to feel unhappy with the lack of some useful works and turn to Google Scholar as a result. We should also notice that Google Scholar and Microsoft Academic Search do not have the database which stores paper. CiteseerX (http://citeseerx.ist.psu.edu ) is an excellent scientific literature digital library which incorporates many useful papers.

Author and Social Network 

         Various kinds of users tend to be interested in different kinds of information about the researchers. A student who is applying for graduate schools may find it useful to know about the recent publications and research interest of his/her prospective advisor. Current researchers may want to keep in touch with those who are conducting research in the same domain. Moreover, social graph and co-author relationships provide the researchers precious information on potential social opportunities.

         In the homepage of Microsoft Academic Search, there are many functions besides publication search. One of them is the author search. From this service, we could get the related information of a researcher, which is not provided in Google Scholar. In contrast, Google Scholar only provides a service that lists the author’s (without name disambiguation) publications.

　　　However, such services provided by Microsoft Academic Search still needs to be improved. Some authors and their organizations are not matched and thus inaccurate information exists. Dan Li, an assistant professor of Tsinghua University, has been mixed up by Microsoft Academic Search with another researcher from Harvard, as shown in the following figures.

         Such problems may be solved by manual correction. Microsoft Academic Search does provide the function for people with Live Accounts to edit the author and paper information. Yet this function is not completed as expected. In the following captured photo, we cannot patently found a way to distinguish two authors from a mixed up entry.

         Aimed at providing comprehensive search and mining services for research social networks, Arnetminer provides more functions to help students as well as researchers to get information about authors. Firstly, the users may easily find the social graph of a specific researcher. The advisor, advisee and coauthor relationship is also clearly displayed.  Secondly, we may also find the social path between two researchers, even in different domains of computer science. This may provide the researchers precious potential social opportunities as well as information when they want to work interdisciplinary. 

         For the standard to evaluate researchers, Microsoft Academic Search provides G-Index and H-index, which is enough for most users. ArnetMiner sets more standards to evaluate the ability of a researcher, such as Activity, Uptrend, Diversity, etc. This may give users a comprehensive view when they want to know more about a specific author. For example, people may be interested in hot prospect of one domain by checking “Uptrend”.

         In sum, ArnetMiner provides fruitful information about authors and social network. Microsoft Academic Search also provides abundant information on this realm. In sum, despite the current limitations such as name disambiguation, both ArnetMiner and Microsoft Academic Search outperform Google Scholar.

Journal & Conference  

These kinds of search are provided by Microsoft Academic Search and ArnetMiner. With the rankings and specific data (publications and some factors), users can readily find out which journals, conferences are top-ranked in certain domains. This will greatly help people in the following ways: (1) For rookies who want to have a fundamental knowledge of one domain, they may find useful information about where to start. (2) For people who want to follow up the current trends of one domain, they may find the paper from the conferences which will help them most while studying.

Organization

　　　An interesting feature provided by Microsoft Academic Search is the “Organization”. For students who want to apply for graduate study in the United States and Europe, they may find this ranking extremely useful. As we know, there are many rankings given by different organizations by different standards. However, these ranking suffer from the problem of inaccurate and subjectivity sometimes. Microsoft Academic Search sorts the Organizations by citations and publications, which gives another way for students to evaluate the research abilities of these organizations. 

Other features

　　　There are also some other features which should be noticed. As we know, Google Scholar also provides Publication Search functionality in all fields (including mathematics, biology, etc), and it can be searched by multi-languages. On the other hand, Microsoft Academic Search suffers from the problem of limited field (only computer science) and English searching. Thus for general users, they would mostly prefer Google Scholar rather than Microsoft Academic Search.

Advises to Microsoft Academic Search

　　　In summary, by compared with Google Scholar and ArnetMiner, we would like to give the following advices to Microsoft Academic Search to improve its performance:

1. Apart from Computer Science and related fields, Microsoft Academic Search should also broaden its service to other subjects.
2. It is more convenient for Microsoft Academic Search to be able to search for paper in various languages.
3. It is better for Microsoft Academic Search to have a larger database.
4. The feature of “hints for input” should be provided, for the convenience of users.
5. More information for the authors might be provided, such as Social Network.
6. The problem of disambiguation is not resolved completely, and requires more manual work.
7. One could sign in an account in Microsoft Academic Search. From the account, MSAS could learn what papers might the person be interested in, then giving useful advices about what to read. With the account, one could also give comments and exchange opinions with other researchers all over the world.

Jiangwei Yu, Kai Hong, Zhirui Wang

                                                                                                                                                                               2011.3.10