WEB数据挖掘（十一）——Aperture数据抽取（7）：在Aperture中重要的API

本人认为，如果介绍Aperture抽象的API，恐怕使人不知所云；抽象的API失去具体的上下文显得有点苍白。人们认识事物的方式从源头上而言总是从特殊到一般，从具体到抽象 。基于此，本文还是实现具有上下文的example

本文先来演示一下一个简单的数据抽取程序，基本流程是：

 １根据InputStream识别文件的mime类型；

 2 根据识别的mime类型获取ExtractorFactory，进一步获取Extractor

３ 调用 Extractor的extract方法填充RDFContainer

 4 输出Model的RDF格式

代码示例如下：

public class ExtractorExample {
    public static void main(String[] args) throws Exception {
        // create a MimeTypeIdentifier
        MimeTypeIdentifier identifier = new MagicMimeTypeIdentifier();
        // create an ExtractorRegistry containing all available
        // ExtractorFactories
        ExtractorRegistry extractorRegistry = new DefaultExtractorRegistry();
        // read as many bytes of the file as desired by the MIME type identifier
        File file = new File("/home/chenying/web/news1.html");
        FileInputStream stream = new FileInputStream(file);
        BufferedInputStream buffer = new BufferedInputStream(stream);
        byte[] bytes = IOUtil.readBytes(buffer, identifier.getMinArrayLength());
        stream.close();
        // let the MimeTypeIdentifier determine the MIME type of this file
        String mimeType = identifier.identify(bytes, file.getPath(), null);
        // skip when the MIME type could not be determined
        if (mimeType == null) {
            System.err.println("MIME type could not be established.");
            return;
        }
        //System.out.println("mimeType:"+mimeType);
        // create the RDFContainer that will hold the RDF model
        URI uri = new URIImpl(file.toURI().toString());
        Model model = RDF2Go.getModelFactory().createModel();
        model.open();
        RDFContainer container = new RDFContainerImpl(model, uri);
        // determine and apply an Extractor that can handle this MIME type
        Set factories=extractorRegistry.getExtractorFactories(mimeType);
        //Set factories = extractorRegistry.get(mimeType);
        if (factories != null && !factories.isEmpty()) {
            // just fetch the first available Extractor
            ExtractorFactory factory = (ExtractorFactory) factories.iterator().next();
            Extractor extractor = factory.get();
 
            // apply the extractor on the specified file
            // (just open a new stream rather than buffer the previous stream)
            stream = new FileInputStream(file);
            buffer = new BufferedInputStream(stream, 8192);
            extractor.extract(uri, buffer, Charset.forName("utf-8"), mimeType, container);
            stream.close();
        }
        // add the MIME type as an additional statement to the RDF model
        container.add(NIE.mimeType, mimeType);
        // report the output to System.out
        //container.getModel().writeTo(new PrintWriter(System.out),Syntax.Ntriples);
        container.getModel().writeTo(new PrintWriter(System.out),Syntax.RdfXml);
    }
}

运行上面的类，会在eclipse的控制台看到Model的Syntax.RdfXml格式的输出，本人的输出如下：

<?xml version="1.0" encoding="UTF-8"?>
<rdf:RDF
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">

<rdf:Description rdf:about="file:/home/chenying/web/news1.html">
    <rdf:type rdf:resource="http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#HtmlDocument"/>
    <plainTextContent xmlns="http://www.semanticdesktop.org/ontologies/2007/01/19/nie#">本文件为测试的解析文件
 </plainTextContent>
    <mimeType xmlns="http://www.semanticdesktop.org/ontologies/2007/01/19/nie#">text/html</mimeType>
</rdf:Description>

</rdf:RDF>

上面的example里面一些核心功能很多是通过手工编写的，如果Aperture的功能是如此不智能，那我们不免为之泄气；不过本人认为，简单的example是了解高级应用的门径，不如此，往往使我们迷失于高级应用的迷宫。

下面本人来点稍微高级点的example，基本流程是：

１创建Model

2 创建RDFContainer包装Ｍodel

3 创建FileSystemDataSource，设置相应属性

４创建FileSystemCrawler并设置DataSource，DataAccessorRegistry和CrawlerHandler（回调处理）

5 FileSystemCrawler的调用crawl方法

代码示例如下：

public class TutorialCrawlingExample {

    public static void main(String[] args) throws Exception {
        // create a new ExampleFileCrawler instance
        TutorialCrawlingExample crawler = new TutorialCrawlingExample();

        if (args.length != 1) {
            System.err.println("Specify the root folder");
            System.exit(-1);
        }      
        // start crawling and exit afterwards
        crawler.doCrawling(new File(args[0]));
    }

    public void doCrawling(File rootFile) throws Exception {
        // create a model that will store the data source configuration
        Model model = RDF2Go.getModelFactory().createModel();
        // open the model
        model.open();
        // .. and wrap it in an RDFContainer
        RDFContainer configuration = new RDFContainerImpl(model, new URIImpl("source:testSource"), false);
        
        // now create the data source
        FileSystemDataSource source = new FileSystemDataSource();
        // and set the configuration container
        source.setConfiguration(configuration);
        // now we can call the type-specific setters in each DataSource class
        source.setRootFolder(rootFile.getAbsolutePath());
        
        // setup a crawler that can handle this type of DataSource
        FileSystemCrawler crawler = new FileSystemCrawler();
        crawler.setDataSource(source);
        crawler.setDataAccessorRegistry(new DefaultDataAccessorRegistry());
        crawler.setCrawlerHandler(new TutorialCrawlerHandler());

        // start crawling
        crawler.crawl();
    }
}

class TutorialCrawlerHandler extends CrawlerHandlerBase {

    // our 'persistent' modelSet
    private ModelSet modelSet;

    public TutorialCrawlerHandler() throws ModelException {
        super (new MagicMimeTypeIdentifier(), new DefaultExtractorRegistry(), 
            new DefaultSubCrawlerRegistry());
        modelSet = RDF2Go.getModelFactory().createModelSet();
        modelSet.open();
    }

    public void crawlStopped(Crawler crawler, ExitCode exitCode) {
        try {
            //modelSet.writeTo(System.out, Syntax.Trix);
            modelSet.writeTo(System.out, Syntax.RdfXml);
            
        }
        catch (Exception e) {
            throw new RuntimeException(e);
        }
        finally {
            modelSet.close();
        }
    }

    public RDFContainer getRDFContainer(URI uri) {
        // we create a new in-memory temporary model for each data source
        Model model = RDF2Go.getModelFactory().createModel(uri);
        // A model needs to be opened before being wrapped in an RDFContainer
        model.open();
        return new RDFContainerImpl(model, uri);
    }
    
    public void objectNew(Crawler crawler, DataObject object) {
        // first we try to extract the information from the binary file
        try {
            processBinary(crawler, object);
        } catch (Exception x) {
            // do some proper logging now in real applications
            x.printStackTrace();
        }
        // then we add this information to our persistent model
        modelSet.addModel(object.getMetadata().getModel());
        // don't forget to dispose of the DataObject
        object.dispose();
    }

    public void objectChanged(Crawler crawler, DataObject object) {
        // first we remove old information about the data object
        modelSet.removeModel(object.getID());
        // then we try to extract metadata and fulltext from the file
        try {
            processBinary(crawler, object);
        } catch (Exception x) {
            // do some proper logging now in real applications
            x.printStackTrace();
        }
        // an then we add the information from the temporary model to our
        // 'persistent' model
        modelSet.addModel(object.getMetadata().getModel());
        // don't forget to dispose of the DataObject
        object.dispose();
    }
    
    public void objectRemoved(Crawler crawler, URI uri) {
        // an object has been removed, we delete it from the rdf store
        modelSet.removeModel(uri);
    }
}

设置参数后，运行上面的类，同样输出Model的Syntax.RdfXml格式

在上面的示例中，我们并没有手动编程方式获取文件的InputStream，这些是Aperture自动完成的，其中TutorialCrawlerHandler是一个内部类，用于FileSystemCrawler对象示例的回调类，这种处理方式与java里面的jaxp规范中的sax方式解析xml文件有点类似，两者感觉是相通的。

我们从TutorialCrawlerHandler类可以发现，Model对象被保存在ModelSet对象里面（大概是集合吧），其实我们在编写回调处理方法时也可以持久化到文件系统，具体详情在此不表。

--------------------------------------------------------------------------- 

本系列WEB数据挖掘系本人原创

作者 博客园 刺猬的温驯 

本文链接 http://www.cnblogs.com/chenying99/archive/2013/06/15/3137067.html

本文版权归作者所有，未经作者同意，严禁转载及用作商业传播，否则将追究法律责任。