简单的网络爬虫程序(Web Crawlers)
程序比较简单,但是能体现基本原理。
package com.wxisme.webcrawlers; import java.io.*; import java.net.*; /** * Web Crawlers * @author wxisme * */ public class WebCrawlers { public static void main(String[] args) { URL url = null; try { url = new URL("http://www.baidu.com"); } catch (MalformedURLException e) { System.out.println("域名不合法!"); e.printStackTrace(); } InputStream is = null; try { is = url.openStream(); } catch (IOException e) { e.printStackTrace(); } FileOutputStream fos = null; try { fos = new FileOutputStream("E:\\baidu.txt"); } catch (FileNotFoundException e) { System.out.println("文件创建失败!"); e.printStackTrace(); } //使用转换流设置字符集 BufferedReader br = null; try { br = new BufferedReader(new InputStreamReader( is,"utf-8")); } catch (UnsupportedEncodingException e) { System.out.println("字符集设置失败!"); e.printStackTrace(); } BufferedWriter bw = new BufferedWriter(new OutputStreamWriter( fos)); String msg = null; try { while((msg = br.readLine()) != null) { bw.write(msg); bw.newLine(); } } catch (IOException e) { System.out.println("文件操作失败!"); e.printStackTrace(); } finally { try { bw.flush(); } catch (IOException e) { e.printStackTrace(); } closeAll(is, fos, br, bw); } } public static void closeAll(Closeable ... io) { for(Closeable temp : io) { if(temp != null) { try { temp.close(); } catch (IOException e) { System.out.println("文件关闭失败!"); e.printStackTrace(); } } } } }
作者:Pickle
声明:对于转载分享我是没有意见的,出于对博客园社区和作者的尊重一定要保留原文地址哈。
致读者:坚持写博客不容易,写高质量博客更难,我也在不断的学习和进步,希望和所有同路人一道用技术来改变生活。觉得有点用就点个赞哈。
![](https://images2015.cnblogs.com/blog/735119/201701/735119-20170111112835275-168981902.gif)
![](https://images2015.cnblogs.com/blog/735119/201701/735119-20170111112841431-2047172832.jpg)
![](https://images2015.cnblogs.com/blog/735119/201701/735119-20170111112847494-1544911856.jpg)
![](https://images2015.cnblogs.com/blog/735119/201701/735119-20170111112904385-918783221.jpg)
![](https://images2015.cnblogs.com/blog/735119/201701/735119-20170111120143119-1302805212.jpg)
![](https://images2015.cnblogs.com/blog/735119/201701/735119-20170111112856369-1466718103.jpg)
![](https://images2015.cnblogs.com/blog/735119/201701/735119-20170111112830416-794416355.jpg)
![](https://images2015.cnblogs.com/blog/735119/201701/735119-20170111113919510-1155220901.gif)