初识网络爬虫
再学习正则表达式之后,可以利用正则表达式进行网络爬虫
首先利用网络编程把网页加载到内存,并且保存到本地
利用正则抽取有用的信息。最终打印输出到控制台
爬取网易首页的所有连接
public class SpiderTest { public static String getUrlContent(String toUrl){ BufferedReader br =null; StringBuilder sb = new StringBuilder(); try { URL url = new URL(toUrl); try { br = new BufferedReader(new InputStreamReader(url.openStream())); String temp = ""; while((temp= br.readLine())!=null){ sb.append(temp); } } catch (IOException e) { e.printStackTrace(); } } catch (MalformedURLException e) { e.printStackTrace(); } return sb.toString(); } public static void main(String[] args) { String str = getUrlContent("https://www.163.com"); //Pattern p = Pattern.compile("<a[\\s\\S]+?</a>");//取得超链接的所有内容 Pattern p2 = Pattern.compile("href=\".+?\""); //Pattern p2 = Pattern.compile("href=\"(.+?)\""); Matcher m = p2.matcher(str); while(m.find()){ System.out.println(m.group()); //System.out.println(m.group(1)); } } }
结果显示:
href="https://ent.163.com/19/0628/07/EIOA5VR000038FO9.html" href="https://ent.163.com/19/0628/07/EIO7VG3U00038FO9.html" href="http://fashion.163.com/" href="http://lady.163.com/photoview/00A70026/115916.html#p=EIOGR4FS00A70026NOS" href="http://lady.163.com/photoview/00A70026/115915.html#p=EIOGI7DD00A70026NOS" href="http://dy.163.com/" href="http://dy.163.com/v2/article/detail/EINGAP5J05259Q0E.html" 后面还有很多。。。。