记录一个简单的HttpClient抓取页面内容
现如今的网络时代,HTTP协议如此重要,随着java的发展,也越来越多的人采用java直接通过HTTP协议访问网络资源,虽然java.net提供了基本的访问HTTP协议的基本功能,但是对于大部分应用程序来说,仍旧还有许多功能不能够灵活使用;HttpClient是Apache Jakarta Common 下的子项目,一个提供访问HTTP协议的java工具包,提供了更多、更快捷、丰富的方法,HttpClient主要常用的功能有:实现了所有 HTTP 的方法(GET,POST,PUT,HEAD,DELETE 等); 支持重定向;支持HTTPS;支持代理服务器。;因此对于HttpClient就有了解一些的必要了,前一个公司用它来动态登录与之借口的其他系统并抓取数据,当时谢了一个小例子,就是下面的程序,
简单的获取一个网页的内容:
HttpClient client = new HttpClient(); try { GetMethod get = new GetMethod("http://www.baidu.com"); System.out.println("executing request " + get.getURI()); client.executeMethod(get); int statuscode = get.getStatusCode(); System.out.println("status:"+statuscode); if(statuscode == HttpStatus.SC_OK) { html = get.getResponseBodyAsString(); } System.out.println(html); get.releaseConnection(); } catch (Exception e) { // TODO: handle exception }
GetMethod()可以有两种方式,第一种是直接像以上的例子传入整个url地址new GetMethod("http://www.baidu.com");另一种可以先设定host和port,PostMethod post = new PostMethod("/jsp/login.jsp");
client.getHostConfiguration().setHost(host, port);
然后调用httpclient的executeMethod执行。getStateCode获取状态码
另外可以用post方法实现后台模拟登陆:
HttpClient client = new HttpClient(); try { String host = "localhost"; int port = 8080; PostMethod post = new PostMethod("/jsp/login.jsp"); client.getHostConfiguration().setHost(host, port); System.out.println("executing request " + post.getURI()); NameValuePair name1=new NameValuePair("username","000001"); NameValuePair name2=new NameValuePair("password","888888"); post.setRequestBody(new NameValuePair[]{name1,name2}); try { client.executeMethod(post); } catch (Exception e) { e.printStackTrace(); } int statuscode = post.getStatusCode(); System.out.println("status:"+statuscode); if(statuscode == HttpStatus.SC_OK) { html = post.getResponseBodyAsString(); } else if ((statuscode == HttpStatus.SC_MOVED_TEMPORARILY) || (statuscode == HttpStatus.SC_MOVED_PERMANENTLY) || (statuscode == HttpStatus.SC_SEE_OTHER) ||(statuscode == HttpStatus.SC_TEMPORARY_REDIRECT)) { //读取新的URL地址 Header header = post.getResponseHeader("location"); if(header != null) { String newuri = header.getValue(); System.out.println("newuri:"+newuri); if ((newuri == null) || (newuri.equals(""))) { newuri = "/"; } GetMethod get = new GetMethod(newuri); try { client.executeMethod(get); } catch (Exception e) { e.printStackTrace(); } byte[] readbody = get.getResponseBody(); String response = new String(readbody); System.out.println("================================================="); System.out.println(response); System.out.println("================================================="); get.releaseConnection(); } } System.out.println(html); post.releaseConnection(); } catch (Exception e) { // TODO: handle exception } return html;
在使用NameValuePair封装数据,采用舌头setRequestBody()发送数据时,若想在前端页面获取所传送的数据,可采用request.getParameter()的方法获取。