记录一个简单的HttpClient抓取页面内容

  现如今的网络时代,HTTP协议如此重要,随着java的发展,也越来越多的人采用java直接通过HTTP协议访问网络资源,虽然java.net提供了基本的访问HTTP协议的基本功能,但是对于大部分应用程序来说,仍旧还有许多功能不能够灵活使用;HttpClient是Apache Jakarta Common 下的子项目,一个提供访问HTTP协议的java工具包,提供了更多、更快捷、丰富的方法,HttpClient主要常用的功能有:实现了所有 HTTP 的方法(GET,POST,PUT,HEAD,DELETE 等); 支持重定向;支持HTTPS;支持代理服务器。;因此对于HttpClient就有了解一些的必要了,前一个公司用它来动态登录与之借口的其他系统并抓取数据,当时谢了一个小例子,就是下面的程序,

简单的获取一个网页的内容: 

HttpClient client = new HttpClient();

       try {

           GetMethod get = new GetMethod("http://www.baidu.com");

           System.out.println("executing request " + get.getURI());

           client.executeMethod(get);

          

           int statuscode = get.getStatusCode();

           System.out.println("status:"+statuscode);

          

           if(statuscode == HttpStatus.SC_OK) {

              html = get.getResponseBodyAsString();

           }

           System.out.println(html);

           get.releaseConnection();

       } catch (Exception e) {

           // TODO: handle exception

       }

  GetMethod()可以有两种方式,第一种是直接像以上的例子传入整个url地址new GetMethod("http://www.baidu.com");另一种可以先设定host和port,PostMethod post = new PostMethod("/jsp/login.jsp");

client.getHostConfiguration().setHost(host, port);

然后调用httpclient的executeMethod执行。getStateCode获取状态码

 另外可以用post方法实现后台模拟登陆:

 

HttpClient client = new HttpClient();

       try {

           String host = "localhost";

           int port = 8080;

           PostMethod post = new PostMethod("/jsp/login.jsp");

           client.getHostConfiguration().setHost(host, port);

          

           System.out.println("executing request " + post.getURI());

          

           NameValuePair name1=new NameValuePair("username","000001");

           NameValuePair name2=new NameValuePair("password","888888");

           post.setRequestBody(new NameValuePair[]{name1,name2});

          

           try {

              client.executeMethod(post);

           } catch (Exception e) {

              e.printStackTrace();

           }

          

           int statuscode = post.getStatusCode();

           System.out.println("status:"+statuscode);

          

           if(statuscode == HttpStatus.SC_OK) {

              html = post.getResponseBodyAsString();

           } else if ((statuscode == HttpStatus.SC_MOVED_TEMPORARILY) ||

                    (statuscode == HttpStatus.SC_MOVED_PERMANENTLY) ||

                    (statuscode == HttpStatus.SC_SEE_OTHER) ||(statuscode == HttpStatus.SC_TEMPORARY_REDIRECT)) {

                    //读取新的URL地址

                    Header header = post.getResponseHeader("location");

                   

                    if(header != null) {

                        String newuri = header.getValue();

                        System.out.println("newuri:"+newuri);

                        if ((newuri == null) || (newuri.equals(""))) {

                            newuri = "/";

                        }

                        GetMethod get = new GetMethod(newuri);

                       

                        try {

                            client.executeMethod(get);

                        } catch (Exception e) {

                         e.printStackTrace();

                     }

                        byte[] readbody = get.getResponseBody();

                        String response = new String(readbody);

                       

                        System.out.println("=================================================");

                        System.out.println(response);

                        System.out.println("=================================================");

                       

                        get.releaseConnection();

                    }

           }

           System.out.println(html);

           post.releaseConnection();

       } catch (Exception e) {

           // TODO: handle exception

       }

       return html;

 在使用NameValuePair封装数据,采用舌头setRequestBody()发送数据时,若想在前端页面获取所传送的数据,可采用request.getParameter()的方法获取。 

posted @ 2016-03-19 15:12  vindanear  阅读(603)  评论(0编辑  收藏  举报