varnish反向代理

ports安装varnish, 在/usr/local/etc/rc.d/下有启动脚本，只要在/etc/rc.conf中加入varnishd_enable="YES"即可实现开机启动,可以修改下varnishd脚本符合自己的需要.

man 7 vcl //显示varnish配置语言

The VCL language is a small domain-specific language designed to be used
     to define request handling and document caching policies for the Varnish
     HTTP accelerator.

     When a new configuration is loaded, the varnishd management process
     translates the VCL code to C and compiles it to a shared object which is
     then dynamically linked into the server process.

   Syntax
     The VCL syntax is very simple, and deliberately similar to C and Perl.
     Blocks are delimited by curly braces, statements end with semicolons, and
     comments may be written as in C, C++ or Perl according to your own pref-
     erences.

     In addition to the C-like assignment (=), comparison (==) and boolean (!,
     && and ||) operators, VCL supports regular expression and ACL matching
     using the ~ operator.

     Unlike C and Perl, the backslash (\) character has no special meaning in
     strings in VCL, so it can be freely used in regular expressions without
     doubling.

     Assignments are introduced with the set keyword. There are no user-
     defined variables; values can only be assigned to variables attached to
     backend, request or document objects. Most of these are typed, and the
     values assigned to them must have a compatible unit suffix.

     VCL has if tests, but no loops.

     The contents of another VCL file may be inserted at any point in the code
     by using the include keyword followed by the name of the other file as a
     quoted string.

   Backend declarations
     A backend declaration creates and initializes a named backend object:

backend www {
      set backend.host = "www.example.com";
      set backend.port = "http";
}

     The backend object can later be used to select a backend at request time:

if (req.http.host ~ "^(www.)?example.com$") {
      set req.backend = www;
}

   ACLs
     An ACL declaration creates and initializes a named access control list
     which can later be used to match client addresses:

acl local {
      "locahost"; /* myself */
      "10.0.0.1"/8; /* and everyone on the local network */
      ! "10.0.0.23"; /* except for the dialin router */
}

     If an ACL entry specifies a host name which Varnish is unable to resolve,
     it will match any address it is compared to. Consequently, if it is pre-
     ceded by a negation mark, it will reject any address it is compared to,
     which may not be what you intended. If the entry is enclosed in paren-
     theses, however, it will simply be ignored.

     To match an IP address against an ACL, simply use the match operator:

if (client.ip ~ local) {
      pipe;
}

   Functions
     The following built-in functions are available:

     regsub(str, regex, sub)
      Returns a copy of str with all occurrences of the regular expres-
      sion regex replaced with sub. Within sub, $0 (which can also be
      spelled &) is replaced with the entire matched string, and $n is
      replaced with the contents of subgroup n in the matched string.

     purge_url(regex)
      Purge all objects in cache whose URLs match regex.

   Subroutines
     A subroutine is used to group code for legibility or reusability:

sub pipe_if_local {
      if (client.ip ~ local) {
   pipe;
      }
}

     Subroutines in VCL do not take arguments, nor do they return values.

     If multiple subroutines with the same name are defined, they are concate-
     nated in the order in which the appear in the source.

     To call a subroutine, use the call keyword followed by the subroutine's
     name:

call pipe_if_local;

     There are a number of special subroutines which hook into the Varnish
     workflow. These subroutines may inspect and manipulate HTTP headers and
     various other aspects of each request, and to a certain extent decide how
     the request should be handled. Each subroutine terminates by calling one
     of a small number of keywords which indicates the desired outcome.

     vcl_recv
      Called at the beginning of a request, after the complete request
      has been received and parsed. Its purpose is to decide whether
      or not to serve the request, how to do it, and, if applicable,
      which backend to use.

      The vcl_recv subroutine may terminate with one of the following
      keywords:

      error code [reason]
       Return the specified error code to the client and abandon
       the request.

      pass    Switch to pass mode. Control will eventually pass to
       vcl_pass.

      pipe    Switch to pipe mode. Control will eventually pass to
       vcl_pipe.

      lookup Look up the requested object in the cache. Control will
       eventually pass to vcl_hit or vcl_miss, depending on
       whether the object is in the cache.

     vcl_pipe
      Called upon entering pipe mode. In this mode, the request is
      passed on to the backend, and any further data from either client
      or backend is passed on unaltered until either end closes the
      connection.

      The vcl_pipe subroutine may terminate with one of the following
      keywords:

      error code [reason]
       Return the specified error code to the client and abandon
       the request.

      pipe    Proceed with pipe mode.

     vcl_pass
      Called upon entering pass mode. In this mode, the request is
      passed on to the backend, and the backend's response is passed on
      to the client, but is not entered into the cache. Subsequent
      requests submitted over the same client connection are handled
      normally.

      The vcl_pass subroutine may terminate with one of the following
      keywords:

      error code [reason]
       Return the specified error code to the client and abandon
       the request.

      pass    Proceed with pass mode.

     vcl_hash
      Currently not used. The vcl_hash subroutine may terminate with
      one of the following keywords:

      hash    Proceed.

     vcl_hit
      Called after a cache lookup if the requested document was found
      in the cache.

      The vcl_hit subroutine may terminate with one of the following
      keywords:

      error code [reason]
       Return the specified error code to the client and abandon
       the request.

      pass    Switch to pass mode. Control will eventually pass to
       vcl_pass.

      deliver
       Deliver the cached object to the client. Control will
       eventually pass to vcl_deliver.

     vcl_miss
      Called after a cache lookup if the requested document was not
      found in the cache. Its purpose is to decide whether or not to
      attempt to retrieve the document from the backend, and which
      backend to use.

      The vcl_miss subroutine may terminate with one of the following
      keywords:

      error code [reason]
       Return the specified error code to the client and abandon
       the request.

      pass    Switch to pass mode. Control will eventually pass to
       vcl_pass.

      fetch   Retrieve the requested object from the backend. Control
       will eventually pass to vcl_fetch.

     vcl_fetch
      Called after a document has been successfully retrieved from the
      backend.

      The vcl_fetch subroutine may terminate with one of the following
      keywords:

      error code [reason]
       Return the specified error code to the client and abandon
       the request.

      pass    Switch to pass mode. Control will eventually pass to
       vcl_pass.

      insert Insert the object into the cache, then deliver it to the
       client. Control will eventually pass to vcl_deliver.

     vcl_deliver
      Called before a cached object is delivered to the client.

      The vcl_deliver subroutine may terminate with one of the follow-
      ing keywords:

      error code [reason]
       Return the specified error code to the client and abandon
       the request.

      deliver
       Deliver the object to the client.

     vcl_timeout
      Called by the reaper thread shortly before a cached document
      reaches its expiry time.

      The vcl_timeout subroutine may terminate with one of the follow-
      ing keywords:

      fetch   Request a fresh copy of the object from the backend.

      discard
       Discard the object.

     vcl_discard
      Called by the reaper thread when a cached document is about to be
      discarded, either because it has expired or because space is run-
      ning low.

      The vcl_discard subroutine may terminate with one of the follow-
      ing keywords:

      discard
       Discard the object.

      keep    Keep the object in cache.

     If one of these subroutines is left undefined or terminates without
     reaching a handling decision, control will be handed over to the builtin
     default. See the EXAMPLES section for a listing of the default code.

   Variables
     Although subroutines take no arguments, the necessary information is made
     available to the handler subroutines through global variables.

     The following variables are always available:

     now   The current time, in seconds since the epoch.

     The following variables are available in backend declarations:

     backend.host
    Host name or IP address of a backend.

     backend.port
    Service name or port number of a backend.

     The following variables are available while processing a request:

     client.ip
    The client's IP address.

     server.ip
    The IP address of the socket on which the client connection was
    received.

     req.request
    The request type (e.g. "GET", "HEAD").

     req.url
    The requested URL.

     req.proto
    The HTTP protocol version used by the client.

     req.backend
    The backend to use to service the request.

     req.http.header
    The corresponding HTTP header.

     The following variables are available while preparing a backend request
     (either for a cache miss or for pass or pipe mode):

     bereq.request
    The request type (e.g. "GET", "HEAD").

     bereq.url
    The requested URL.

     bereq.proto
    The HTTP protocol version used to talk to the server.

     bereq.http.header
    The corresponding HTTP header.

     The following variables are available after the requested object has been
     retrieved from cache or from the backend:

     obj.proto
    The HTTP protocol version used when the object was retrieved.

     obj.status
    The HTTP status code returned by the server.

     obj.response
    The HTTP status message returned by the server.

     obj.valid
    True if the request resulted in a valid HTTP response.

     obj.cacheable
    True if the request resulted in a cacheable response. A response
    is considered cacheable if it is valid (see above), the HTTP status
    code is 200, 203, 300, 301, 302, 404 or 410 and it has a non-zero
    time-to-live when Expires and Cache-Control headers are taken into
    account.

     obj.ttl
    The object's remaining time to live, in seconds.

     obj.lastuse
    The approximate time elapsed since the object was last requests, in
    seconds.

     The following variables are available while preparing a response to the
     client:

     resp.proto
    The HTTP protocol version to use for the response.

     resp.status
    The HTTP status code that will be returned.

     resp.response
    The HTTP status message that will be returned.

     resp.http.header
    The corresponding HTTP header.

     Values may be assigned to variables using the set keyword:

sub vcl_recv {
      # Normalize the Host: header
      if (req.http.host ~ "^(www.)?example.com$") {
   set req.http.host = "www.example.com";
      }
}

     HTTP headers can be removed entirely using the remove keyword:

sub vcl_fetch {
      # Don't cache cookies
      remove obj.http.Set-Cookie;
}

EXAMPLES
     The following code is the equivalent of the default configuration with
     the backend address set to "backend.example.com" and no backend port
     specified.

backend default {
      set backend.host = "backend.example.com";
      set backend.port = "http";
}

sub vcl_recv {
      if (req.request != "GET" && req.request != "HEAD") {
   pipe;
      }
      if (req.http.Expect) {
   pipe;
      }
      if (req.http.Authenticate || req.http.Cookie) {
   pass;
      }
      lookup;
}

sub vcl_pipe {
      pipe;
}

sub vcl_pass {
      pass;
}

sub vcl_hash {
      set req.hash += req.url;
      set req.hash += req.http.host;
      hash;
}

sub vcl_hit {
      if (!obj.cacheable) {
   pass;
      }
      deliver;
}

sub vcl_miss {
      fetch;
}

sub vcl_fetch {
      if (!obj.valid) {
   error;
      }
      if (!obj.cacheable) {
   pass;
      }
      if (obj.http.Set-Cookie) {
   pass;
      }
      insert;
}

sub vcl_deliver {
      deliver;
}

sub vcl_timeout {
      discard;
}

sub vcl_discard {
      discard;
}

     The following example shows how to support multiple sites running on sep-
     arate backends in the same Varnish instance, by selecting backends based
     on the request URL.

backend www {
      set backend.host = "www.example.com";
      set backend.port = "80";
}

backend images {
      set backend.host = "images.example.com";
      set backend.port = "80";
}

sub vcl_recv {
      if (req.http.host ~ "^(www.)?example.com$") {
   set req.http.host = "www.example.com";
   set req.backend = www;
      } elsif (req.http.host ~ "^images.example.com$") {
   set req.backend = images;
      } else {
   error 404 "Unknown virtual host";
      }
}

     The following snippet demonstrates how to force a minimum TTL for all
     documents. Note that this is not the same as setting the default_ttl
     run-time parameter, as that only affects document for which the backend
     did not specify a TTL.

sub vcl_fetch {
      if (obj.ttl < 120s) {
   set obj.ttl = 120s;
      }
}

     The following snippet demonstrates how to force Varnish to cache docu-
     ments even when cookies are present.

sub vcl_recv {
      if (req.request == "GET" && req.http.cookie) {
   lookup;
      }
}

sub vcl_fetch {
      if (obj.http.Set-Cookie) {
   insert;
      }
}

     The following code implements the HTTP PURGE method as used by Squid for
     object invalidation:

acl purge {
   "localhost";
   "10.0.0.1"/8;
}

sub vcl_recv {
      if (req.request == "PURGE") {
   if (!client.ip ~ purge) {
       error 405 "Not allowed.";
   }
   lookup;
      }
}

sub vcl_hit {
      if (req.request == "PURGE") {
   set obj.ttl = 0s;
   error 200 "Purged.";
      }
}

sub vcl_miss {
      if (req.request == "PURGE") {
   error 404 "Not in cache.";
      }
}

#Freebsd

varnish cache 反向代理服务器和http加速器的安装和配置

张映 发表于 2010-04-18

分类目录： cache

一，什么是varnish

Varnish是一款高性能的开源HTTP加速器，挪威最大的在线报纸 Verdens Gang 使用3台Varnish代替了原来的12台Squid，性能比以前更好。
Varnish 的作者Poul-Henning Kamp是FreeBSD的内核开发者之一，他认为现在的计算机比起1975年已经复杂许多。在1975年时，储存媒介只有两种：内存与硬盘。但现在计算机系统的内存除了主存外，还包括了CPU内的L1、L2，甚至有L3快取。硬盘上也有自己的快取装置，因此Squid Cache自行处理物件替换的架构不可能得知这些情况而做到最佳化，但操作系统可以得知这些情况，所以这部份的工作应该交给操作系统处理，这就是 Varnish cache设计架构。
Varnish的理念是好的，但是Varnish还没有稳定到可以完全替代Squid的程度，现在就抛弃Squid选择Varnish是不明智的。

 

二，安装varnish

http://sourceforge.net/projects/varnish/files/
tar zxvf varnish-2.1.tar.gz
cd varnish-2.1
./configure --prefix=/usr/local/varnish
make && make install

三，varnish的配置文件

vi /usr/local/varnish/vcl.conf

输入以下内容

backend testserver {
       set backend.host = "127.0.0.1";
       set backend.port = "10000";
}

backend imageserver {
       set backend.host = "127.0.0.1";
       set backend.port = "10001";
}

acl purge {
    "localhost";
       "127.0.0.1";
       "192.168.1.0"/24;
}

sub vcl_recv {
       if (req.request == "PURGE") {
            if (!client.ip ~ purge) {
                   error 405 "Not allowed.";
                     }
               lookup;
             }

       if (req.http.host ~ "^localhost") {
               set req.backend = testserver;

       } elseif (req.http.host ~ "^127.0.0.1") {
               set req.backend = imageserver;

       } else{
               error 404 "test Cache Server";
             }

           if (req.request != "GET" && req.request != "HEAD") {
                           pipe;
                   }
            if (req.http.Expect) {
                pipe;
            }
            if (req.http.Authenticate || req.http.Cookie || req.url ~ "\.(php|cgi)($|\?)") {
                pass;
            }
                if (req.request == "GET" && req.url ~ "\.(txt|js|jpg|gif|png|bmp|css)$") {
                        lookup;
                }
        lookup;
}

sub vcl_hit {
       if (req.request == "PURGE") {
               set obj.ttl = 0s;
               error 200 "Purged.";
       }
}

sub vcl_miss {
       if (req.request == "PURGE") {
               error 404 "Not in cache.";
       }
}

sub vcl_fetch {
       if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" || obj.http.Cache-Control ~ "private")
       {
              pass;
       }
       if (req.request == "GET" && req.url ~ "\.(txt|js|jpg|gif|png|bmp|css)$") {
               set obj.ttl = 3600s;
       }
       else {
               set obj.ttl = 30d;
       }
}

sub vcl_hash
{
       set req.hash += req.url;
       if (req.http.host)
       {
       set req.hash += req.http.host;
       }else{
     set req.hash += server.ip;
       }
       hash;
}

sub vcl_pipe {
    pipe;
}

sub vcl_pass {
    pass;
}

简单说明一下：

1，设置反向请求的IP，127.0.0.1:10000和127.0.0.1:10001

2，允许缓存管理,localhost,127.0.0.1,192.168.0.*

3,varnish对匹配localhost请求跳到testserver，对匹配127.0.0.1的请求跳 到imageserver，如果都没有，显示test Cache Server

4，Varnish对HTTP协议中的GET、HEAD请求进行缓存，对POST请求透过，让其直接访问后端Web服务器。放过php，cgi不对其进行缓存，只缓存，静态文件

5，对以.txt.js.jpg.gif.png.bmp.css结尾的URL缓存时间设置1小时，对其他的URL缓存时间设置为30天。

四，优化Linux内核参数

vi /etc/sysctl.conf

加入

net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 5000    65000

ldconfig进行重新加载

五，启动varnish和varnishncsa，并查看

ulimit -SHn 51200
/usr/local/varnish/sbin/varnishd -n /var/vcache -f /usr/local/varnish/vcl.conf -a :8080 -s file,/var/vcache/varnish_cache.data,1G -g users -u zhangy -w 30000,51200,10 -T 127.0.0.1:3500 -p client_http11=on
/usr/local/varnish/bin/varnishncsa -n /var/vcache -w /var/vlogs/varnish.log &

解释一下8080是varnish监听的端口，例如127.0.0.1:8080会反向请求127.0.0.1：10001图片服务器，127.0.0.1:3500这个是管理缓存的ip

[zhangy@BlackGhost ~]$ ps -e|grep varnish
3970 ?        00:00:00 varnishd
3971 ?        00:00:00 varnishd
3977 pts/0    00:00:00 varnishncsa

六、通过Varnish管理端口，使用正则表达式批量清除缓存：

1，例：清除类似http://127.0.0.1:8080/00/01的URL地址：

[root@BlackGhost varnish]# /usr/local/varnish/bin/varnishadm -T 127.0.0.1:3500 url.purge /00/

2，例：清除类似http://127.0.0.1:8080/00/0144的URL地址：

[root@BlackGhost varnish]# /usr/local/varnish/bin/varnishadm -T 127.0.0.1:3500 url.purge \d*$

3，例：清除类似http://127.0.0.1:8080/00/01/RwGowEtWvcQAAAAAAAAWHH0Rklg81.gif的url地址

[root@BlackGhost varnish]# /usr/local/varnish/bin/varnishadm -T 127.0.0.1:3500 url.purge gif$

4，清除所有

[root@BlackGhost varnish]# /usr/local/varnish/bin/varnishadm -T 127.0.0.1:3500 url.purge *$

5,怎么测试你知道有没有清除缓存

在地址栏输入http://127.0.0.1:8080/00/01/RwGowEtWvcQAAAAAAAAWHH0Rklg81.gif
然后涮新几次你发现以下内容

清缓存前

说明：请求了8次，后台请求1次，缓存7次

清除缓存
[root@BlackGhost varnish]# /usr/local/varnish/bin/varnishadm -T 127.0.0.1:3500 url.purge gif$

然后在请求http://127.0.0.1:8080/00/01/RwGowEtWvcQAAAAAAAAWHH0Rklg81.gif

你会发现以下内容

缓存清后

说明：请求数变了，缓存数没变，后台请求数变了，说明刚才请缓存是有效果的。

七，进行缓存测试，以及部分参数说明

varnish

上图中199719表示用户请求数；

上图中199704表示用户请求收到数

上图中199688表示这个缓存命中次数，就是缓了多少次

上图中第一个2表示没有缓存数，第一次访问都不会被缓存的，memcache也是

上图中第二个2表示后台处理数，没缓存当然要交到后台处理

收藏、分享这篇文章!

转载请注明
作者:海底苍鹰
地址:http://blog.51yip.com/cache/618.html