goreplay(gor) golang 流量拷贝工具试用

1. 项目地址

https://github.com/buger/goreplay

2. 安装

wget https://github.com/buger/goreplay/releases/download/v0.16.1/gor_0.16.1_x64.tar.gz
tar xvf  gor_0.16.1_x64.tar.gz
cp goreplay /usr/bin

3. 简单使用

//  测试环境使用的是nginx 有连个机器一台进行反向代理，具体的安装配置比较简单

a. 进行流量拷贝并输出到标准输出
goreplay  --input-raw :80 --output-stdout
response message
GET / HTTP/1.1
Host: XXXXXXXXX
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cache-Control: max-age=0
If-Modified-Since: Sat, 04 Nov 2017 13:52:14 GMT
If-None-Match: W/"59fdc60e-232"
Proxy-Connection: keep-alive
Upgrade-Insecure-Requests: 1
X-Forwarded-For: XXXXXXXXX
X-Lantern-Version: 4.4.0

b. 流量拷贝其他环境

     goreplay --input-raw :80 --output-http="http://xxxxx:port"
     
c. 流量拷贝存储为本地文件（也可以使用s3存储，开源的有minio）
    
     copy:
     goreplay --input-raw :80 --output-file=requests.gor
     replay:
     goreplay --input-file requests.gor --output-http="http://XXXXXXXX:port" 

4. 可用的插件

a. input 

--input-raw - used to capture HTTP traffic, you should specify IP address or 
interface and application port. More about Capturing and replaying traffic.
--input-file - accepts file which previously was recorded using --output-file. 
More about Saving and Replaying from file
--input-tcp - used by Gor aggregation instance if you decided forward traffic 
from multiple forwarder Gor instances to it. 

b. output

--output-http - replay HTTP traffic to given endpoint, accepts base url
--output-file - records incoming traffic to the file. More about Saving and Replaying from file
--output-tcp - forward incoming data to another Gor instance, used in conjunction with --input-tcp. Read more about Aggregator-forwarder setup.
--output-stdout - used for debugging, outputs all data to stdout.

5. 扩展

Rate limiting

a. Limiting replay using absolute number

# staging.server will not get more than ten requests per second
gor --input-tcp :28020 --output-http "http://staging.com|10"

b. Limiting listener using percentage based limiter

# replay server will not get more than 10% of requests 
# useful for high-load environments
gor --input-raw :80 --output-tcp "replay.local:28020|10%"

c. Consistent limiting based on Header or URL param value

# Limit based on header value
gor --input-raw :80 --output-tcp "replay.local:28020|10%" --http-header-limiter "X-API-KEY: 10%"

# Limit based on header value
gor --input-raw :80 --output-tcp "replay.local:28020|10%" --http-param-limiter "api_key: 10%"


Request filtering

a. Allow url regexp

# only forward requests being sent to the /api endpoint
gor --input-raw :8080 --output-http staging.com --http-allow-url /api

b. Disallow url regexp

# only forward requests NOT being sent to the /api... endpoint
gor --input-raw :8080 --output-http staging.com --http-disallow-url /api

c. Filter based on regexp of header

# only forward requests with an api version of 1.0x
gor --input-raw :8080 --output-http staging.com --http-allow-header api-version:^1\.0\d

# only forward requests NOT containing User-Agent header value "Replayed by Gor"
gor --input-raw :8080 --output-http staging.com --http-disallow-header "User-Agent: Replayed by Gor"

d. Filter based on HTTP method

gor --input-raw :80 --output-http "http://staging.server" \
    --http-allow-method GET \
    --http-allow-method OPTIONS

6. 参考资料

// wiki 
https://github.com/buger/goreplay/wiki
https://github.com/buger/goreplay
// kafka
https://github.com/buger/goreplay/wiki/Streaming-from-and-to-Apache-Kafka
// Distributed-configuration
https://github.com/buger/goreplay/wiki/Distributed-configuration