[k8s]nginx-ingress配置4/7层测试
基本原理
default-backend提供了2个功能:
1. 404报错页面
2. healthz页面
# Any image is permissable as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
创建svc,外面访问80 映射到容器的8080.
deploy+svc
kubectl create -f default-backend.yaml
nginx-ingress搭建
参考: https://github.com/kubernetes/ingress-nginx/blob/master/deploy/README.md
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/without-rbac.yaml \
| kubectl apply -f -
默认使用的镜像
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0
gcr.io/google_containers/defaultbackend:1.4
docker pull lanny/gcr.io_google_containers_defaultbackend_1.4:v1.4
docker tag lanny/gcr.io_google_containers_defaultbackend_1.4:v1.4 gcr.io/google_containers/defaultbackend:1.4
贴上ingress的yaml
主要修改点:
- 通过18080访问状态页面(ingress-controller的nginx.conf决定)
http://192.168.x,x:18080/nginx_status
- ingress-controller需要开启 hostNetwork: true
便于暴漏ingress的80端口和其他ingress-controller的nginx.conf暴漏的端口
namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
default-backend.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: default-http-backend
labels:
app: default-http-backend
namespace: ingress-nginx
spec:
replicas: 1
template:
metadata:
labels:
app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
# Any image is permissable as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: gcr.io/google_containers/defaultbackend:1.4
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: ingress-nginx
labels:
app: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
app: default-http-backend
without-rbac.yaml
这个yaml官网已经更新了, 多了个
sysctl -w net.core.somaxconn=32768; sysctl -w net.ipv4.ip_local_port_range="1024 65535"
而我这里还没更新.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
hostNetwork: true
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --annotations-prefix=nginx.ingress.kubernetes.io
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
ingress-controller的本质是:
/nginx-ingress-controller 加启动参数
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --annotations-prefix=nginx.ingress.kubernetes.io
tcp-services-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
udp-services-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
启动nginx测试ingress的http 7层负载
kubectl run --image=nginx nginx --replicas=2
kubectl expose deployment nginx --port=80 ## 这里是svc端口,默认和容器的端口一致
nginx-ingress.conf
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: app-nginx-ingress
namespace: default
spec:
rules:
- host: mynginx.maotai.com
http:
paths:
- path: /
backend:
serviceName: nginx
servicePort: 80
注意: ingress虽然调用的是svc,貌似转发是client--nginx--svc--pod; 实际上ingress监控svc 自动将svc下的podip填充到nginx.conf.转发是client--nginx--pod
测试4层负载
参考: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/exposing-tcp-udp-services.md
udp-services-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
data:
53: "kube-system/kube-dns:53"
修改后,apply即可,nginx-ingress可以热更新
kubectl apply -f udp-services-configmap.yaml
$ host -t A nginx.default.svc.cluster.local 192.168.14.132
Using domain server:
Name: 192.168.x.x
Address: 192.168.x.x#53
Aliases:
nginx.default.svc.cluster.local has address 10.254.160.155
另一个tcp的示例
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
data:
2200: "default/gitlab:22"
3306: "kube-public/mysql:3306"
2202: "kube-public/centos:22"
2203: "kube-public/mongodb:27017"
以下是nginx-ingress镜像的dockerfile.进这个容器可以看到
Dockerfile
FROM quay.io/kubernetes-ingress-controller/nginx-amd64:0.30
RUN clean-install \
diffutils \
dumb-init
# Create symlinks to redirect nginx logs to stdout and stderr docker log collector
# This only works if nginx is started with CMD or ENTRYPOINT
RUN mkdir -p /var/log/nginx \
&& ln -sf /dev/stdout /var/log/nginx/access.log \
&& ln -sf /dev/stderr /var/log/nginx/error.log
COPY . /
ENTRYPOINT ["/usr/bin/dumb-init"]
CMD ["/nginx-ingress-controller"]
默认启动后nginx-ingres的nginx.conf
root@n1:/etc/nginx# cat nginx.conf
daemon off;
worker_processes 4;
pid /run/nginx.pid;
worker_rlimit_nofile 15360;
worker_shutdown_timeout 10s ;
events {
multi_accept on;
worker_connections 16384;
use epoll;
}
http {
real_ip_header X-Forwarded-For;
real_ip_recursive on;
set_real_ip_from 0.0.0.0/0;
geoip_country /etc/nginx/GeoIP.dat;
geoip_city /etc/nginx/GeoLiteCity.dat;
geoip_proxy_recursive on;
vhost_traffic_status_zone shared:vhost_traffic_status:10m;
vhost_traffic_status_filter_by_set_key $geoip_country_code country::*;
sendfile on;
aio threads;
aio_write on;
tcp_nopush on;
tcp_nodelay on;
log_subrequest on;
reset_timedout_connection on;
keepalive_timeout 75s;
keepalive_requests 100;
client_header_buffer_size 1k;
client_header_timeout 60s;
large_client_header_buffers 4 8k;
client_body_buffer_size 8k;
client_body_timeout 60s;
http2_max_field_size 4k;
http2_max_header_size 16k;
types_hash_max_size 2048;
server_names_hash_max_size 1024;
server_names_hash_bucket_size 32;
map_hash_bucket_size 64;
proxy_headers_hash_max_size 512;
proxy_headers_hash_bucket_size 64;
variables_hash_bucket_size 128;
variables_hash_max_size 2048;
underscores_in_headers off;
ignore_invalid_headers on;
include /etc/nginx/mime.types;
default_type text/html;
brotli on;
brotli_comp_level 4;
brotli_types application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
gzip on;
gzip_comp_level 5;
gzip_http_version 1.1;
gzip_min_length 256;
gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
gzip_proxied any;
gzip_vary on;
# Custom headers for response
server_tokens on;
# disable warnings
uninitialized_variable_warn off;
# Additional available variables:
# $namespace
# $ingress_name
# $service_name
log_format upstreaminfo '$the_real_ip - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';
map $request_uri $loggable {
default 1;
}
access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
error_log /var/log/nginx/error.log notice;
resolver 192.168.14.2 valid=30s;
# Retain the default nginx handling of requests without a "Connection" header
map $http_upgrade $connection_upgrade {
default upgrade;
''
close;
}
map $http_x_forwarded_for $the_real_ip {
default $remote_addr;
}
# trust http_x_forwarded_proto headers correctly indicate ssl offloading
map $http_x_forwarded_proto $pass_access_scheme {
default $http_x_forwarded_proto;
''
$scheme;
}
map $http_x_forwarded_port $pass_server_port {
default $http_x_forwarded_port;
''
$server_port;
}
map $http_x_forwarded_host $best_http_host {
default $http_x_forwarded_host;
''
$this_host;
}
map $pass_server_port $pass_port {
443 443;
default $pass_server_port;
}
# Obtain best http host
map $http_host $this_host {
default $http_host;
''
$host;
}
server_name_in_redirect off;
port_in_redirect off;
ssl_protocols TLSv1.2;
# turn on session caching to drastically improve performance
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_session_timeout 10m;
# allow configuring ssl session tickets
ssl_session_tickets on;
# slightly reduce the time-to-first-byte
ssl_buffer_size 4k;
# allow configuring custom ssl ciphers
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_prefer_server_ciphers on;
ssl_ecdh_curve auto;
proxy_ssl_session_reuse on;
upstream upstream-default-backend {
# Load balance algorithm; empty for round robin, which is the default
least_conn;
keepalive 32;
server 10.2.98.3:8080 max_fails=0 fail_timeout=0;
}
## start server _
server {
server_name _ ;
listen 80 default_server reuseport backlog=511;
listen [::]:80 default_server reuseport backlog=511;
set $proxy_upstream_name "-";
listen 443 default_server reuseport backlog=511 ssl http2;
listen [::]:443 default_server reuseport backlog=511 ssl http2;
# PEM sha: 479f4653ff7d901e313895dbaafbbe64b0805346
ssl_certificate /ingress-controller/ssl/default-fake-certificate.pem;
ssl_certificate_key /ingress-controller/ssl/default-fake-certificate.pem;
more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
location / {
set $proxy_upstream_name "upstream-default-backend";
set $namespace "";
set $ingress_name "";
set $service_name "";
port_in_redirect off;
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
proxy_set_header ssl-client-cert "";
proxy_set_header ssl-client-verify "";
proxy_set_header ssl-client-dn "";
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_redirect off;
proxy_buffering off;
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
proxy_pass http://upstream-default-backend;
}
# health checks in cloud providers require the use of port 80
location /healthz {
access_log off;
return 200;
}
# this is required to avoid error if nginx is being monitored
# with an external software (like sysdig)
location /nginx_status {
allow 127.0.0.1;
allow ::1;
deny all;
access_log off;
stub_status on;
}
}
## end server _
# default server, used for NGINX healthcheck and access to nginx stats
server {
# Use the port 18080 (random value just to avoid known ports) as default port for nginx.
# Changing this value requires a change in:
# https://github.com/kubernetes/ingress-nginx/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
listen 18080 default_server reuseport backlog=511;
listen [::]:18080 default_server reuseport backlog=511;
set $proxy_upstream_name "-";
location /healthz {
access_log off;
return 200;
}
location /nginx_status {
set $proxy_upstream_name "internal";
vhost_traffic_status_display;
vhost_traffic_status_display_format html;
}
location / {
set $proxy_upstream_name "upstream-default-backend";
proxy_pass http://upstream-default-backend;
}
}
}
stream {
log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;
access_log /var/log/nginx/access.log log_stream;
error_log /var/log/nginx/error.log;
# TCP services
# UDP services
}
多路径ingress
参考: https://github.com/kubernetes-helm/monocular/blob/master/deployment/monocular/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: stultified-snail-monocular
annotations:
ingress.kubernetes.io/rewrite-target: "/"
kubernetes.io/ingress.class: "nginx"
labels:
app: stultified-snail-monocular
chart: "monocular-0.5.0"
release: "stultified-snail"
heritage: "Tiller"
spec:
rules:
- http:
paths:
- backend:
serviceName: stultified-snail-monocular-ui
servicePort: 80
path: /
- backend:
serviceName: stultified-snail-monocular-api
servicePort: 80
path: /api/
host:
DNS Policy导致hostnetwork网络问题
参考: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
遇到个坑: nginx-ingress是hostnetwork模式, 而这种模式的pod的/etc/resolve.conf的dns是继承宿主机的(相当于dokcer run --net=host,共享宿主机网络协议栈,因此也继承了网卡dns),导致访问集群里的 api svc无法解析
By default, DNS policy for a pod is ‘ClusterFirst’. So pods running with hostNetwork cannot resolve DNS names. To have DNS options set along with hostNetwork, you should specify DNS policy explicitly to ‘ClusterFirstWithHostNet’. Update the busybox.yaml as following:
nginx-ingress生产中使用方法(因为我没加证书,所以只能手动引导ingress连http的api)
args:
- /nginx-ingress-controller
- --apiserver-host=http://kube-api-http.kube-public
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --tcp-services-configmap=$(POD_NAMESPACE)/nginx-tcp-ingress-configmap
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- image: busybox
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
name: busybox
restartPolicy: Always
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
使用mysql测试nginx的4层负载
mysql的deploy和4层端口ingress创建
[root@m1 yaml]# cat mysql/mysql-deploy.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456"
[root@m1 yaml]# cat ingress/tcp-services-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
data:
3306: "default/mysql:3306"
注:
1.这里发现apply还有,ingress的nginx.conf配置没发生变化,因为我重启过了启动,重新建了下ingress正常了.
2.ingress创建的端口只在ingress-controller的节点上监听.
nginx配置参考
[root@m1 yaml]# cat ingress/tcp-services-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
data:
3306: "default/mysql:3306"
nginx其他功能参考
错误日志记录
日志json格式
stub_status & 开启认证
404错误页配置,并重定向
某些后缀文件拒绝访问(default.conf)
配置include(简化)
nginx日志json格式
log-format-upstream: '{ "time": "$time_iso8601", "remote_addr": "$proxy_protocol_addr",
"x-forward-for": "$proxy_add_x_forwarded_for", "request_id": "$request_id", "remote_user":
"$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status":
$status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri",
"request_query": "$args", "request_length": $request_length, "duration": $request_time,
"method": "$request_method", "http_referrer": "$http_referer", "http_user_agent":
"$http_user_agent" }'
| 字段名 | 字段解释 |
|---|---|
| $remote_addr, $http_x_forwarded_for | 记录客户端IP地址 |
| $remote_user | 记录客户端用户名称 |
| $request | 记录请求的URL和HTTP协议 |
| $status | 记录请求状态 |
| $body_bytes_sent | 发送给客户端的字节数，不包括响应头的大小； 该变量与Apache模块mod_log_config里的“%B”参数兼容。 |
| $bytes_sent | 发送给客户端的总字节数。 |
| $connection | 连接的序列号。 |
| $connection_requests | 当前通过一个连接获得的请求数量。 |
| $msec | 日志写入时间。单位为秒，精度是毫秒。 |
| $pipe | 如果请求是通过HTTP流水线(pipelined)发送，pipe值为“p”，否则为“.”。 |
| $http_referer | 记录从哪个页面链接访问过来的 |
| $http_user_agent | 记录客户端浏览器相关信息 |
| $request_length | 请求的长度（包括请求行，请求头和请求正文）。 |
| $request_time | 请求处理时间，单位为秒，精度毫秒； 从读入客户端的第一个字节开始，直到把最后一个字符发送给客户端后进行日志写入为止。 |
| $time_iso8601 | ISO8601标准格式下的本地时间。 |
| $time_local | 通用日志格式下的本地时间。 |
nginx快速测试
mkdir -p /data/nginx-html
echo "maotai" > /data/nginx-html/index.html
docker run -d \
--net=host \
--restart=always \
-v /etc/nginx/nginx.conf:/etc/nginx/nginx.conf:ro \
-v /etc/localtime:/etc/localtime:ro \
-v /data/nginx-html:/usr/share/nginx/html \
--name nginx \
nginx
nginx4层负载和7层负载示例
单个端口映射负载:一对多
cat > nginx.conf <<EOF
error_log stderr notice;
worker_processes auto;
events {
multi_accept on;
use epoll;
worker_connections 1024;
}
stream {
upstream kube_apiserver {
least_conn;
server 192.168.8.161:6443;
server 192.168.8.162:6443;
server 192.168.8.163:6443;
}
server {
listen 127.0.0.1:6443;
proxy_pass kube_apiserver;
proxy_timeout 10m;
proxy_connect_timeout 1s;
}
}
EOF
单个端口映射:一对一
cat > nginx.conf <<EOF
error_log stderr notice;
worker_processes auto;
events {
multi_accept on;
use epoll;
worker_connections 1024;
}
stream {
log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;
access_log /var/log/nginx/access.log log_stream;
error_log /var/log/nginx/error.log;
# TCP services
# UDP services
upstream udp-53-kube-system-kube-dns-53 {
server 10.2.54.9:53;
}
server {
listen 53 udp;
listen [::]:53 udp;
proxy_responses 1;
proxy_timeout 600s;
proxy_pass udp-53-kube-system-kube-dns-53;
}
}
EOF
一个tcp&一个udp端口映射
stream {
log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;
access_log /var/log/nginx/access.log log_stream;
error_log /var/log/nginx/error.log;
# TCP services
upstream tcp-3306-default-mysql-3306 {
server 10.2.30.2:3306;
}
server {
listen 3306;
listen [::]:3306;
proxy_timeout 600s;
proxy_pass tcp-3306-default-mysql-3306;
}
# UDP services
upstream udp-53-kube-system-kube-dns-53 {
server 10.2.54.9:53;
}
server {
listen 53 udp;
listen [::]:53 udp;
proxy_responses 1;
proxy_timeout 600s;
proxy_pass udp-53-kube-system-kube-dns-53;
}
}
nginx的proxy-pass7层负载配置
user nginx nginx;
worker_processes auto;
worker_rlimit_nofile 65535;
error_log /usr/local/nginxlogs/error.log;
# pid logs/nginx.pid;
events {
use epoll;
worker_connections 51200;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr $remote_user [$time_local] "$request" $http_host '
'$status $upstream_status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $ssl_protocol $ssl_cipher $upstream_addr '
'$request_time $upstream_response_time';
server_info off;
server_tag off;
server_name_in_redirect off;
access_log /usr/local/tengine-2.1.2/logs/access.log main;
client_max_body_size 80m;
client_header_buffer_size 16k;
large_client_header_buffers 4 16k;
sendfile on;
tcp_nopush on;
keepalive_timeout 65;
server_tokens on;
gzip on;
gzip_min_length 1k;
gzip_buffers 4 16k;
gzip_proxied any;
gzip_http_version 1.1;
gzip_comp_level 3;
gzip_types text/plain application/x-javascript text/css application/xml;
gzip_vary on;
upstream kube-api-vip {
least_conn;
server 192.168.x.20:8080;
server 192.168.x.21:8080;
server 192.168.x.22:8080;
}
server {
listen 80;
server_name kube-api-vip.maotai.net;
proxy_connect_timeout 1s;
# proxy_read_timeout 600;
# proxy_send_timeout 600;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
location / {
proxy_next_upstream error timeout invalid_header http_500 http_503 http_404 http_502 http_504;
proxy_pass http://kube-api-vip;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}
混合:端口映射和7层映射放在一起(这是个ingress的例子)
root@n2:/# cat /etc/nginx/nginx.conf
daemon off;
worker_processes 4;
pid /run/nginx.pid;
worker_rlimit_nofile 15360;
worker_shutdown_timeout 10s ;
events {
multi_accept on;
worker_connections 16384;
use epoll;
}
http {
real_ip_header X-Forwarded-For;
real_ip_recursive on;
set_real_ip_from 0.0.0.0/0;
geoip_country /etc/nginx/GeoIP.dat;
geoip_city /etc/nginx/GeoLiteCity.dat;
geoip_proxy_recursive on;
vhost_traffic_status_zone shared:vhost_traffic_status:10m;
vhost_traffic_status_filter_by_set_key $geoip_country_code country::*;
sendfile on;
aio threads;
aio_write on;
tcp_nopush on;
tcp_nodelay on;
log_subrequest on;
reset_timedout_connection on;
keepalive_timeout 75s;
keepalive_requests 100;
client_header_buffer_size 1k;
client_header_timeout 60s;
large_client_header_buffers 4 8k;
client_body_buffer_size 8k;
client_body_timeout 60s;
http2_max_field_size 4k;
http2_max_header_size 16k;
types_hash_max_size 2048;
server_names_hash_max_size 1024;
server_names_hash_bucket_size 64;
map_hash_bucket_size 64;
proxy_headers_hash_max_size 512;
proxy_headers_hash_bucket_size 64;
variables_hash_bucket_size 128;
variables_hash_max_size 2048;
underscores_in_headers off;
ignore_invalid_headers on;
include /etc/nginx/mime.types;
default_type text/html;
brotli on;
brotli_comp_level 4;
brotli_types application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
gzip on;
gzip_comp_level 5;
gzip_http_version 1.1;
gzip_min_length 256;
gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
gzip_proxied any;
gzip_vary on;
# Custom headers for response
server_tokens on;
# disable warnings
uninitialized_variable_warn off;
# Additional available variables:
# $namespace
# $ingress_name
# $service_name
log_format upstreaminfo '$the_real_ip - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';
map $request_uri $loggable {
default 1;
}
access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
error_log /var/log/nginx/error.log notice;
resolver 192.168.14.2 valid=30s;
# Retain the default nginx handling of requests without a "Connection" header
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
map $http_x_forwarded_for $the_real_ip {
default $remote_addr;
}
# trust http_x_forwarded_proto headers correctly indicate ssl offloading
map $http_x_forwarded_proto $pass_access_scheme {
default $http_x_forwarded_proto;
'' $scheme;
}
map $http_x_forwarded_port $pass_server_port {
default $http_x_forwarded_port;
'' $server_port;
}
map $http_x_forwarded_host $best_http_host {
default $http_x_forwarded_host;
'' $this_host;
}
map $pass_server_port $pass_port {
443 443;
default $pass_server_port;
}
# Obtain best http host
map $http_host $this_host {
default $http_host;
'' $host;
}
server_name_in_redirect off;
port_in_redirect off;
ssl_protocols TLSv1.2;
# turn on session caching to drastically improve performance
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_session_timeout 10m;
# allow configuring ssl session tickets
ssl_session_tickets on;
# slightly reduce the time-to-first-byte
ssl_buffer_size 4k;
# allow configuring custom ssl ciphers
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_prefer_server_ciphers on;
ssl_ecdh_curve auto;
proxy_ssl_session_reuse on;
upstream kube-public-spring-web {
# Load balance algorithm; empty for round robin, which is the default
least_conn;
keepalive 32;
server 10.2.54.4:8080 max_fails=0 fail_timeout=0;
}
upstream upstream-default-backend {
# Load balance algorithm; empty for round robin, which is the default
least_conn;
keepalive 32;
server 10.2.30.3:8080 max_fails=0 fail_timeout=0;
}
## start server _
server {
server_name _ ;
listen 80 default_server reuseport backlog=511;
listen [::]:80 default_server reuseport backlog=511;
set $proxy_upstream_name "-";
listen 443 default_server reuseport backlog=511 ssl http2;
listen [::]:443 default_server reuseport backlog=511 ssl http2;
# PEM sha: 20103470e60aa51135afee9244c7c831559f04b8
ssl_certificate /ingress-controller/ssl/default-fake-certificate.pem;
ssl_certificate_key /ingress-controller/ssl/default-fake-certificate.pem;
more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
location / {
set $proxy_upstream_name "upstream-default-backend";
set $namespace "";
set $ingress_name "";
set $service_name "";
port_in_redirect off;
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
proxy_set_header ssl-client-cert "";
proxy_set_header ssl-client-verify "";
proxy_set_header ssl-client-dn "";
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_redirect off;
proxy_buffering off;
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
proxy_pass http://upstream-default-backend;
}
# health checks in cloud providers require the use of port 80
location /healthz {
access_log off;
return 200;
}
# this is required to avoid error if nginx is being monitored
# with an external software (like sysdig)
location /nginx_status {
allow 127.0.0.1;
allow ::1;
deny all;
access_log off;
stub_status on;
}
}
## end server _
## start server spring.maotai.com
server {
server_name spring.maotai.com ;
listen 80;
listen [::]:80;
set $proxy_upstream_name "-";
location / {
set $proxy_upstream_name "kube-public-spring-web";
set $namespace "kube-public";
set $ingress_name "spring";
set $service_name "spring";
port_in_redirect off;
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
proxy_set_header ssl-client-cert "";
proxy_set_header ssl-client-verify "";
proxy_set_header ssl-client-dn "";
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_redirect off;
proxy_buffering off;
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
proxy_pass http://kube-public-spring-web;
}
}
## end server spring.maotai.com
# default server, used for NGINX healthcheck and access to nginx stats
server {
# Use the port 18080 (random value just to avoid known ports) as default port for nginx.
# Changing this value requires a change in:
# https://github.com/kubernetes/ingress-nginx/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
listen 18080 default_server reuseport backlog=511;
listen [::]:18080 default_server reuseport backlog=511;
set $proxy_upstream_name "-";
location /healthz {
access_log off;
return 200;
}
location /nginx_status {
set $proxy_upstream_name "internal";
vhost_traffic_status_display;
vhost_traffic_status_display_format html;
}
location / {
set $proxy_upstream_name "upstream-default-backend";
proxy_pass http://upstream-default-backend;
}
}
}
stream {
log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;
access_log /var/log/nginx/access.log log_stream;
error_log /var/log/nginx/error.log;
# TCP services
upstream tcp-3306-default-mysql-3306 {
server 10.2.30.2:3306;
}
server {
listen 3306;
listen [::]:3306;
proxy_timeout 600s;
proxy_pass tcp-3306-default-mysql-3306;
}
# UDP services
upstream udp-53-kube-system-kube-dns-53 {
server 10.2.54.9:53;
}
server {
listen 53 udp;
listen [::]:53 udp;
proxy_responses 1;
proxy_timeout 600s;
proxy_pass udp-53-kube-system-kube-dns-53;
}
nginx-ingress-vts里backend显示,但是server zone不显示
原来是本地ie浏览器连接里配了代理导致.
