云原生之旅 - 9）云原生时代网关的后起之秀Envoy Proxy 和基于Envoy 的 Emissary Ingress
前言
前一篇文章讲述了基于Nginx代理的Kuberenetes Ingress Nginx【云原生时代的网关 Ingress Nginx】这次给大家介绍下基于Envoy的 Emissary Ingress。
首先什么是Enovy？
Envoy 是由 Lyft 开源的高性能网络代理软件，后来捐赠给了 CNCF 基金会，已经毕业于CNCF。 相比于 Nginx、HAProxy 等经典代理软件，Envoy 具备丰富的可观察性和灵活的可扩展性，并且引入了基于 xDS API 的动态配置方案，Envoy 还提供了大量的开箱即用的 Filter 以满足各种场景下流量治理的需求。
Envoy 与 Nginx 代理的区别
- Envoy 对 HTTP/2 的支持比 Nginx 更好，支持包括 upstream 和 downstream在内的双向通信，而 Nginx 只支持 downstream 的连接。
- 高级负载均衡功能是免费的，Nginx 的高级负载均衡功能则需要商业版 Nginx Plus 支持。
- Envoy 支持热更新，Nginx 配置更新之后需要 Reload。
- Envoy 更贴近 Service Mesh 的使用习惯，Nginx 更贴近传统服务的使用习惯。
Envoy 有典型的两种工作模式。一种作为中心代理，代理集群的南北向流量，这种模式下，Envoy 一般就是负载均衡设备或者是 API 网关的基础数据面，比如 Ambassador 现在叫 Emissary，Gloo 都是新兴的开源的基于 Envoy 的开源网关。另一种模式，就是作为业务进程的 Sidecar，当有业务请求访问业务的时候，流量会被劫持到 Sidecar Envoy 当中，之后再被转发给业务进程，典型代表 Istio 和 Linkerd.
今天我们介绍的就是代理南北向流量的网关 Emissary Ingress（原名 Ambassador）。Emissary-ingress已经是CNCF的孵化项目，并且在去年被顶级服务网格项目Linkerd和Istio正式支持。如需集成参考文档。
关键词：基于Enovy的Emissary Ingress实践，Emissary Ingress入门，云原生网关Emissary Ingress，Emissary Ingress实践
为什么选择 Emissary Ingress
https://www.getambassador.io/docs/emissary/latest/about/alternatives/
https://www.getambassador.io/docs/emissary/latest/about/faq/#why-emissary-ingress
功能列表
常见的云原生网关功能都有，像流量管理，限流，熔断，canary release， authentication，详见下面列表。
Reference: https://github.com/emissary-ingress/emissary
- Manage ingress traffic with load balancing, support for multiple protocols (gRPC and HTTP/2, TCP, and web sockets), and Kubernetes integration
- Manage changes to routing with an easy to use declarative policy engine and self-service configuration, via Kubernetes CRDs or annotations
- Secure microservices with authentication, rate limiting, and TLS
- Ensure high availability with sticky sessions, rate limiting, and circuit breaking
- Leverage observability with integrations with Grafana, Prometheus, and Datadog, and comprehensive metrics support
- Enable progressive delivery with canary releases
- Connect service meshes including Consul, Linkerd, and Istio
- Knative serverless integration
安装
使用Terraform Helm Provider
从 emissary-ingress 2.1开始, 它把 CRDs 从Helm Charts移除了, 现在首先需要手动 apply CRDs。
kubectl apply -f https://app.getambassador.io/yaml/emissary/3.2.0/emissary-crds.yaml
所以我做了一个Helm Charts 专门装下CRDs，否则无法全流程安装自动化。
如果不了解Helm Chart 请参考这篇文章【Kubernetes时代的包管理工具 Helm】入门。
resource "helm_release" "emissary_crds" {
name = "emissary-crds"
create_namespace = true # create emissary default namespace `emissary-system`
namespace = local.emissary_ns
chart = "../common/helm/repos/emissary-crds-8.2.0.tgz"
}
CRDs是默认装在`emissary-system` namespace下面的，不建议修改namespace，如果要在不同的Namespace下装多个Emissary ingress， 是可以共用这个CRDs的。
# Install Emissary-ingress from Chart Repository
resource "helm_release" "emissary_ingress" {
name = "emissary-ingress"
repository = "https://app.getambassador.io"
chart = "emissary-ingress"
version = local.chart_version
create_namespace = true
namespace = local.emissary_ns
values = [
templatefile("${local.common_yaml_d}/emissary-ingress-template.yaml", local.emissary_ingress_map)
]
depends_on = [
helm_release.emissary_crds
]
}
最后一部分，也是自制 chart 专门负责config
# This is for install Host/Listener/Mapping/TLSContext from a local custom chart
# also can upload chart to a bucket or a public github for install from a url
# e.g. [Publish to a GCS bucket](https://github.com/hayorov/helm-gcs)
resource "helm_release" "emissary_config" {
name = "emissary-config"
namespace = local.emissary_ns
chart = "../common/helm/repos/emissary-config-8.2.0.tgz"
values = [
templatefile("${local.common_yaml_d}/emissary-listeners-template.yaml", local.emissary_listeners_map),
local.emissary_config_yaml
]
depends_on = [
helm_release.emissary_ingress
]
}
locals 变量
locals {
project_id = "global-sre-dev"
cluster_name = "sre-gke"
cluster_region = "us-central1"
emissary_ns = "emissary"
chart_version = "8.2.0"
common_yaml_d = "../common/helm/yamls"
ambassador_id = "ambassador"
emissary_ingress_map = {
ambassadorID = local.ambassador_id
loadBalancerIP = "35.232.98.249" # Prepare a Static IP first instead to use Ephemeral
replicaCount = 2
minReplicas = 2
maxReplicas = 3
canaryEnabled = false # set to true in Prod
logLevel = "error" # valid log levels are error, warn/warning, info, debug, and trace
endpointEnable = true
endpointName = "my-resolver"
diagnosticsEnable = false
clusterRequestTimeout = 120000 # milliseconds
}
emissary_listeners_map = {
ambassadorID = local.ambassador_id
listenersEnabled = true # custom listeners
}
}
config文件
locals {
emissary_config_yaml = <<-EOT
hosts:
- name: my-host-dev
spec:
ambassador_id:
- ${local.ambassador_id}
hostname: '*.wadexu.cloud'
requestPolicy:
insecure:
action: Redirect
tlsContext:
name: my-tls-context
tlsSecret:
name: tls-secret
namespace: secret
mappings:
- name: my-nginx-mapping
spec:
ambassador_id:
- ${local.ambassador_id}
hostname: dev.wadexu.cloud
prefix: /
service: my-nginx.nginx:80
tlscontexts:
- name: my-tls-context
spec:
ambassador_id:
- ${local.ambassador_id}
hosts:
- "*.wadexu.cloud"
min_tls_version: v1.2
EOT
}
完整代码请参考 my repo
kubectl create secret -n secret tls tls-secret \ --key ./xxx.key \ --cert ./xxx.pem
Install from local， (Optional) 如果要学习自动化Terraform安装，请参考【部署Terrform基础设施代码的自动化利器 Atlantis】
cd terraform_helm_install/dev terraform init terraform plan terraform apply
Install result
% helm list -n emissary-system NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION emissary-crds emissary-system 1 2022-10-20 10:09:30.72553 +0800 CST deployed emissary-crds-8.2.0 3.2.0 % helm list -n emissary NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION emissary-config emissary 1 2022-10-20 10:31:24.819555 +0800 CST deployed emissary-config-8.2.0 3.2.0 emissary-ingress emissary 1 2022-10-20 10:29:33.705888 +0800 CST deployed emissary-ingress-8.2.0 3.2.0
使用 Kustomize
参考我的 quick start
如果不了解 Kustomize， 请移步我这篇文章【不能错过的一款 Kubernetes 应用编排管理神器 Kustomize】
一个集群安装多个Emissary Ingress
我这个例子 This example 展示了 multiple Emissary deployed in one cluster.
在一个集群里安装多个 Emissary 一定要设置 ambassador_id 并且替换 ClusterRoleBinding name， 否则资源冲突。
- emissary-ingress-init: CRDs will be installed.
- emissary-ingress-public: An emissary-ingress with allow list = all (face to internet).
- emissary-ingress-private: Another emissary-ingress with an allow list (restrict connection) installed in same cluster.
Test in local
# apply CRDs first kustomize build emissary-ingress-init/sre-mgmt-dev > ~/init.yaml kubectl apply -f ~/init.yaml # deploy first public Emissary, this allow list = all, face to internet kustomize build emissary-ingress-public/sre-mgmt-dev > ~/emissary_deploy1.yaml kubectl apply -f ~/emissary_deploy1.yaml # deploy second private Emissary with a restrict allow list to access kustomize build emissary-ingress-private/sre-mgmt-dev > ~/emissary_deploy2.yaml kubectl apply -f ~/emissary_deploy2.yaml
通过Terraform安装 Kustomize资源，请参考 my repo
如：
module "example_custom_manifests" {
source = "kbst.xyz/catalog/custom-manifests/kustomization"
version = "0.3.0"
configuration_base_key = "default"
configuration = {
default = {
resources = [
"${path.root}/../../infra/emissary-ingress-init/sre-mgmt-dev"
]
common_labels = {
"env" = "dev"
}
}
}
}
Test
建一个nginx service 测试下
helm install my-nginx bitnami/nginx --set service.type="ClusterIP" -n nginx --create-namespace
curl
% curl https://dev.wadexu.cloud
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
FAQ
1. 这个error 代表 tls-secret 有问题，确保正确创建
error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
2. Connection refused, 最大的可能是 Listeners 没有配置好。
curl: (7) Failed to connect to dev.wadexu.cloud port 443 after 255 ms: Connection refused
3. CRDs 没创建。
│ Error: unable to build kubernetes objects from release manifest: [resource mapping not found for name: "my-resolver" namespace: "emissary-system" from "": no matches for kind "KubernetesEndpointResolver" in version "getambassador.io/v2" │ ensure CRDs are installed first, resource mapping not found for name: "ambassador" namespace: "emissary-system" from "": no matches for kind "Module" in version "getambassador.io/v2" │ ensure CRDs are installed first]
