ICN开发指导
ICN
ICN (github)addresses the overall challenges of edge deployments.
更多了解请参看srini的blogs
ICN all project repo wiew, you can git clone all-projects
only clone icn repo
only clone sd-ewan
SD-EWAN
SD-EWAN(GitHub) means software define edge WAN, it is used to address the network between multi edge clusters or between edge and internet.
SD-EWAN 是ICN的核心组件。
overview:
对于SD-EWAN的整体理解,请参看 srini的两篇blog
Software Defined Edge WAN : SD-EWAN CNF & Cloud Native configuration
Software Defined Edge WAN : Central Control and Traffic Hub
SDEWAN CNF API
hands up:
org example
对于 SD-EWAN 的实践,我们可以从官方的github的test用例来理解。https://github.com/akraino-edge-stack/icn-sdwan/tree/master/platform/test/e2e-test-crd
git clone https://github.com/akraino-edge-stack/icn-sdwan cd icn-sdwan/platform/test/e2e-test-crd ./test.sh
test.sh 的 进入到 edge-a edge-b sdewan-hub三个目录,通过vagrant建立了两个edge cluster a(vagrant脚本)和b的虚拟主机,同时建立了sdewan-hub的一个虚拟主机。
具体开发过程,可以参考 Li Cheng的github https://github.com/cheng1li/sdewan/tree/e6d9c67f342cc1703697b4859d4b242b600684c3
每个edge的目录下面,会启动两个虚拟主机,一个是k8s edge cluster, 另一个是installer, edge cluseter通过installer的虚拟主机来部署的。参考,vagrant启动多虚拟主机
installer的provision过程,会调用installer.sh, 通过ansible部署每个edge cluser, 后面会通过kud(emco)来部署。
kub其实也是通过ansible, 主要是通过kubernetes-sigs/kubespray 定义cluster.yaml 来部署。
在部署的过程中,kubespray/roles/kubernetes/client/tasks/main.yml 会将admin kubeconfig拷贝到ansible主机,同时也会将kubectl binary拷贝到ansible主机。
有关kubespray部署k8s请参考 https://kubernetes.feisky.xyz/setup/cluster/kubespray
CRD开发
请参看Kubebuilder (github)的这篇文章,这篇文章系统的介绍了K8s 的Job开发过程中,涉及到的各种知识。 k8s, 1.6以下,采用kubebuilder V1版本,以上为V2版本,需要迁移。
Vagrant:
licheng同学为k8s的vagrant已经准备了一些脚本,请参考GitHub
ansible:
使用篇:
python脚本化ansible篇:
python 插件扩展篇:
关于ansible自定义lookup_plugins插件实现playbook扩展
step one by one:
-
setup host
# install vagrant on ubuntu18.04 on Host sudo apt update sudo apt install -y libvirt-bin qemu dnsmasq sudo apt install vagrant vagrant plugin install vagrant-libvirt
-
set up icn VM by vagrant
git clone https://gerrit.akraino.org/r/icn git clone https://gerrit.akraino.org/r/icn/sdwan cd icn git grep "vagrant ssh" vagrant up vagrant ssh sudo su make bm_verify_nestedk8s
-
insteall docker
# install docker, ref: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker wget -O- https://get.docker.com/ |bash sudo usermod -aG docker `whoami` # Set up the Docker daemon sudo bash -c "cat > /etc/docker/daemon.json" <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2" } EOF sudo mkdir -p /etc/systemd/system/docker.service.d # Restart Docker sudo systemctl daemon-reload sudo systemctl restart docker sudo systemctl enable docker
-
install kubeadm
# install kubeadm # ref: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl sudo apt-get update && sudo apt-get install -y apt-transport-https curl curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl # [ERROR Swap]: running with swap on is not supported. Please disable swap # ref: https://stackoverflow.com/questions/56287494/why-does-kubeadm-not-start-even-after-disabling-swap sudo swapoff -a API_SERVER_IP=`ip route get 1 | awk '{match($0, /.+src\s([.0-9]+)/, a);print a[1];exit}'`
-
install k8s
# https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/ # for flannel --pod-network-cidr 10.244.0.0/16 , for calico --pod-network-cidr 192.168.64.0/20 # install k8s sudo kubeadm init --pod-network-cidr 10.244.0.0/16 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config TOKEN=`kubeadm token create` kubeadm token list TOKEN_INFO=`kubeadm token create --print-join-command` TOKEN_CA_HASH=`echo $TOKEN_INFO |awk '{match($0, /.+discovery-token-ca-cert-hash\s(.+)/, a);print a[1];exit}'` TOKEN=`echo $TOKEN_INFO | awk '{match($0, /.+--token\s([^ ]+)/, a);print a[1];exit}'` # join any number of worker nodes by running the following on each as root: # kubeadm join $API_SERVER_IP:6443 --token $TOKEN --discovery-token-ca-cert-hash $TOKEN_CA_HASH
-
deploy a pod network to the cluster
# choose one of CNI plugin from https://kubernetes.io/docs/concepts/cluster-administration/addons/ # here we choose flannel, ref https://github.com/coreos/flannel kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml kubectl get daemonset -n kube-system kubectl get pod -n kube-system
-
build OVN4NFV controller
sudo apt install make sudo apt install make-guile sudo apt install jq # https://gist.github.com/Zate/b3c8e18cbb2bbac2976d79525d95f893 # GOURLREGEX='https://dl.google.com/go/go[0-9\.]+\.linux-amd64.tar.gz' url="$(wget -qO- https://golang.org/dl/ | grep -oP '\/dl\/go([0-9\.]+)\.linux-amd64\.tar\.gz' | head -n 1 )" latest="$(echo $url | grep -oP 'go[0-9\.]+' | grep -oP '[0-9\.]+' | head -c -2 )" wget --quiet --continue --show-progress "https://golang.org/dl/go${latest}.linux-amd64.tar.gz" sudo tar -C /usr/local -xzf go"${latest}".linux-amd64.tar.gz mkdir -p ~/go/{bin,pkg,src} echo "export GOPATH=~/go" >> ~/.profile && source ~/.profile echo "export PATH='$PATH':/usr/local/go/bin:$GOPATH/bin" >> ~/.profile && source ~/.profile go get -u github.com/golang/dep/cmd/dep # https://github.com/opnfv/ovn4nfv-k8s-plugin mkdir -p k8s/plugin && cd k8s/plugin # work around for makefile GOPARH var git clone https://github.com/opnfv/ovn4nfv-k8s-plugin.git cd ovn4nfv-k8s-plugin ln -s $GOPATH "$PWD/../../go" make
tips: "export variable …" export specific variables to a sub-make
https://www.gnu.org/software/make/manual/make.html
-
install OVN
# install by k8s, https://github.com/ovn-org/ovn-kubernetes sudo apt-get install apt-transport-https echo "deb http://3.19.28.122/openvswitch/stable /" | sudo tee /etc/apt/sources.list.d/openvswitch.list wget -O - http://3.19.28.122/openvswitch/keyFile | sudo apt-key add - sudo apt-get update sudo apt-get build-dep dkms sudo apt-get install openvswitch-datapath-dkms -y # ---------------------------------------------------------------------------------
# https://github.com/onap/multicloud-k8s/blob/master/kud/deployment_infra/playbooks/configure-ovn.yml # DISTRIB_CODENAME=`grep "DISTRIB_CODENAME" /etc/lsb-release |cut -d "=" -f 2` # https://packages.wand.net.nz/ DISTRIB_CODENAME=$(lsb_release -sc) sudo apt-get install apt-transport-https echo "deb https://packages.wand.net.nz $DISTRIB_CODENAME ovs-2.10" | sudo tee /etc/apt/sources.list.d/openvswitch.list sudo curl https://packages.wand.net.nz/keyring.gpg -o /etc/apt/trusted.gpg.d/wand.gpg sudo apt-get update sudo apt-get install -y openvswitch-common openvswitch-switch sudo apt-get install -y ovn-common ovn-host sudo apt-get install -y ovn-central echo "OVN_CTL_OPTS=\" --db-sb-create-insecure-remote=yes --db-nb-create-insecure-remote=yes\"" | sudo tee -a /etc/default/ovn-central sudo systemctl restart ovn-central sudo systemctl stop ovn-host # https://www.openvswitch.org/support/dist-docs-2.5/IntegrationGuide.md.txt # http://www.openvswitch.org/support/dist-docs/ovs-vswitchd.conf.db.5.html # http://www.openvswitch.org/support/dist-docs/ovs-vsctl.8.txt sudo ovs-vsctl set open_vswitch . external_ids:ovn-remote="\"tcp:$API_SERVER_IP:6642\"" sudo ovs-vsctl get open_vswitch . external_ids:ovn-remote sudo ovs-vsctl set open_vswitch . external_ids:ovn-encap-type="\"geneve\"" # https://stackoverflow.com/questions/39819378/ansible-get-current-target-hosts-ip-address sudo ovs-vsctl set open_vswitch . external_ids:ovn-encap-ip="\"$API_SERVER_IP\"" sudo ovs-vsctl get open_vswitch . external_ids # https://docs.openvswitch.org/en/latest/tutorials/ovs-advanced/#setup sudo ovs-vsctl add-br br-int -- set Bridge br-int fail-mode=secure sudo systemctl start ovn-host
-
install multus
kubectl apply -f https://raw.githubusercontent.com/onap/multicloud-k8s/master/kud/deployment_infra/images/multus-daemonset.yml kubectl describe CustomResourceDefinition network-attachment-definitions.k8s.cni.cncf.io kubectl describe ClusterRole multus kubectl describe ClusterRoleBinding multus kubectl -n kube-system describe ServiceAccount multus kubectl -n kube-system describe ConfigMap multus kubectl -n kube-system get DaemonSet
-
deploy OVN4NFV
# ref: https://github.com/onap/multicloud-k8s # follow the step from: https://github.com/onap/multicloud-k8s/blob/master/kud/deployment_infra/playbooks/configure-ovn4nfv.yml # setp 1: define a CRD network object specification cat > /tmp/ovn4nfvnetwork.yml << EOF apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: ovn-networkobj spec: config: '{ "cniVersion": "0.3.1", "name": "ovn4nfv-k8s-plugin", "type": "ovn4nfvk8s-cni" }' EOF kubectl apply -f /tmp/ovn4nfvnetwork.yml kubectl describe net-attach-def kubectl create namespace operator kubectl label node `hostname` nfnType=operator --overwrite kubectl apply -f https://raw.githubusercontent.com/onap/multicloud-k8s/master/kud/deployment_infra/images/nfn.yml # check status sudo kubectl describe CustomResourceDefinition networks.k8s.plugin.opnfv.org sudo kubectl describe CustomResourceDefinition providernetworks.k8s.plugin.opnfv.org sudo kubectl -n operator describe ServiceAccount k8s-nfn-sa sudo kubectl describe ClusterRole k8s-nfn-cr sudo kubectl describe ClusterRoleBinding k8s-nfn-crb sudo kubectl -n operator describe Service nfn-operator sudo kubectl -n operator describe Deployment nfn-operator sudo kubectl -n operator describe ConfigMap ovn4nfv-cni-config sudo kubectl -n operator describe DaemonSet ovn4nfv-cni sudo kubectl -n operator describe DaemonSet nfn-agent
-
deploy openwrt
# https://github.com/akraino-edge-stack/icn/blob/master/sdwan/test/sdwan-openwrt-ovn.yml # https://github.com/akraino-edge-stack/icn-sdwan/blob/master/platform/test/e2e-test/edge-scripts/setup-cnf.sh mkdir ~/sdwan_setup cd ~/sdwan_setup # https://github.com/akraino-edge-stack/icn-sdwan/blob/master/platform/test/e2e-test/edge-scripts/setup-cnf.sh curl https://raw.githubusercontent.com/akraino-edge-stack/icn-sdwan/master/platform/test/e2e-test/edge-a/scripts/variables -o ./variables source ./variables curl https://raw.githubusercontent.com/akraino-edge-stack/icn-sdwan/master/platform/test/e2e-test/edge-scripts/sdwan_verifier.sh -o sdwan_verifier.sh curl https://raw.githubusercontent.com/akraino-edge-stack/icn-sdwan/master/platform/test/e2e-test/edge-scripts/setup-cnf.sh -o setup-cnf.sh sed -i -e 's/openwrt:test/openwrt:${OPENWRT_V:-test}/' setup-cnf.sh kubectl taint nodes --all node-role.kubernetes.io/master- OPENWRT_V=0.3.0 bash setup-cnf.sh
-
Add a new API(used for demo)
# https://kubernetes.io/zh/docs/tasks/debug-application-cluster/get-shell-running-container/ kubectl exec -it $sdewan_cnf_name sh cd /usr/lib/lua/luci/controller/rest_v1/ ps |grep http # https://oldwiki.archive.openwrt.org/doc/techref/ubus ubus list ubus -v list network.interface ubus call network.interface.wan0 status
PODIP=`kubectl get pod $sdewan_cnf_name -o template='{{ .status.podIP }}'` BASE_URL="http://$PODIP/cgi-bin/luci/" TOKEN=`wget -S --spider --post-data "luci_username=root&luci_password=" $BASE_URL 2>&1 | grep -o "sysauth=\([0-9a-fA-F]\+\)" |cut -d "=" -f2` cat > demo.lua <<<'-- Licensed to the public under the GNU General Public License v2. module("luci.controller.demo", package.seeall) sys = require "luci.sys" ut = require "luci.util" io = require "io" ip = "ip -4 " function index() entry({"admin", "config", "command"}, call("execute")).dependent = false end function trim(s) return s:match("^%s*(.-)%s*$") end function split_and_trim(str, sep) local array = {} local reg = string.format("([^%s]+)", sep) for item in string.gmatch(str, reg) do item_trimed = trim(item) if string.len(item_trimed) > 0 then table.insert(array, item_trimed) end end return array end function execute() local commands = luci.http.formvalue("command") io.stderr:write("Execute command: %s\n" % commands) local command_array = split_and_trim(commands, ";") for index, command in ipairs(command_array) do -- io.stderr:write("%d: [%s]\n" % {index, command}) sys.exec(command) -- sys.exec("uci commit sample") -- sys.exec("/etc/init.d/firewall restart") end luci.http.prepare_content("application/json") luci.http.write_json("{'status':'ok'}") end' scp demo.lua root@$PODIP:/usr/lib/lua/luci/controller wget -S --header="Cookie:sysauth=$TOKEN" --post-data "command=fw3 reload" $BASE_URL/admin/config/command curl -s -X "POST" -H "Cookie:sysauth=$TOKEN" -d "command=fw3 reload" $BASE_URL/admin/config/demo # login pod kubectl exec -it $sdewan_cnf_name sh logread sed -i -e 's/"config", "command"/"config", "demo"/' /usr/lib/lua/luci/controller/demo.lua exit wget -S --header="Cookie:sysauth=$TOKEN" --post-data "command=fw3 reload" $BASE_URL/admin/config/demo curl -s -X "POST" -H "Cookie:sysauth=$TOKEN" -d "command=fw3 reload" $BASE_URL/admin/config/demo
-
add a really interface API:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
1 cat > interface.lua <<EOF 2 -- Licensed to the public under the GNU General Public License v2. 3 4 module("luci.controller.rest_v1.interface", package.seeall) 5 6 NX = require("nixio") 7 json = require "luci.jsonc" 8 io = require "io" 9 sys = require "luci.sys" 10 utils = require "luci.controller.rest_v1.utils" 11 util = require "luci.util" 12 13 fields_table = { 14 {field="ip_address", key="inet addr", type="array"}, 15 {field="ip6_address", key="inet6 addr", type="array"}, 16 {field="mac_address", key="HWaddr"}, 17 {field="received_packets", key="RX packets"}, 18 {field="send_packets", key="TX packets"}, 19 {field="status", key=function(data) if string.match(data, "[%s]UP[%s]") ~= nil then return "UP" else return "DOWN" end end}, 20 } 21 22 interface_validator = { 23 {name="action", required=true, validator=function(value) return utils.in_array(value, {"up", "down"}) end, message="wrong action"} 24 } 25 26 function index() 27 ver = "v1" 28 entry({"sdewan", ver, "interfaces"}, call("handle_request")).leaf = true 29 end 30 31 function get_field(data, key, field_type) 32 if type(key) == "function" then 33 return key(data) 34 end 35 36 local reg = { 37 key .. ":[^%s]+[%s]", 38 key .. " [^%s]+[%s]", 39 key .. ": [^%s]+[%s]", 40 } 41 42 local ret = nil 43 for index=1, #reg do 44 for item in string.gmatch(data, reg[index]) do 45 local value = nil 46 local i,j = string.find(item, key .. ": ") 47 if i ~= nil then 48 value = string.sub(item, j+1, string.len(item)-1) 49 else 50 i,j = string.find(item, key .. ":") 51 if i ~= nil then 52 value = string.sub(item, j+1, string.len(item)-1) 53 else 54 i,j = string.find(item, key .. " ") 55 if i ~= nil then 56 value = string.sub(item, j+1, string.len(item)-1) 57 end 58 end 59 end 60 if value ~= nil then 61 if field_type == "array" then 62 if ret == nil then 63 ret = {value} 64 else 65 ret[#ret+1] = value 66 end 67 else 68 ret = value 69 break 70 end 71 end 72 end 73 end 74 return ret 75 end 76 77 function is_interface_available(ifname) 78 for interface in util.execi("ifconfig | awk '/^[^ \t]+/{print $1}'") do 79 if interface ~= "lo" and interface == ifname then 80 return true 81 end 82 end 83 84 return false 85 end 86 87 function get_interface(interface, bypasscheck) 88 if bypasscheck ~= true then 89 if is_interface_available(interface) == false then 90 return nil 91 end 92 end 93 94 local ret = {} 95 local data = util.exec("ifconfig " .. interface) 96 if data == nil then 97 for j=1, 3 do 98 utils.log("ifconfig failed, retrying ... ") 99 NX.nanosleep(1) 100 data = util.exec("ifconfig " .. interface) 101 if data ~= nil then 102 break 103 end 104 end 105 end 106 ret["name"] = interface 107 for i,v in pairs(fields_table) do 108 local value = get_field(data, v["key"], v["type"]) 109 if value ~= nil then 110 ret[v["field"]] = value 111 end 112 end 113 return ret 114 end 115 116 function get_interfaces() 117 local ret = {} 118 ret["interfaces"] = {} 119 local index = 1 120 for interface in util.execi("ifconfig | awk '/^[^ \t]+/{print $1}'") do 121 if interface ~= "lo" then 122 ret["interfaces"][index] = get_interface(interface, true) 123 index = index + 1 124 end 125 end 126 return ret 127 end 128 129 -- Request Handler 130 function handle_request() 131 local method = utils.get_req_method() 132 if method == "GET" then 133 return getRequest() 134 elseif method == "PUT" then 135 return setRequest() 136 else 137 utils.response_error(405, "Method Not Allowed") 138 end 139 end 140 141 function getRequest() 142 local uri_list = utils.get_URI_list() 143 if uri_list == nil then 144 return 145 end 146 147 if #uri_list == 5 then 148 utils.response_object(get_interfaces()) 149 elseif #uri_list == 6 then 150 local ifname = uri_list[#uri_list] 151 local obj = get_interface(ifname) 152 153 if obj == nil then 154 utils.response_error(404, "Cannot find interface[" .. ifname .. "]" ) 155 return 156 end 157 utils.response_object(obj) 158 else 159 utils.response_error(400, "Bad request URI") 160 return 161 end 162 end 163 164 function setRequest() 165 local uri_list = utils.get_URI_list() 166 if uri_list == nil then 167 return 168 end 169 170 if #uri_list == 6 then 171 local ifname = uri_list[#uri_list] 172 -- local obj = get_interface(ifname) 173 -- if obj == nil then 174 -- utils.response_error(404, "Cannot find interface[" .. ifname .. "]" ) 175 -- return 176 -- end 177 -- check content 178 local body_obj = utils.get_and_validate_body_object(interface_validator) 179 if body_obj == nil then 180 return 181 end 182 183 local action = body_obj["action"] 184 local exec_command = "ifconfig " .. ifname .. " " .. action 185 -- execute command 186 utils.log("Execute Command: %s" % exec_command) 187 sys.exec(exec_command) 188 NX.nanosleep(1) 189 utils.response_success() 190 else 191 utils.response_error(400, "Bad request URI") 192 return 193 end 194 end 195 EOF
-
test the interface API Manual
# copy to CNF POD scp interface.lua root@$PODIP:/usr/lib/lua/luci/controller/rest_v1 # get json respon curl -s -H "Cookie:sysauth=$TOKEN" -H "Content-Type: application/json" $BASE_URL/sdewan/v1/interfaces | jq # debug: kubectl exec -it $sdewan_cnf_name logread
- Add a label for CNF pod
kubectl label pod $sdewan_cnf_name sdewanPurpose=cnf
-
create interface empty CRD
os=$(go env GOOS) arch=$(go env GOARCH) curl -L https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH) | tar -xz -C /tmp/ VURL=`curl -Ls -o /dev/null -w %{url_effective} https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)` VERSION=${VURL##*/} sudo mv /tmp/${VERSION%%.*} /usr/local/kubebuilder echo "export PATH=$PATH:/usr/local/kubebuilder/bin" >> ~/.profile && source ~/.profile # https://kubernetes-sigs.github.io/kustomize/installation/source/ GOBIN=$(go env GOPATH)/bin GO111MODULE=on go get sigs.k8s.io/kustomize/kustomize/v3 sudo apt-get install gcc mkdir sdewan && cd sdewan go mod init sdewan.akraino.org/sdewan kubebuilder init --domain sdewan.akraino.org kubebuilder create api --group batch --version v1alpha1 --kind CniInterface make install make run # it will run, see Makefile # /home/vagrant/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..." # go fmt ./... # go vet ./... # /home/vagrant/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases # go run ./main.go kubectl describe CustomResourceDefinition cniinterfaces.batch.sdewan.akraino.org kubectl apply -f config/samples/ kubectl describe CniInterface kubectl delete -f config/samples/
make uninstall
-
Add base openwrt client and test it.
# currently the CNF is manager openwrt by rust API # we need to implementation openwrt client, ref https://github.com/cheng1li/sdewan/ mkdir openwrt/ curl https://raw.githubusercontent.com/cheng1li/sdewan/master/openwrt/openwrtclient.go -o openwrt/openwrtclient.go curl https://raw.githubusercontent.com/cheng1li/sdewan/8c34d5f2d1da238daf20272b57c4e9b72943c405/openwrt/service.go -o openwrt/service.go curl https://raw.githubusercontent.com/cheng1li/sdewan/8c34d5f2d1da238daf20272b57c4e9b72943c405/openwrt/utils.go openwrt/utils.go mkdir cmd cat > cmd/client.go <<EOF package main import ( "fmt" "flag" "sdewan.akraino.org/sdewan/openwrt" ) func main() { fmt.Println("hello world") podip := flag.String("ip", "127.0.0.1", "A string for pod ip.") flag.Parse() fmt.Println("OpenWrt IP:", *podip) openwrtClient := openwrt.NewOpenwrtClient(*podip, "root", "") fmt.Println("openwrtClient:", openwrtClient) service := openwrt.ServiceClient{OpenwrtClient: openwrtClient} sc, _ := service.GetAvailableServices() fmt.Println("AvailableServices:", sc) } EOF go run cmd/client.go -ip=$(kubectl get pod $sdewan_cnf_name -o template='{{ .status.podIP }}')
-
Add base interfaces client and test it.
cat > openwrt/interfaces.go <<EOF package openwrt import ( "encoding/json" ) const ( interfaceBaseURL = "sdewan/v1/" ) type InterfaceClient struct { OpenwrtClient *openwrtClient } // Interface API struct type Interface struct { Name string `json:"name,omitempty"` ReceivedPackets string `json:"received_packets,omitempty"` SendPackets string `json:"send_packets,omitempty"` Status string `json:"status,omitempty"` MacAddress string `json:"mac_address,omitempty"` IpAddress []string `json:"ip_address,omitempty"` } type AvailableInterfaces struct { Interfaces []Interface `json:"interfaces"` } // get available interfaces func (s *InterfaceClient) GetAvailableInterfaces() (*AvailableInterfaces, error) { response, err := s.OpenwrtClient.Get(interfaceBaseURL + "interfaces") if err != nil { return nil, err } var servs AvailableInterfaces err2 := json.Unmarshal([]byte(response), &servs) if err2 != nil { return nil, err2 } return &servs, nil } EOF cat > cmd/client.go <<EOF package main import ( "fmt" "flag" "sdewan.akraino.org/sdewan/openwrt" ) func main() { fmt.Println("hello world") podip := flag.String("ip", "127.0.0.1", "A string for pod ip.") flag.Parse() fmt.Println("OpenWrt IP:", *podip) openwrtClient := openwrt.NewOpenwrtClient(*podip, "root", "") fmt.Println("openwrtClient:", openwrtClient) service := openwrt.InterfaceClient{OpenwrtClient: openwrtClient} fcs, _ := service.GetAvailableInterfaces() fmt.Println("AvailableInterfaces:", fcs) fmt.Println("Interface 0, name:", fcs.Interfaces[0].Name) } EOF go run cmd/client.go -ip=$(kubectl get pod $sdewan_cnf_name -o template='{{ .status.podIP }}')
-
fill the interface CRD
patch
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
1 cat > /tmp/sdwan_interface.patch << EOF 2 commit fdeff619eb0e4270d25c3f656f72a3378f922e9b 3 Author: vagrant <vagrant@ubuntu> 4 Date: Fri Sep 11 15:57:25 2020 +0000 5 6 add more field for CniInterfaceSpec 7 8 diff --git a/api/v1alpha1/cniinterface_types.go b/api/v1alpha1/cniinterface_types.go 9 index 03623c0..963771f 100644 10 --- a/api/v1alpha1/cniinterface_types.go 11 +++ b/api/v1alpha1/cniinterface_types.go 12 @@ -29,13 +29,22 @@ type CniInterfaceSpec struct { 13 // Important: Run "make" to regenerate code after modifying this file 14 15 // Foo is an example field of CniInterface. Edit CniInterface_types.go to remove/update 16 - Foo string `json:"foo,omitempty"` 17 + Name string `json:"name,omitempty"` 18 + Status string `json:"status,omitempty"` 19 + MacAddress string `json:"mac_address,omitempty"` 20 + IpAddress []string `json:"ip_address,omitempty"` 21 + CnfSelector metav1.LabelSelector `json:"cnfSelector,omitempty"` 22 } 23 24 // CniInterfaceStatus defines the observed state of CniInterface 25 type CniInterfaceStatus struct { 26 // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster 27 // Important: Run "make" to regenerate code after modifying this file 28 + ReceivedPackets string `json:"received_packets,omitempty"` 29 + SendPackets string `json:"send_packets,omitempty"` 30 + Status string `json:"status,omitempty"` 31 + MacAddress string `json:"mac_address,omitempty"` 32 + IpAddress []string `json:"ip_address,omitempty"` 33 } 34 35 // +kubebuilder:object:root=true 36 diff --git a/config/samples/batch_v1alpha1_cniinterface.yaml b/config/samples/batch_v1alpha1_cniinterface.yaml 37 index 0d9ff78..4171ced 100644 38 --- a/config/samples/batch_v1alpha1_cniinterface.yaml 39 +++ b/config/samples/batch_v1alpha1_cniinterface.yaml 40 @@ -4,4 +4,7 @@ metadata: 41 name: cniinterface-sample 42 spec: 43 # Add fields here 44 - foo: bar 45 + cnfSelector: 46 + matchLabels: 47 + sdewanPurpose: cnf 48 + name: eth0 49 diff --git a/controllers/cniinterface_controller.go b/controllers/cniinterface_controller.go 50 index dfd4008..416f89d 100644 51 --- a/controllers/cniinterface_controller.go 52 +++ b/controllers/cniinterface_controller.go 53 @@ -20,11 +20,13 @@ import ( 54 "context" 55 56 "github.com/go-logr/logr" 57 + corev1 "k8s.io/api/core/v1" 58 "k8s.io/apimachinery/pkg/runtime" 59 ctrl "sigs.k8s.io/controller-runtime" 60 "sigs.k8s.io/controller-runtime/pkg/client" 61 62 batchv1alpha1 "sdewan.akraino.org/sdewan/api/v1alpha1" 63 + "sdewan.akraino.org/sdewan/openwrt" 64 ) 65 66 // CniInterfaceReconciler reconciles a CniInterface object 67 @@ -38,10 +40,48 @@ type CniInterfaceReconciler struct { 68 // +kubebuilder:rbac:groups=batch.sdewan.akraino.org,resources=cniinterfaces/status,verbs=get;update;patch 69 70 func (r *CniInterfaceReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) { 71 - _ = context.Background() 72 - _ = r.Log.WithValues("cniinterface", req.NamespacedName) 73 + ctx := context.Background() 74 + log := r.Log.WithValues("cniinterface", req.NamespacedName) 75 76 // your logic here 77 + var cniInterface batchv1alpha1.CniInterface 78 + if err := r.Get(ctx, req.NamespacedName, &cniInterface); err != nil { 79 + log.Error(err, "unable to fetch CniInterface") 80 + // we'll ignore not-found errors, since they can't be fixed by an immediate 81 + // requeue (we'll need to wait for a new notification), and we can get them 82 + // on deleted requests. 83 + return ctrl.Result{}, client.IgnoreNotFound(err) 84 + } 85 + 86 + podList := &corev1.PodList{} 87 + label := cniInterface.Spec.CnfSelector.MatchLabels["sdewanPurpose"] 88 + err := r.Client.List(ctx, podList, client.MatchingLabels{"sdewanPurpose": label}) 89 + if err != nil { 90 + log.Error(err, "Failed to get pod list") 91 + return ctrl.Result{}, err 92 + } 93 + 94 + fcs := &openwrt.AvailableInterfaces{} 95 + for _, pod := range podList.Items { 96 + openwrtClient := openwrt.NewOpenwrtClient(pod.Status.PodIP, "root", "") 97 + ifc := openwrt.InterfaceClient{OpenwrtClient: openwrtClient} 98 + fcs, _ = ifc.GetAvailableInterfaces() // TODO, support get one interface? 99 + } 100 + 101 + for _, fc := range fcs.Interfaces { 102 + if cniInterface.Spec.Name == fc.Name { 103 + cniInterface.Status.Status = fc.Status 104 + cniInterface.Status.ReceivedPackets = fc.ReceivedPackets 105 + cniInterface.Status.SendPackets = fc.SendPackets 106 + cniInterface.Status.MacAddress = fc.MacAddress 107 + cniInterface.Status.IpAddress = fc.IpAddress 108 + break 109 + } 110 + } 111 + if err := r.Update(ctx, &cniInterface); err != nil { 112 + log.Error(err, "unable to update CniInterface status") 113 + return ctrl.Result{}, err 114 + } 115 116 return ctrl.Result{}, nil 117 } 118 EOF
test
patch -p1 < sdwan_interface.patch make install && make run kubectl apply -f config/samples/ kubectl describe CniInterface kubectl delete -f config/samples/ make uninstall
exmaple: controller-runtime/tree/master/examples/builtins
-
All the code is in github
-
deploy Admission webhook
# https://github.com/kubernetes-sigs/kubebuilder/blob/master/docs/book/src/cronjob-tutorial/webhook-implementation.md kubebuilder create webhook --group batch --version v1alpha1 --kind CniInterface --programmatic-validation --defaulting # https://segmentfault.com/a/1190000020338350
mkdir -p /tmp/k8s-webhook-server/serving-certs/ cp /etc/kubernetes/pki/apiserver.crt /tmp/k8s-webhook-server/serving-certs/tls.crt sudo cp /etc/kubernetes/pki/apiserver.key /tmp/k8s-webhook-server/serving-certs/tls.key sudo chown $(whoami):$(whoami) /tmp/k8s-webhook-server/serving-certs/tls.key CABUNDLE=`kubectl config view --raw -o json | jq -r '.clusters[0].cluster."certificate-authority-data"' | tr -d '"'` sed -i -e "s/caBundle: Cg==/caBundle: $CABUNDLE/" config/webhook/manifests.yaml # sed -i -e "s/caBundle: $CABUNDLE/caBundle: Cg==/" config/webhook/manifests.yaml sed -i -e "/service:/,+2d" config/webhook/manifests.yaml sed -i -e "s/ path: \(.*\)$/url: https:\/\/$API_SERVER_IP:9443\1/" config/webhook/manifests.yaml sed -i -e "s/namespace: system/namespace: kube-system/" config/webhook/manifests.yaml config/webhook/service.yaml # sed -i -e "s/namespace: kube-system/namespace: system/" config/webhook/manifests.yaml config/webhook/service.yaml kubectl apply -f config/webhook/service.yaml kubectl -n kube-system describe Service webhook-service # kubectl describe Service webhook-service kubectl apply -f config/webhook/manifests.yaml kubectl describe ValidatingWebhookConfiguration validating-webhook-configuration kubectl describe MutatingWebhookConfiguration mutating-webhook-configuration make run kubectl apply -f config/samples/ # kubectl delete Service webhook-service kubectl -n kube-system delete Service webhook-service
-
create user
OPERATOR=cnfop ORGANIZATION=icn:cnf:sdewan # https://zhuanlan.zhihu.com/p/43237959 # https://github.com/openssl/openssl/issues/7754 cat /etc/ssl/openssl.cnf | sed "s/RANDFILE\s*=\s*\$ENV::HOME\/\.rnd/#/" dd if=/dev/urandom of=~/.rnd bs=256 count=1 openssl genrsa -out $OPERATOR.key 2048 openssl req -new -key $OPERATOR.key -out $OPERATOR.csr -subj "/CN=$OPERATOR/O=$ORGANIZATION" sudo openssl x509 -req -in $OPERATOR.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out $OPERATOR.crt -days 36500 openssl x509 -in $OPERATOR.crt -noout -text kubectl config set-credentials $OPERATOR --client-certificate=$OPERATOR.crt --client-key=$OPERATOR.key kubectl config get-clusters K8SUSER=`kubectl config get-contexts | tail -n 1 | awk '{print $4}'` CLUSTERNAME=`kubectl config get-contexts | tail -n 1 | awk '{print $3}'` # CLUSTERNAME=`grep -n3 "\- cluster:" ~/.kube/config | grep -o "name:.*" | cut -d " " -f 2` # kubectl config set-context $OPERATOR@$CLUSTERNAME --cluster=$CLUSTERNAME --namespace=default --user=$OPERATOR kubectl config set-context $OPERATOR@$CLUSTERNAME --cluster=$CLUSTERNAME --user=$OPERATOR kubectl config view kubectl config current-context kubectl config use-context $OPERATOR@$CLUSTERNAME kubectl config use-context $OPERATOR@$CLUSTERNAME CURRENTNAME=`kubectl config get-contexts -o name | head -n1` LASTNAME=`kubectl config get-contexts -o name | tail -n1` CRTDATA=`cat $OPERATOR.crt | base64 --wrap=0` KEYDATA=`cat $OPERATOR.key | base64 --wrap=0` sed -i -e "s/client-certificate:.*$OPERATOR.crt/client-certificate-data: $CRTDATA/" ~/.kube/config sed -i -e "s/client-key:\s.*$OPERATOR.key/client-key-data: $KEYDATA/" ~/.kube/config
-
Add/Delete clust role binding
kubectl --context=$LASTNAME create clusterrolebinding ${OPERATOR}-cluster-admin-binding --clusterrole=cluster-admin --user=${OPERATOR}
kubectl --context=$LASTNAME delete clusterrolebinding ${OPERATOR}-cluster-admin-binding
-
Add role binding
# ref: https://kubernetes.io/zh/docs/reference/access-authn-authz/rbac/ # "kind": "batch.sdewan.akraino.org/v1alpha1, Kind=CniInterface" # "resource": {"group":"batch.sdewan.akraino.org","version":"v1alpha1","resource":"cniinterfaces"} OPERATOR=cnfop ORGANIZATION=icn:nfc:sdewan K8SUSER=`grep -n1 "users:" ~/.kube/config |grep -o "name:.*" | cut -d " " -f 2` cat > sdewan-cnf-role-binding.yaml << EOF apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: icn:cnf-operator # namespace: kube-system rules: - apiGroups: - "batch.sdewan.akraino.org" resources: - cniinterfaces verbs: ["*"] # resourceNames: [cniinterfaces-eth0] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: icn:cnf-operator-binding # namespace: kube-system roleRef: apiGroup: "" kind: Role name: icn:cnf-operator subjects: - apiGroup: "" kind: User name: $OPERATOR - kind: Group name: $ORGANIZATION apiGroup: rbac.authorization.k8s.io EOF kubectl --context=$LASTNAME apply -f sdewan-cnf-role-binding.yaml rm sdewan-cnf-role-binding.yaml kubectl --context=$LASTNAME describe Role icn:cnf-operator kubectl --context=$LASTNAME describe Rolebinding icn:cnf-operator-binding kubectl --context=$LASTNAME delete Role icn:cnf-operator kubectl --context=$LASTNAME delete Rolebinding icn:cnf-operator-binding
-
switch context
kubectl config use-context $LASTNAME
-
create CRT and USER/GROUP by kubectl
CSRDATA=$(cat ${OPERATOR}.csr | base64 | tr -d '\n') # Ref: # https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/ # https://github.com/rancher/k3s/issues/1764 # https://thenewstack.io/a-practical-approach-to-understanding-kubernetes-authentication/ # https://dev.to/focusedlabs/provisioning-users-and-groups-for-kubernetes-4251 # https://docs.bitnami.com/tutorials/configure-rbac-in-your-kubernetes-cluster/ cat <<EOF | kubectl apply -f - apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: $OPERATOR spec: username: $OPERATOR groups: - system:authenticated request: ${CSRDATA} usages: - digital signature - key encipherment - client auth EOF kubectl certificate approve $OPERATOR kubectl get csr $OPERATOR -o jsonpath={.status.certificate} | base64 --decode > $OPERATOR.pem USERNAME=$OPERATOR kubectl create namespace $USERNAME cat <<EOF | kubectl apply -f - apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: default name: $USERNAME-namespace rules: - apiGroups: [""] resources: ["*"] verbs: ["*"] EOF cat <<EOF | kubectl apply -f - apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: access-$USERNAME-namespace namespace: $USERNAME subjects: - kind: User name: $USERNAME apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: $USERNAME-namespace apiGroup: rbac.authorization.k8s.io EOF
-
others
REF:
kubespray deploy k8s
使用Kubespray自动化部署Kubernetes 1.13.1
使用 Kubespray 在基础设施或云平台上安装 Kubernetes
openwrt
Run Custom Lua Script as CGI with uhttpd
ubus (OpenWrt micro bus architecture)
Kss PKI CA
Calculating the CA Certificate Hash for Kubeadm
Reconstructing the Join Command for Kubeadm
join cluster after init token expired?
go
get_go.sh (install latest)
update-golang (github repo, wget -O- https://raw.githubusercontent.com/udhos/update-golang/master/update-golang.sh | sudo bash)
github
go-latest (get project latest version in github)
go-github (github v3 API)
ovn
How to use Open Virtual Networking with Kubernetes
YAML
K8S
flannel (github)
LUA 语言 (简易教程)
other
Get final URL after curl is redirected
Submitting ONAP Code Reviews through the Nordix Gerrit
kustomize
Configuring Kubeflow with kfctl and kustomize
Declarative Management of Kubernetes Objects Using Kustomize
Install the kustomize CLI from source without cloning the repo
ksonnet (github)
godoc
sigs.k8s.io/controller-runtime/pkg/client (ListOptions, List, controller/webhook examples/builtins, handler/enqueue_mapped, eventhandler)
k8s.io/client-go/kubernetes type Clientset
k8s.io/api/core/v1 / k8s.io/api/apps/v1
k8s.io/apimachinery/pkg/apis/meta/v1
k8s.io/apimachinery/pkg/fields#Set
sigs.k8s.io/controller-runtime/pkg/builder#Builder.For
k8s.io/apimachinery/pkg/fields#Set
Webhook
kubebuilder2.0学习笔记——搭建和使用 /进阶使用 (更多)
Writing and testing Kubernetes webhooks using Kubebuilder v2
使用 Kubebuilder 创建自定义 K8s AdmissionWebhooks
部署 cert manager (官网)
TIPS:
1 Filed 和 label都可以做selector,但是annotations不可以
概念: Annotations(CN 注解), label, l标签和选择算符
kubectl get pods --field-selector status.phase=Running
Differentiate between Kubernetes labels vs annotations
Kubernetes labels and annotations are both ways of adding metadata to Kubernetes objects. The similarities end there, however. Kubernetes labels allow you to identify, select and operate on Kubernetes objects. Annotations are non-identifying metadata and do none of these things.
Annotations allow you to add non-identifying metadata to Kubernetes objects. Examples include phone numbers of persons responsible for the object or tool information for debugging purposes. In short, annotations can hold any kind of information that is useful and can provide context to DevOps teams.
Conclusion
Kubernetes labels are a great way of identifying and organizing Kubernetes objects and resources. Kubernetes team leads and IT managers are best placed to develop and implement Kubernetes labeling plans. By ensuring labeling conventions are followed across teams, they can tame Kubernetes environments and prevent sprawl. Since Labels make it easier to operate on Kubernetes objects in bulk, adhering to labeling conventions will also make teams more productive in the long run.
你可以使用 Kubernetes 注解为对象附加任意的非标识的元数据。客户端程序(例如工具和库)能够获取这些元数据信息。
你可以使用标签或注解将元数据附加到 Kubernetes 对象。 标签可以用来选择对象和查找满足某些条件的对象集合。 相反,注解不用于标识和选择对象。 注解中的元数据,可以很小,也可以很大,可以是结构化的,也可以是非结构化的,能够包含标签不允许的字符。
注解和标签一样,是键/值对。
2 debug
insatll
# https://github.com/go-delve/delve/blob/master/Documentation/usage/dlv.md # https://github.com/go-delve/delve/blob/master/Documentation/installation/linux/install.md go get github.com/go-delve/delve/cmd/dlv
hack
diff --git a/Makefile b/Makefile index 5da22b5..f571f03 100644 --- a/Makefile +++ b/Makefile @@ -23,6 +23,7 @@ manager: generate fmt vet # Run against the configured Kubernetes cluster in ~/.kube/config run: generate fmt vet manifests + # dlv debug ./main.go go run ./main.go # Install CRDs into a cluster
3. simple "Reconcile
" Pseudocode example:
# https://github.com/kubernetes/kubernetes/issues/84430 func (r *ReconcileObject) Reconcile(req reconcileRequest) { myObj = r.client.Get(context.TODO(), "my Namespace", "my object") myObj.Foo = "bar" sleep(30) r.client.Update(context.TODO(), myObj) }
4. Owns VS Watches
In icn-sdwan projects, it uses Watches for serveral related resources and customize the enqueue handler / event filter. Manual for Watches/Owns from builder doc.
And it tells us:
func (*Builder) Owns ¶ Uses
func (blder *Builder) Owns(object runtime.Object, opts ...OwnsOption) *Builder
Owns defines types of Objects being *generated* by the ControllerManagedBy, and configures the ControllerManagedBy to respond to create / delete / update events by *reconciling the owner object*. This is the equivalent of calling Watches(&source.Kind{Type: <ForType-forInput>}, &handler.EnqueueRequestForOwner{OwnerType: apiType, IsController: true})
func (*Builder) Watches ¶ Uses
func (blder *Builder) Watches(src source.Source, eventhandler handler.EventHandler, opts ...WatchesOption) *Builder
Watches exposes the lower-level ControllerManagedBy Watches functions through the builder. Consider using Owns or For instead of Watches directly. Specified predicates are registered only for given source.
文档建议使用Owns.
handle doc about customize EnqueueRequestsFromMapFunc (源代码):
func EnqueueRequestsFromMapFunc ¶ Uses
func EnqueueRequestsFromMapFunc(mapFN func(MapObject) []reconcile.Request) EventHandler
EnqueueRequestsFromMapFunc enqueues Requests by running a transformation function that outputs a collection of reconcile.Requests on each Event. The reconcile.Requests may be for an arbitrary set of objects defined by some user specified transformation of the source Event. (e.g. trigger Reconciler for a set of objects in response to a cluster resize event caused by adding or deleting a Node)
EnqueueRequestsFromMapFunc is frequently used to fan-out updates from one object to one or more other objects of a differing type.
For UpdateEvents which contain both a new and old object, the transformation function is run on both objects and both sets of Requests are enqueue.
5. 理解 func set
EnqueueRequestsFromMapFunc 内部实现,使用了functions type, 根据/handler/enqueue_mapped.go的源码,实现了一个简化的例子:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
1 package main 2 3 import ( 4 "fmt" 5 ) 6 7 type NamespacedName struct { 8 Namespace string 9 Name string 10 } 11 12 type Request struct { 13 // NamespacedName is the name and namespace of the object to reconcile. 14 NamespacedName 15 } 16 17 type Object interface { 18 GetObjectKind(MapObject) []Request 19 } 20 21 type MapObject struct { 22 Object Object 23 } 24 25 type ObjectIns struct { 26 Name string 27 } 28 29 func (o ObjectIns ) GetObjectKind(i MapObject) []Request { 30 var enqueueRequest []Request 31 return enqueueRequest 32 } 33 34 35 // mapper maps an object to a collection of keys to be enqueued 36 type mapper interface { 37 // Map maps an object 38 Map(MapObject) []Request 39 } 40 41 type EventHandler interface { 42 Create(string, string) 43 } 44 type enqueueRequestsFromMapFunc struct { 45 // Mapper transforms the argument into a slice of keys to be reconciled 46 ToRequests mapper 47 } 48 func (e *enqueueRequestsFromMapFunc) Create(evt string, q string) { 49 fmt.Println(evt, q) 50 } 51 52 53 // var _ EventHandler = &enqueueRequestsFromMapFunc{} 54 55 type toRequestsFunc func(MapObject) []Request 56 57 func EnqueueRequestsFromMapFunc(mapFN func(MapObject) []Request) EventHandler { 58 return &enqueueRequestsFromMapFunc{ 59 ToRequests: toRequestsFunc(mapFN), 60 } 61 } 62 63 func (m toRequestsFunc) Map(i MapObject) []Request { 64 return m(i) 65 } 66 func (m toRequestsFunc) Say() { 67 fmt.Println("Function sets, say hello") 68 } 69 70 func GetToRequestsFunc(r string, crliststruct string) func(h MapObject) []Request { 71 72 return func(h MapObject) []Request { 73 fmt.Println("GetToRequestsFunc r parameter", r) 74 fmt.Println("GetToRequestsFunc crliststruct parameter", crliststruct ) 75 var enqueueRequest []Request 76 req := Request{ 77 NamespacedName: NamespacedName{ 78 Name: "meta.GetName", 79 Namespace: "meta.GetNamespace()", 80 }} 81 enqueueRequest = append(enqueueRequest, req) 82 83 return enqueueRequest 84 } 85 } 86 87 func main() { 88 fmt.Println("Hello, playground") 89 oi := ObjectIns{Name: "ObjectIns"} 90 mo := MapObject{Object: oi} 91 a := GetToRequestsFunc("requert", "test") 92 fmt.Println("=======================") 93 fmt.Println(a(mo)) 94 fmt.Println("=======================") 95 fmt.Println(&a) 96 TR := toRequestsFunc(a) 97 fmt.Println(TR) 98 TR.Say() 99 fmt.Println("=======================") 100 TR.Map(mo) // == a(mo) 101 fmt.Println("=======================") 102 103 eq := &enqueueRequestsFromMapFunc{ToRequests: toRequestsFunc(a)} 104 fmt.Println(eq) 105 // &EnqueueRequestsFromMapFunc{TR: "xxxx"} 106 }
and try it in https://play.golang.org/
6. 关键数据结构 和方法
pkg/handler#EnqueueRequestForOwner
pkg/controller/controllerutil#Object,pkg/runtime#Object
controller-runtime/pkg/event#GenericEvent
controller-runtime/pkg/controller#example-Controller
controller-runtime/pkg/predicate#Predicate
controller-runtime/pkg/handler#example-EnqueueRequestsFromMapFunc
7. kubectl 格式化输出
8. 获取struct的名字
Golang- Getting struct attribute name
1 package main 2 3 import ( 4 "fmt" 5 "reflect" 6 ) 7 8 type MultiQuestions struct { 9 QuestionId int64 10 QuestionType string 11 QuestionText string 12 } 13 14 func (q *MultiQuestions) StructAttrName() string{ 15 v := reflect.ValueOf(q).Elem() 16 t := v.Type() 17 for i := 0; i < t.NumField(); i++ { 18 fmt.Println(reflect.TypeOf(q).Elem().Field(i).Name) 19 } 20 return reflect.TypeOf(q).Elem().Field(0).Name 21 } 22 23 func main() { 24 fmt.Println((&MultiQuestions{}).StructAttrName()) 25 }
9. EnqueueRequestsFromMapFunc系列函数和 shared_informer的关系(L18-23)
pkg/handler/enqueue_mapped.go 实现了怎么从将request 放入到workqueue中。
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
1 E0921 17:51:40.304838 26318 runtime.go:78] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map) 2 goroutine 309 [running]: 3 k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1441b80, 0x173f1a0) 4 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/runtime/runtime.go:74 +0xa6 5 k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) 6 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/runtime/runtime.go:48 +0x89 7 panic(0x1441b80, 0x173f1a0) 8 /usr/local/go/src/runtime/panic.go:969 +0x175 9 sdewan.akraino.org/sdewan/controllers.createCniInterfaces(0x0, 0x0, 0x0, 0x0, 0xc000040f80, 0xd, 0x0, 0x0, 0xc000040f99, 0x7, ...) 10 /home/vagrant/sdwan_setup/sdewan/controllers/cniinterface_controller.go:88 +0xa6 11 sdewan.akraino.org/sdewan/controllers.GetToRequestsFunc2.func1(0x7fe810117918, 0xc000601000, 0x175c5a0, 0xc000601000, 0x0, 0x0, 0x40be1f) 12 /home/vagrant/sdwan_setup/sdewan/controllers/cniinterface_controller.go:348 +0x16f 13 sigs.k8s.io/controller-runtime/pkg/handler.ToRequestsFunc.Map(0xc00061afc0, 0x7fe810117918, 0xc000601000, 0x175c5a0, 0xc000601000, 0x15b4da0, 0x7fe810117918, 0xc000986120) 14 /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/handler/enqueue_mapped.go:104 +0x4e 15 sigs.k8s.io/controller-runtime/pkg/handler.(*EnqueueRequestsFromMapFunc).mapAndEnqueue(0xc000632c80, 0x1798060, 0xc000986120, 0x7fe810117918, 0xc000601000, 0x175c5a0, 0xc0006010 16 00) 17 /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/handler/enqueue_mapped.go:67 +0x7c 18 sigs.k8s.io/controller-runtime/pkg/handler.(*EnqueueRequestsFromMapFunc).Create(0xc000632c80, 0x7fe810117918, 0xc000601000, 0x175c5a0, 0xc000601000, 0x1798060, 0xc000986120) 19 /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/handler/enqueue_mapped.go:47 +0x67 20 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnAdd(0x1786f00, 0xc000632c80, 0x1798060, 0xc000986120, 0xc000632d50, 0x1, 0x1, 0x15b4da0, 0xc000601000) 21 /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/source/internal/eventsource.go:74 +0x1a5 22 k8s.io/client-go/tools/cache.(*processorListener).run.func1() 23 /home/vagrant/go/pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/shared_informer.go:744 +0xc2 24 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0003cf760) 25 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155 +0x5f 26 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000365f60, 0x1753b40, 0xc0009340f0, 0xc00018e001, 0xc000936000) 27 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156 +0xad 28 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0003cf760, 0x3b9aca00, 0x0, 0x1, 0xc000936000) 29 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133 +0xe5 30 k8s.io/apimachinery/pkg/util/wait.Until(...) 31 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90 32 k8s.io/client-go/tools/cache.(*processorListener).run(0xc000730180) 33 /home/vagrant/go/pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/shared_informer.go:738 +0x95 34 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000170140, 0xc0004c89f0) 35 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:73 +0x51 36 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start 37 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:71 +0x65 38 panic: assignment to entry in nil map [recovered] 39 panic: assignment to entry in nil map
10. predicate Create/Update 与 shared_informer的关系(L7-10)
How to dump goroutine stacktraces?
1 runtime/debug.Stack(0x36, 0x0, 0x0) 2 /usr/local/go/src/runtime/debug/stack.go:24 +0x9f 3 runtime/debug.PrintStack() 4 /usr/local/go/src/runtime/debug/stack.go:16 +0x25 5 sdewan.akraino.org/sdewan/controllers.glob..func1(0x7fd35829a0d8, 0xc000384e80, 0x175c5a0, 0xc000384e80, 0x175c5a0) 6 /home/vagrant/sdwan_setup/sdewan/controllers/cniinterface_controller.go:396 +0xbf 7 sigs.k8s.io/controller-runtime/pkg/predicate.Funcs.Create(...) 8 /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/predicate/predicate.go:63 9 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnAdd(0x1786f00, 0xc00020d690, 0x1798060, 0xc000986220, 0xc00020d720, 0x1, 0x1, 0x15b4da0, 0xc000384e80) 10 /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/source/internal/eventsource.go:68 +0x130 11 k8s.io/client-go/tools/cache.(*processorListener).run.func1() 12 /home/vagrant/go/pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/shared_informer.go:744 +0xc2 13 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000280760) 14 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155 +0x5f 15 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000153f60, 0x1753b40, 0xc00037c0f0, 0xc000393201, 0xc0000d94a0) 16 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156 +0xad 17 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000280760, 0x3b9aca00, 0x0, 0x1, 0xc0000d94a0) 18 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133 +0xe5 19 k8s.io/apimachinery/pkg/util/wait.Until(...) 20 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90 21 k8s.io/client-go/tools/cache.(*processorListener).run(0xc000410000) 22 /home/vagrant/go/pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/shared_informer.go:738 +0x95 23 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0001da140, 0xc00038e510) 24 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:73 +0x51 25 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start 26 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:71 +0x65
11. reconcile 与 与 shared_informer的关系(L6-12)
1 goroutine 343 [running]: 2 runtime/debug.Stack(0xc00053a900, 0x0, 0x0) 3 /usr/local/go/src/runtime/debug/stack.go:24 +0x9f 4 runtime/debug.PrintStack() 5 /usr/local/go/src/runtime/debug/stack.go:16 +0x25 6 sdewan.akraino.org/sdewan/controllers.(*CniInterfaceReconciler).Reconcile(0xc0006c7470, 0xc0000eb179, 0x7, 0xc00002a840, 0x13, 0x2a473adbda, 0xc0004506c0, 0xc000450638, 0xc000450630) 7 /home/vagrant/sdwan_setup/sdewan/controllers/cniinterface_controller.go:202 +0x107 8 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000232000, 0x14901e0, 0xc00033a5a0, 0x1567300) 9 /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256 +0x166 10 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000232000, 0x203000) 11 /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232 +0xb0 12 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000232000)
13 /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211 +0x2b 14 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0003d64d0) 15 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155 +0x5f 16 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0003d64d0, 0x1753b40, 0xc0003de360, 0xc00038c001, 0xc00010e240) 17 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156 +0xad 18 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0003d64d0, 0x3b9aca00, 0x0, 0xc000337701, 0xc00010e240) 19 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133 +0xe5 20 k8s.io/apimachinery/pkg/util/wait.Until(0xc0003d64d0, 0x3b9aca00, 0xc00010e240) 21 /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90 +0x4d 22 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1 23 /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:193 +0x32d
12. kubernetes cluster inited by kubeadm does not start automatically after rebooting os #110
node 重启之后需要,After sudo swapoff -a
, I try systemctl enable kubelet
again.
13. missing .rnd
Purpose of RANDFILE in OpenSSL?
Can't load ./.rnd into RNG #7754 (fixed in this issue)
14. 增加用户
Certificate Signing Requests (org)
Provisioning Users and Groups for Kubernetes (step by step, create user by kubectl)
A Practical Approach to Understanding Kubernetes Authentication
CertificateSigningRequest / user creation
15. RBAC
Configure RBAC in your Kubernetes Cluster (step by step)
Authenticating (org)
QUESTION: How to create the k8s users and groups to map to? #139
Using RBAC Authorization (org/ CN 使用 RBAC 鉴权)
Users and RBAC authorizations in Kubernetes (提供了几个很有用的tools,不过有个小错误: s/access_matrix/access-matrix
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
1 # Installation of KREW 2 # Krew is a tool that makes it easy to use kubectl plugins. Krew helps you discover plugins, install and manage them on your machine. It is similar to tools like apt, dnf or brew. Krew is only compatible with kubectl v1.12 and above. 3 4 set -x; cd "$(mktemp -d)" && 5 curl -fsSLO "https://storage.googleapis.com/krew/v0.2.1/krew.{tar.gz,yaml}" && 6 tar zxvf krew.tar.gz && 7 ./krew-"$(uname | tr '[:upper:]' '[:lower:]')_amd64" install \ 8 --manifest=krew.yaml --archive=krew.tar.gz 9 10 export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" 11 12 kubectl krew install access_matrix 13 14 kubectl access-matrix -n my-project-dev --as $OPERATOR # username 15 16 go get -v github.com/aquasecurity/kubectl-who-can 17 kubectl-who-can list pods -n default 18 19 kubectl krew install rbac-lookup 20 kubectl-rbac_lookup $OPERATOR # username 21 22 kubectl-rbac_lookup --kind user 23 kubectl-rbac_lookup --kind group
16. service account
Access Clusters Using the Kubernetes API
在kubernetes 集群(POD)内访问k8s API服务
example
How To Create Kubernetes Service Account For API Access:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
1 kubectl apply -f - <<EOF 2 apiVersion: v1 3 kind: ServiceAccount 4 metadata: 5 name: api-service-account 6 EOF 7 8 kubectl create serviceaccount api-service-account 9 10 cat <<EOF | kubectl apply -f - 11 apiVersion: rbac.authorization.k8s.io/v1 12 kind: ClusterRole 13 metadata: 14 name: api-access 15 rules: 16 - 17 apiGroups: 18 - "" 19 - apps 20 - autoscaling 21 - batch 22 - extensions 23 - policy 24 - rbac.authorization.k8s.io 25 resources: 26 - componentstatuses 27 - configmaps 28 - daemonsets 29 - deployments 30 - events 31 - endpoints 32 - horizontalpodautoscalers 33 - ingress 34 - jobs 35 - limitranges 36 - namespaces 37 - nodes 38 - pods 39 - persistentvolumes 40 - persistentvolumeclaims 41 - resourcequotas 42 - replicasets 43 - replicationcontrollers 44 - serviceaccounts 45 - services 46 verbs: ["*"] 47 - nonResourceURLs: ["*"] 48 verbs: ["*"] 49 --- 50 apiVersion: rbac.authorization.k8s.io/v1 51 kind: ClusterRoleBinding 52 metadata: 53 name: api-access 54 roleRef: 55 apiGroup: rbac.authorization.k8s.io 56 kind: ClusterRole 57 name: api-access 58 subjects: 59 - kind: ServiceAccount 60 name: api-service-account 61 namespace: default 62 EOF 63 64 kubectl describe serviceaccount api-service-account 65 kubectl get serviceaccount api-service-account -o json | jq -Mr '.secrets[].name' 66 SVCSERVICE=$(kubectl get serviceaccount api-service-account -o json | jq -Mr '.secrets[].name') 67 kubectl get secrets ${SVCSERVICE} -o json | jq -Mr '.data.token' | base64 -d 68 TOKEN=`kubectl get secrets ${SVCSERVICE} -o json | jq -Mr '.data.token' | base64 -d` 69 kubectl get endpoints -o custom-columns=MATELABLE:.metadata.labels 70 71 72 SERVERIP=$(kubectl get endpoints kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}') 73 SERVERPORT=$(kubectl get endpoints kubernetes -o jsonpath='{.subsets[0].ports[0].port}') 74 curl -k https://$SERVERIP:$SERVERPORT/api/v1/namespaces 75 curl -k https://$SERVERIP:$SERVERPORT/api/v1/namespaces -H "Authorization: Bearer $TOKEN" 76 curl -k -X GET https://$SERVERIP:$SERVERPORT/api/v1/namespaces/default/pods -H "Authorization: Bearer $TOKEN" |python3 -m json.tool 77 78 kubectl delete ClusterRoleBinding api-access 79 kubectl delete ClusterRole api-access 80 kubectl delete serviceaccount api-service-account
k8s 名词解释:Service Account (包括: 使用Vagrant, 各种部署例子: Spark例子)
16. controller examples in github
indexer GetFieldIndexer().IndexField
Reconcile icn provisioning_controller / pod_controller
17. admission webhook
动态准入控制 (org)
api: k8s.io/api/admissionregistration/ v1beta1
最简单的webhook,Creating Your Own Admission Controller, 纯手工写的, 起了一个server,配置了证书,监听某个地址,返回v1beta1.AdmissionReview对象。
the AdmissionReview
consists of AdmissionRequest
and AdmissionResponse
objects ,详情请参考
serve 将
http请求的body反序列化为
AdmissionReview, 传给了mutate 和 validating 处理函数, 具体实现的一个例子(
深入理解 Kubernetes Admission Webhook)介绍及代码。
红帽 Custom Admission Controllers yaml文件元素具体解读,官方配置解读
经典例子: admission-webhook-example(blog) 和 kube-mutating-webhook-tutorial
但是我们这个kubebuilder帮我们生成的脚手架代码基于controller-runtime的framework,比没有用上面的framwork, 我们想获取跟多的用户信息,很是受限。
// log is for logging in this package. var cniinterfacelog = logf.Log.WithName("cniinterface-resource") func (r *CniInterface) SetupWebhookWithManager(mgr ctrl.Manager) error { return ctrl.NewWebhookManagedBy(mgr). For(r). Complete() }
改代码的官方例子,example-WebhookBuilder,
controller-runtime的NewWebhookManagedBy 返回WebhookBuilder, 实现了 func (*WebhookBuilder) For ¶ Uses 和 func (*WebhookBuilder) Complete ¶ Uses 方法。
WebhookBuilder 的For (传入的参数要实现Validator 接口)方法返回了WebhookBuilder,Complete方法通过registerWebhooks实现了注册了三个webhook, registerDefaultingWebhook, registerValidatingWebhook, registerConversionWebhook。这三个函数最终通过controllerManager 的GetWebhookServer实现了server的Register. 我们看 registerValidatingWebhook流程,通过ValidatingWebhookFor来生成validatingHandler。 validatingHandler的Handle会通过Validator来调用Create,Update和Delete方法。
没有办法,需要自己来定制webhook,重新注册handler。
Manager 提供了GetWebhookServer 方法, 返回webhook.Server, Server提供了Register 方法, 注册handler和对应的url路径。
webhook的Admission的Webhoo就是一个Handler (type HandlerFunc ¶ Uses,参考官方example), 两个参数context.Context 和 type Request ¶ Uses 其返回值是 type Response ¶ Uses。