ICN开发指导

 

ICN  

 ICN （github）addresses the overall challenges of edge deployments. 

更多了解请参看srini的blogs 

ICN all project repo wiew, you can git clone all-projects 

only clone icn repo

only clone sd-ewan

SD-EWAN  

SD-EWAN（GitHub） means software define edge WAN, it is used to address the network between multi edge clusters or between edge and internet.

SD-EWAN  是ICN的核心组件。 

SD-EWAN Scenarios

overview：

对于SD-EWAN的整体理解，请参看 srini的两篇blog

Software Defined Edge WAN : SD-EWAN CNF & Cloud Native configuration   

Software Defined Edge WAN : Central Control and Traffic Hub  

SDEWAN CNF API

hands up: 

org example  

对于 SD-EWAN 的实践，我们可以从官方的github的test用例来理解。https://github.com/akraino-edge-stack/icn-sdwan/tree/master/platform/test/e2e-test-crd

git clone https://github.com/akraino-edge-stack/icn-sdwan
cd icn-sdwan/platform/test/e2e-test-crd
./test.sh

test.sh 的 进入到 edge-a edge-b sdewan-hub三个目录，通过vagrant建立了两个edge cluster a（vagrant脚本）和b的虚拟主机，同时建立了sdewan-hub的一个虚拟主机。

具体开发过程，可以参考 Li Cheng的github https://github.com/cheng1li/sdewan/tree/e6d9c67f342cc1703697b4859d4b242b600684c3

 

每个edge的目录下面，会启动两个虚拟主机，一个是k8s edge cluster， 另一个是installer， edge cluseter通过installer的虚拟主机来部署的。参考，vagrant启动多虚拟主机

installer的provision过程，会调用installer.sh， 通过ansible部署每个edge cluser, 后面会通过kud（emco）来部署。 

kub其实也是通过ansible， 主要是通过kubernetes-sigs/kubespray 定义cluster.yaml 来部署。 

在部署的过程中，kubespray/roles/kubernetes/client/tasks/main.yml 会将admin kubeconfig拷贝到ansible主机，同时也会将kubectl binary拷贝到ansible主机。

有关kubespray部署k8s请参考 https://kubernetes.feisky.xyz/setup/cluster/kubespray 

CRD开发

请参看Kubebuilder （github）的这篇文章，这篇文章系统的介绍了K8s 的Job开发过程中，涉及到的各种知识。 k8s， 1.6以下，采用kubebuilder V1版本，以上为V2版本，需要迁移。

Vagrant:

Vagrant tutorial   

Vagrant Documentation 

licheng同学为k8s的vagrant已经准备了一些脚本，请参考GitHub 

ansible:

使用篇：

各种实例 

非常好的Ansible入门教程（超简单）    

python脚本化ansible篇：

ansible作为python模块库使用的方法实例  

python 插件扩展篇： 

关于ansible自定义lookup_plugins插件实现playbook扩展  

Ansible 插件开发  

Ansible插件扩展

step one by one:

• setup host  

# install vagrant on ubuntu18.04 on Host
sudo apt update
sudo apt install -y libvirt-bin qemu dnsmasq
sudo apt install vagrant
vagrant plugin install vagrant-libvirt

• set up icn VM by vagrant 

git clone https://gerrit.akraino.org/r/icn
git clone https://gerrit.akraino.org/r/icn/sdwan
cd icn
git grep "vagrant ssh"
vagrant up
vagrant ssh
sudo su
make bm_verify_nestedk8s

•  insteall docker

# install docker, ref: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker

wget -O- https://get.docker.com/ |bash
sudo usermod -aG docker  `whoami`

# Set up the Docker daemon
sudo bash -c "cat > /etc/docker/daemon.json" <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

sudo mkdir -p /etc/systemd/system/docker.service.d

# Restart Docker
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl enable docker

• install kubeadm   

# install kubeadm
# ref: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl
sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

# [ERROR Swap]: running with swap on is not supported. Please disable swap
# ref: https://stackoverflow.com/questions/56287494/why-does-kubeadm-not-start-even-after-disabling-swap
sudo swapoff -a

API_SERVER_IP=`ip route get 1 | awk '{match($0, /.+src\s([.0-9]+)/, a);print a[1];exit}'`

• install k8s  

# https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
# for flannel --pod-network-cidr 10.244.0.0/16 ,  for calico --pod-network-cidr 192.168.64.0/20
# install k8s
sudo kubeadm init --pod-network-cidr 10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

TOKEN=`kubeadm token create`
kubeadm token list

TOKEN_INFO=`kubeadm token create --print-join-command`
TOKEN_CA_HASH=`echo $TOKEN_INFO |awk '{match($0, /.+discovery-token-ca-cert-hash\s(.+)/, a);print a[1];exit}'`
TOKEN=`echo $TOKEN_INFO | awk '{match($0, /.+--token\s([^ ]+)/, a);print a[1];exit}'`


# join any number of worker nodes by running the following on each as root:
# kubeadm join $API_SERVER_IP:6443 --token $TOKEN  --discovery-token-ca-cert-hash $TOKEN_CA_HASH

• deploy a pod network to the cluster

# choose one of CNI plugin from https://kubernetes.io/docs/concepts/cluster-administration/addons/
# here we choose flannel, ref https://github.com/coreos/flannel
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

kubectl get daemonset -n kube-system
kubectl get pod -n kube-system

• build OVN4NFV controller

sudo apt install make
sudo apt install make-guile
sudo apt  install jq
# https://gist.github.com/Zate/b3c8e18cbb2bbac2976d79525d95f893
# GOURLREGEX='https://dl.google.com/go/go[0-9\.]+\.linux-amd64.tar.gz'
url="$(wget -qO- https://golang.org/dl/ | grep -oP '\/dl\/go([0-9\.]+)\.linux-amd64\.tar\.gz' | head -n 1 )"
latest="$(echo $url | grep -oP 'go[0-9\.]+' | grep -oP '[0-9\.]+' | head -c -2 )"
wget --quiet --continue --show-progress "https://golang.org/dl/go${latest}.linux-amd64.tar.gz"
sudo tar -C /usr/local -xzf go"${latest}".linux-amd64.tar.gz
mkdir -p ~/go/{bin,pkg,src}
echo "export GOPATH=~/go" >> ~/.profile && source ~/.profile
echo "export PATH='$PATH':/usr/local/go/bin:$GOPATH/bin" >> ~/.profile && source ~/.profile
go get -u github.com/golang/dep/cmd/dep


# https://github.com/opnfv/ovn4nfv-k8s-plugin
mkdir -p k8s/plugin && cd k8s/plugin  # work around for makefile GOPARH var
git clone https://github.com/opnfv/ovn4nfv-k8s-plugin.git
cd ovn4nfv-k8s-plugin
ln -s $GOPATH "$PWD/../../go"
make

tips: "export variable …"  export specific variables to a sub-make

https://www.gnu.org/software/make/manual/make.html 

• install OVN

# install by k8s, https://github.com/ovn-org/ovn-kubernetes
sudo apt-get install apt-transport-https
echo "deb http://3.19.28.122/openvswitch/stable /" |  sudo tee /etc/apt/sources.list.d/openvswitch.list
wget -O - http://3.19.28.122/openvswitch/keyFile |  sudo apt-key add -
sudo apt-get update
sudo apt-get build-dep dkms
sudo apt-get install openvswitch-datapath-dkms -y

# ---------------------------------------------------------------------------------

 

# https://github.com/onap/multicloud-k8s/blob/master/kud/deployment_infra/playbooks/configure-ovn.yml

# DISTRIB_CODENAME=`grep "DISTRIB_CODENAME" /etc/lsb-release |cut -d "=" -f 2`
# https://packages.wand.net.nz/ 
DISTRIB_CODENAME=$(lsb_release -sc)
sudo apt-get install apt-transport-https
echo "deb https://packages.wand.net.nz $DISTRIB_CODENAME ovs-2.10" |  sudo tee /etc/apt/sources.list.d/openvswitch.list
sudo curl https://packages.wand.net.nz/keyring.gpg -o /etc/apt/trusted.gpg.d/wand.gpg
sudo apt-get update

sudo apt-get install -y openvswitch-common openvswitch-switch
sudo apt-get install -y ovn-common ovn-host
sudo apt-get install -y ovn-central

echo "OVN_CTL_OPTS=\" --db-sb-create-insecure-remote=yes --db-nb-create-insecure-remote=yes\"" | sudo tee -a /etc/default/ovn-central
sudo systemctl restart ovn-central

sudo systemctl stop ovn-host
# https://www.openvswitch.org/support/dist-docs-2.5/IntegrationGuide.md.txt
# http://www.openvswitch.org/support/dist-docs/ovs-vswitchd.conf.db.5.html
# http://www.openvswitch.org/support/dist-docs/ovs-vsctl.8.txt
sudo ovs-vsctl set open_vswitch . external_ids:ovn-remote="\"tcp:$API_SERVER_IP:6642\""
sudo ovs-vsctl get open_vswitch . external_ids:ovn-remote

sudo ovs-vsctl set open_vswitch . external_ids:ovn-encap-type="\"geneve\""
# https://stackoverflow.com/questions/39819378/ansible-get-current-target-hosts-ip-address
sudo ovs-vsctl set open_vswitch . external_ids:ovn-encap-ip="\"$API_SERVER_IP\""

sudo ovs-vsctl get open_vswitch . external_ids

# https://docs.openvswitch.org/en/latest/tutorials/ovs-advanced/#setup
sudo ovs-vsctl add-br br-int -- set Bridge br-int fail-mode=secure
sudo systemctl start ovn-host

• install multus 

kubectl apply -f https://raw.githubusercontent.com/onap/multicloud-k8s/master/kud/deployment_infra/images/multus-daemonset.yml

kubectl describe CustomResourceDefinition network-attachment-definitions.k8s.cni.cncf.io
kubectl describe ClusterRole multus
kubectl describe ClusterRoleBinding multus
kubectl -n kube-system describe ServiceAccount multus
kubectl -n kube-system describe ConfigMap multus
kubectl -n kube-system get DaemonSet

• deploy OVN4NFV

# ref:  https://github.com/onap/multicloud-k8s

# follow the step from:  https://github.com/onap/multicloud-k8s/blob/master/kud/deployment_infra/playbooks/configure-ovn4nfv.yml

# setp 1: define a CRD network object specification
cat > /tmp/ovn4nfvnetwork.yml << EOF
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: ovn-networkobj
spec:
  config: '{
     "cniVersion": "0.3.1",
     "name": "ovn4nfv-k8s-plugin",
     "type": "ovn4nfvk8s-cni"
  }'
EOF

kubectl apply -f /tmp/ovn4nfvnetwork.yml
kubectl describe net-attach-def

kubectl create namespace operator
kubectl label node `hostname` nfnType=operator --overwrite
kubectl apply -f https://raw.githubusercontent.com/onap/multicloud-k8s/master/kud/deployment_infra/images/nfn.yml

# check status
sudo kubectl describe CustomResourceDefinition networks.k8s.plugin.opnfv.org
sudo kubectl describe CustomResourceDefinition providernetworks.k8s.plugin.opnfv.org
sudo kubectl -n operator describe ServiceAccount k8s-nfn-sa
sudo kubectl describe ClusterRole k8s-nfn-cr
sudo kubectl describe ClusterRoleBinding k8s-nfn-crb
sudo kubectl -n operator describe Service nfn-operator
sudo kubectl -n operator describe Deployment nfn-operator
sudo kubectl -n operator describe ConfigMap ovn4nfv-cni-config
sudo kubectl -n operator describe DaemonSet ovn4nfv-cni
sudo kubectl -n operator describe DaemonSet nfn-agent

• deploy openwrt  

# https://github.com/akraino-edge-stack/icn/blob/master/sdwan/test/sdwan-openwrt-ovn.yml
# https://github.com/akraino-edge-stack/icn-sdwan/blob/master/platform/test/e2e-test/edge-scripts/setup-cnf.sh
mkdir ~/sdwan_setup
cd ~/sdwan_setup
# https://github.com/akraino-edge-stack/icn-sdwan/blob/master/platform/test/e2e-test/edge-scripts/setup-cnf.sh
curl https://raw.githubusercontent.com/akraino-edge-stack/icn-sdwan/master/platform/test/e2e-test/edge-a/scripts/variables -o  ./variables
source ./variables
curl https://raw.githubusercontent.com/akraino-edge-stack/icn-sdwan/master/platform/test/e2e-test/edge-scripts/sdwan_verifier.sh -o sdwan_verifier.sh
curl https://raw.githubusercontent.com/akraino-edge-stack/icn-sdwan/master/platform/test/e2e-test/edge-scripts/setup-cnf.sh -o setup-cnf.sh
sed -i -e 's/openwrt:test/openwrt:${OPENWRT_V:-test}/' setup-cnf.sh
kubectl taint nodes --all node-role.kubernetes.io/master-

OPENWRT_V=0.3.0 bash setup-cnf.sh

• Add a new API（used for demo）

# https://kubernetes.io/zh/docs/tasks/debug-application-cluster/get-shell-running-container/
kubectl exec -it $sdewan_cnf_name sh
cd /usr/lib/lua/luci/controller/rest_v1/
ps |grep http
# https://oldwiki.archive.openwrt.org/doc/techref/ubus
ubus list
ubus -v list network.interface
ubus call network.interface.wan0 status

PODIP=`kubectl get pod $sdewan_cnf_name -o template='{{ .status.podIP }}'`

BASE_URL="http://$PODIP/cgi-bin/luci/"
TOKEN=`wget -S --spider --post-data "luci_username=root&luci_password=" $BASE_URL 2>&1 | grep -o "sysauth=\([0-9a-fA-F]\+\)" |cut -d "=" -f2`

cat > demo.lua <<<'-- Licensed to the public under the GNU General Public License v2.

module("luci.controller.demo", package.seeall)

sys = require "luci.sys"
ut = require "luci.util"
io = require "io"

ip = "ip -4 "

function index()
    entry({"admin", "config", "command"},
        call("execute")).dependent = false
end

function trim(s)
    return s:match("^%s*(.-)%s*$")
end

function split_and_trim(str, sep)
    local array = {}
    local reg = string.format("([^%s]+)", sep)
    for item in string.gmatch(str, reg) do
        item_trimed = trim(item)
        if string.len(item_trimed) > 0 then
            table.insert(array, item_trimed)
        end
    end
    return array
end

function execute()
    local commands = luci.http.formvalue("command")
    io.stderr:write("Execute command: %s\n" % commands)
    
    local command_array = split_and_trim(commands, ";")
    for index, command in ipairs(command_array) do
--        io.stderr:write("%d: [%s]\n" % {index, command})
        sys.exec(command)
--        sys.exec("uci commit sample")
--        sys.exec("/etc/init.d/firewall restart")
    end

    luci.http.prepare_content("application/json")
    luci.http.write_json("{'status':'ok'}")
end'

scp demo.lua root@$PODIP:/usr/lib/lua/luci/controller
wget -S --header="Cookie:sysauth=$TOKEN" --post-data "command=fw3 reload" $BASE_URL/admin/config/command
curl -s -X "POST" -H "Cookie:sysauth=$TOKEN" -d "command=fw3 reload" $BASE_URL/admin/config/demo


# login pod
kubectl exec -it $sdewan_cnf_name sh
logread
sed -i -e 's/"config", "command"/"config", "demo"/' /usr/lib/lua/luci/controller/demo.lua
exit  

wget -S --header="Cookie:sysauth=$TOKEN" --post-data "command=fw3 reload" $BASE_URL/admin/config/demo
curl -s -X "POST" -H "Cookie:sysauth=$TOKEN" -d "command=fw3 reload" $BASE_URL/admin/config/demo

• add a really interface API:   

cat > interface.lua <<EOF
-- Licensed to the public under the GNU General Public License v2.

module("luci.controller.rest_v1.interface", package.seeall)

NX = require("nixio")
json = require "luci.jsonc"
io = require "io"
sys = require "luci.sys"
utils = require "luci.controller.rest_v1.utils"
util = require "luci.util"

fields_table = {
    {field="ip_address", key="inet addr", type="array"},
    {field="ip6_address", key="inet6 addr", type="array"},
    {field="mac_address", key="HWaddr"},
    {field="received_packets", key="RX packets"},
    {field="send_packets", key="TX packets"},
    {field="status", key=function(data) if string.match(data, "[%s]UP[%s]") ~= nil then return "UP" else return "DOWN" end end},
}

interface_validator = {
    {name="action", required=true, validator=function(value) return utils.in_array(value, {"up", "down"}) end, message="wrong action"}
}

function index()
    ver = "v1"
    entry({"sdewan", ver, "interfaces"}, call("handle_request")).leaf = true
end

function get_field(data, key, field_type)
    if type(key) == "function" then
        return key(data)
    end

    local reg = {
        key .. ":[^%s]+[%s]",
        key .. " [^%s]+[%s]",
        key .. ": [^%s]+[%s]",
    }

    local ret = nil
    for index=1, #reg do
        for item in string.gmatch(data, reg[index]) do
            local value = nil
            local i,j = string.find(item, key .. ": ")
            if i ~= nil then
                value = string.sub(item, j+1, string.len(item)-1)
            else
                i,j = string.find(item, key .. ":")
                if i ~= nil then
                    value = string.sub(item, j+1, string.len(item)-1)
                else
                    i,j = string.find(item, key .. " ")
                    if i ~= nil then
                        value = string.sub(item, j+1, string.len(item)-1)
                    end
                end
            end
            if value ~= nil then
                if field_type == "array" then
                    if ret == nil then
                        ret = {value}
                    else
                        ret[#ret+1] = value
                    end
                else
                    ret = value
                    break
                end
            end
        end
    end
    return ret
end

function is_interface_available(ifname)
    for interface in util.execi("ifconfig | awk '/^[^ \t]+/{print $1}'") do
        if interface ~= "lo" and interface == ifname then
           return true
        end
    end

    return false
end

function get_interface(interface, bypasscheck)
    if bypasscheck ~= true then
        if is_interface_available(interface) == false then
            return nil
        end
    end

    local ret = {}
    local data = util.exec("ifconfig " .. interface)
    if data == nil then
        for j=1, 3 do
            utils.log("ifconfig failed, retrying ... ")
            NX.nanosleep(1)
            data = util.exec("ifconfig " .. interface)
            if data ~= nil then
                break
            end
        end
    end
    ret["name"] = interface
    for i,v in pairs(fields_table) do
        local value = get_field(data, v["key"], v["type"])
        if value ~= nil then
            ret[v["field"]] = value
        end
    end
    return ret
end

function get_interfaces()
    local ret = {}
    ret["interfaces"] = {}
    local index = 1
    for interface in util.execi("ifconfig | awk '/^[^ \t]+/{print $1}'") do
        if interface ~= "lo" then
           ret["interfaces"][index] = get_interface(interface, true)
           index = index + 1
        end
    end
    return ret
end

-- Request Handler
function handle_request()
    local method = utils.get_req_method()
    if method == "GET" then
        return getRequest()
    elseif method == "PUT" then
        return setRequest()
    else
        utils.response_error(405, "Method Not Allowed")
    end
end

function getRequest()
    local uri_list = utils.get_URI_list()
    if uri_list == nil then
        return
    end

    if #uri_list == 5 then
        utils.response_object(get_interfaces())
    elseif #uri_list == 6 then
        local ifname = uri_list[#uri_list]
        local obj = get_interface(ifname)

        if obj == nil then
            utils.response_error(404, "Cannot find interface[" .. ifname .. "]" )
            return
        end
        utils.response_object(obj)
    else
        utils.response_error(400, "Bad request URI")
        return
    end
end

function setRequest()
    local uri_list = utils.get_URI_list()
    if uri_list == nil then
        return
    end

    if #uri_list == 6 then
        local ifname = uri_list[#uri_list]
        -- local obj = get_interface(ifname)
        -- if obj == nil then
        --    utils.response_error(404, "Cannot find interface[" .. ifname .. "]" )
        --    return
        -- end
        -- check content
        local body_obj = utils.get_and_validate_body_object(interface_validator)
        if body_obj == nil then
            return
        end

        local action = body_obj["action"]
        local exec_command = "ifconfig " .. ifname .. " " .. action
        -- execute command
        utils.log("Execute Command: %s" % exec_command)
        sys.exec(exec_command)
        NX.nanosleep(1)
        utils.response_success()
    else
        utils.response_error(400, "Bad request URI")
        return
    end
end
EOF

 

• test the interface API Manual

# copy to CNF POD
scp interface.lua root@$PODIP:/usr/lib/lua/luci/controller/rest_v1
# get  json respon
curl -s -H "Cookie:sysauth=$TOKEN" -H "Content-Type: application/json"  $BASE_URL/sdewan/v1/interfaces  | jq

# debug:
kubectl exec -it $sdewan_cnf_name logread

• Add a label for CNF pod

 kubectl label pod $sdewan_cnf_name sdewanPurpose=cnf

• create interface empty CRD

os=$(go env GOOS)
arch=$(go env GOARCH)
curl -L https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH) | tar -xz -C /tmp/
VURL=`curl -Ls -o /dev/null -w %{url_effective} https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)`
VERSION=${VURL##*/}
sudo mv /tmp/${VERSION%%.*} /usr/local/kubebuilder
echo "export PATH=$PATH:/usr/local/kubebuilder/bin" >> ~/.profile  && source ~/.profile

# https://kubernetes-sigs.github.io/kustomize/installation/source/
GOBIN=$(go env GOPATH)/bin GO111MODULE=on go get sigs.k8s.io/kustomize/kustomize/v3
sudo apt-get install gcc

mkdir sdewan && cd sdewan 
go mod init sdewan.akraino.org/sdewan

kubebuilder init --domain sdewan.akraino.org
kubebuilder create api --group batch --version  v1alpha1  --kind  CniInterface

make install
make run
# it will run, see Makefile
# /home/vagrant/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
# go fmt ./...
# go vet ./...
# /home/vagrant/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
# go run ./main.go

kubectl describe CustomResourceDefinition cniinterfaces.batch.sdewan.akraino.org
kubectl apply -f config/samples/
kubectl describe CniInterface
kubectl delete -f config/samples/

make uninstall

 

• Add base openwrt client and test it.

# currently the CNF is manager openwrt by rust API

# we need to implementation openwrt client, ref https://github.com/cheng1li/sdewan/

mkdir openwrt/
curl https://raw.githubusercontent.com/cheng1li/sdewan/master/openwrt/openwrtclient.go -o openwrt/openwrtclient.go

curl https://raw.githubusercontent.com/cheng1li/sdewan/8c34d5f2d1da238daf20272b57c4e9b72943c405/openwrt/service.go  -o openwrt/service.go

curl https://raw.githubusercontent.com/cheng1li/sdewan/8c34d5f2d1da238daf20272b57c4e9b72943c405/openwrt/utils.go openwrt/utils.go 


mkdir cmd
cat > cmd/client.go <<EOF
package main

import (
    "fmt"
    "flag"
    "sdewan.akraino.org/sdewan/openwrt"
)

func main() {
    fmt.Println("hello world")
    podip := flag.String("ip", "127.0.0.1", "A string for pod ip.")
    flag.Parse()
    fmt.Println("OpenWrt IP:", *podip)
    openwrtClient := openwrt.NewOpenwrtClient(*podip, "root", "")
    fmt.Println("openwrtClient:", openwrtClient)
    service := openwrt.ServiceClient{OpenwrtClient: openwrtClient}
    sc, _ := service.GetAvailableServices()
    fmt.Println("AvailableServices:", sc)
}
EOF

go run cmd/client.go -ip=$(kubectl get pod $sdewan_cnf_name -o template='{{ .status.podIP }}')

 

• Add base interfaces client and test it.

cat > openwrt/interfaces.go <<EOF
package openwrt

import (
        "encoding/json"
)

const (
        interfaceBaseURL = "sdewan/v1/"
)


type InterfaceClient struct {
        OpenwrtClient *openwrtClient
}

// Interface API struct
type Interface struct {
        Name            string   `json:"name,omitempty"`
        ReceivedPackets string   `json:"received_packets,omitempty"`
        SendPackets     string   `json:"send_packets,omitempty"`
        Status          string   `json:"status,omitempty"`
        MacAddress      string   `json:"mac_address,omitempty"`
        IpAddress       []string `json:"ip_address,omitempty"`
}

type AvailableInterfaces struct {
        Interfaces []Interface `json:"interfaces"`
}

// get available interfaces
func (s *InterfaceClient) GetAvailableInterfaces() (*AvailableInterfaces, error) {
        response, err := s.OpenwrtClient.Get(interfaceBaseURL + "interfaces")
        if err != nil {
                return nil, err
        }

        var servs AvailableInterfaces
        err2 := json.Unmarshal([]byte(response), &servs)
        if err2 != nil {
                return nil, err2
        }

        return &servs, nil
}
EOF

cat > cmd/client.go <<EOF
package main

import (
    "fmt"
    "flag"
    "sdewan.akraino.org/sdewan/openwrt"
)

func main() {
    fmt.Println("hello world")
    podip := flag.String("ip", "127.0.0.1", "A string for pod ip.")
    flag.Parse()
    fmt.Println("OpenWrt IP:", *podip)
    openwrtClient := openwrt.NewOpenwrtClient(*podip, "root", "")
    fmt.Println("openwrtClient:", openwrtClient)
    service := openwrt.InterfaceClient{OpenwrtClient: openwrtClient}
    fcs, _ := service.GetAvailableInterfaces()
    fmt.Println("AvailableInterfaces:", fcs)
    fmt.Println("Interface 0, name:", fcs.Interfaces[0].Name)
}
EOF

go run cmd/client.go -ip=$(kubectl get pod $sdewan_cnf_name -o template='{{ .status.podIP }}')

 

• fill the interface CRD

patch 

cat > /tmp/sdwan_interface.patch << EOF
commit fdeff619eb0e4270d25c3f656f72a3378f922e9b
Author: vagrant <vagrant@ubuntu>
Date:   Fri Sep 11 15:57:25 2020 +0000

    add more field for CniInterfaceSpec

diff --git a/api/v1alpha1/cniinterface_types.go b/api/v1alpha1/cniinterface_types.go
index 03623c0..963771f 100644
--- a/api/v1alpha1/cniinterface_types.go
+++ b/api/v1alpha1/cniinterface_types.go
@@ -29,13 +29,22 @@ type CniInterfaceSpec struct {
        // Important: Run "make" to regenerate code after modifying this file

        // Foo is an example field of CniInterface. Edit CniInterface_types.go to remove/update
-       Foo string `json:"foo,omitempty"`
+       Name        string               `json:"name,omitempty"`
+       Status      string               `json:"status,omitempty"`
+       MacAddress  string               `json:"mac_address,omitempty"`
+       IpAddress   []string             `json:"ip_address,omitempty"`
+       CnfSelector metav1.LabelSelector `json:"cnfSelector,omitempty"`
 }

 // CniInterfaceStatus defines the observed state of CniInterface
 type CniInterfaceStatus struct {
        // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
        // Important: Run "make" to regenerate code after modifying this file
+       ReceivedPackets string   `json:"received_packets,omitempty"`
+       SendPackets     string   `json:"send_packets,omitempty"`
+       Status          string   `json:"status,omitempty"`
+       MacAddress      string   `json:"mac_address,omitempty"`
+       IpAddress       []string `json:"ip_address,omitempty"`
 }

 // +kubebuilder:object:root=true
diff --git a/config/samples/batch_v1alpha1_cniinterface.yaml b/config/samples/batch_v1alpha1_cniinterface.yaml
index 0d9ff78..4171ced 100644
--- a/config/samples/batch_v1alpha1_cniinterface.yaml
+++ b/config/samples/batch_v1alpha1_cniinterface.yaml
@@ -4,4 +4,7 @@ metadata:
   name: cniinterface-sample
 spec:
   # Add fields here
-  foo: bar
+  cnfSelector:
+    matchLabels:
+      sdewanPurpose: cnf
+  name: eth0
diff --git a/controllers/cniinterface_controller.go b/controllers/cniinterface_controller.go
index dfd4008..416f89d 100644
--- a/controllers/cniinterface_controller.go
+++ b/controllers/cniinterface_controller.go
@@ -20,11 +20,13 @@ import (
        "context"

        "github.com/go-logr/logr"
+       corev1 "k8s.io/api/core/v1"
        "k8s.io/apimachinery/pkg/runtime"
        ctrl "sigs.k8s.io/controller-runtime"
        "sigs.k8s.io/controller-runtime/pkg/client"

        batchv1alpha1 "sdewan.akraino.org/sdewan/api/v1alpha1"
+       "sdewan.akraino.org/sdewan/openwrt"
 )

 // CniInterfaceReconciler reconciles a CniInterface object
@@ -38,10 +40,48 @@ type CniInterfaceReconciler struct {
 // +kubebuilder:rbac:groups=batch.sdewan.akraino.org,resources=cniinterfaces/status,verbs=get;update;patch

 func (r *CniInterfaceReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
-       _ = context.Background()
-       _ = r.Log.WithValues("cniinterface", req.NamespacedName)
+       ctx := context.Background()
+       log := r.Log.WithValues("cniinterface", req.NamespacedName)

        // your logic here
+       var cniInterface batchv1alpha1.CniInterface
+       if err := r.Get(ctx, req.NamespacedName, &cniInterface); err != nil {
+               log.Error(err, "unable to fetch CniInterface")
+               // we'll ignore not-found errors, since they can't be fixed by an immediate
+               // requeue (we'll need to wait for a new notification), and we can get them
+               // on deleted requests.
+               return ctrl.Result{}, client.IgnoreNotFound(err)
+       }
+
+       podList := &corev1.PodList{}
+       label := cniInterface.Spec.CnfSelector.MatchLabels["sdewanPurpose"]
+       err := r.Client.List(ctx, podList, client.MatchingLabels{"sdewanPurpose": label})
+       if err != nil {
+               log.Error(err, "Failed to get pod list")
+               return ctrl.Result{}, err
+       }
+
+       fcs := &openwrt.AvailableInterfaces{}
+       for _, pod := range podList.Items {
+               openwrtClient := openwrt.NewOpenwrtClient(pod.Status.PodIP, "root", "")
+               ifc := openwrt.InterfaceClient{OpenwrtClient: openwrtClient}
+               fcs, _ = ifc.GetAvailableInterfaces() // TODO, support get one interface?
+       }
+
+       for _, fc := range fcs.Interfaces {
+               if cniInterface.Spec.Name == fc.Name {
+                       cniInterface.Status.Status = fc.Status
+                       cniInterface.Status.ReceivedPackets = fc.ReceivedPackets
+                       cniInterface.Status.SendPackets = fc.SendPackets
+                       cniInterface.Status.MacAddress = fc.MacAddress
+                       cniInterface.Status.IpAddress = fc.IpAddress
+                       break
+               }
+       }
+       if err := r.Update(ctx, &cniInterface); err != nil {
+               log.Error(err, "unable to update CniInterface status")
+               return ctrl.Result{}, err
+       }

        return ctrl.Result{}, nil
 }
EOF

test

patch -p1 < sdwan_interface.patch
make install && make run

kubectl apply -f config/samples/
kubectl describe CniInterface
kubectl delete -f config/samples/

make uninstall

exmaple: controller-runtime/tree/master/examples/builtins

• All the code is in github

• deploy Admission webhook

# https://github.com/kubernetes-sigs/kubebuilder/blob/master/docs/book/src/cronjob-tutorial/webhook-implementation.md
kubebuilder create webhook --group batch --version  v1alpha1  --kind  CniInterface --programmatic-validation --defaulting

# https://segmentfault.com/a/1190000020338350

mkdir -p /tmp/k8s-webhook-server/serving-certs/
cp /etc/kubernetes/pki/apiserver.crt /tmp/k8s-webhook-server/serving-certs/tls.crt
sudo cp /etc/kubernetes/pki/apiserver.key /tmp/k8s-webhook-server/serving-certs/tls.key
sudo chown $(whoami):$(whoami) /tmp/k8s-webhook-server/serving-certs/tls.key

CABUNDLE=`kubectl config view --raw -o json | jq -r '.clusters[0].cluster."certificate-authority-data"' | tr -d '"'`

sed -i -e "s/caBundle: Cg==/caBundle: $CABUNDLE/" config/webhook/manifests.yaml

# sed -i -e "s/caBundle: $CABUNDLE/caBundle: Cg==/" config/webhook/manifests.yaml

sed -i -e "/service:/,+2d" config/webhook/manifests.yaml
sed -i -e "s/  path: \(.*\)$/url: https:\/\/$API_SERVER_IP:9443\1/" config/webhook/manifests.yaml     


sed -i -e "s/namespace: system/namespace: kube-system/" config/webhook/manifests.yaml config/webhook/service.yaml
# sed -i -e "s/namespace: kube-system/namespace: system/" config/webhook/manifests.yaml config/webhook/service.yaml

kubectl apply -f config/webhook/service.yaml
kubectl -n kube-system describe Service webhook-service
# kubectl describe Service webhook-service
kubectl apply -f config/webhook/manifests.yaml
kubectl describe ValidatingWebhookConfiguration validating-webhook-configuration
kubectl describe MutatingWebhookConfiguration mutating-webhook-configuration

make run
kubectl apply -f config/samples/

# kubectl delete Service webhook-service
kubectl -n kube-system delete Service webhook-service

• create user

OPERATOR=cnfop
ORGANIZATION=icn:cnf:sdewan
# https://zhuanlan.zhihu.com/p/43237959
# https://github.com/openssl/openssl/issues/7754
cat /etc/ssl/openssl.cnf | sed "s/RANDFILE\s*=\s*\$ENV::HOME\/\.rnd/#/"
dd if=/dev/urandom of=~/.rnd bs=256 count=1 
openssl genrsa -out $OPERATOR.key 2048
openssl req -new -key $OPERATOR.key -out $OPERATOR.csr -subj "/CN=$OPERATOR/O=$ORGANIZATION" 
sudo openssl x509 -req -in $OPERATOR.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out $OPERATOR.crt -days 36500
openssl x509 -in $OPERATOR.crt -noout -text  
kubectl config set-credentials $OPERATOR --client-certificate=$OPERATOR.crt --client-key=$OPERATOR.key

kubectl config get-clusters
K8SUSER=`kubectl config get-contexts | tail -n 1 | awk '{print $4}'`
CLUSTERNAME=`kubectl config get-contexts | tail -n 1 | awk '{print $3}'`
# CLUSTERNAME=`grep -n3 "\- cluster:" ~/.kube/config | grep -o "name:.*" | cut -d " " -f 2`
# kubectl config set-context $OPERATOR@$CLUSTERNAME --cluster=$CLUSTERNAME --namespace=default --user=$OPERATOR
kubectl config set-context $OPERATOR@$CLUSTERNAME --cluster=$CLUSTERNAME --user=$OPERATOR
kubectl config view
kubectl config current-context  
kubectl config use-context $OPERATOR@$CLUSTERNAME
kubectl config use-context $OPERATOR@$CLUSTERNAME
CURRENTNAME=`kubectl config get-contexts -o name | head -n1`
LASTNAME=`kubectl config get-contexts -o name | tail -n1`

CRTDATA=`cat $OPERATOR.crt | base64 --wrap=0`
KEYDATA=`cat $OPERATOR.key | base64 --wrap=0` 
sed -i -e "s/client-certificate:.*$OPERATOR.crt/client-certificate-data: $CRTDATA/" ~/.kube/config
sed -i -e "s/client-key:\s.*$OPERATOR.key/client-key-data: $KEYDATA/" ~/.kube/config

• Add/Delete clust role binding

kubectl --context=$LASTNAME create clusterrolebinding ${OPERATOR}-cluster-admin-binding --clusterrole=cluster-admin --user=${OPERATOR}
kubectl --context=$LASTNAME delete clusterrolebinding ${OPERATOR}-cluster-admin-binding

• Add role binding  

# ref: https://kubernetes.io/zh/docs/reference/access-authn-authz/rbac/
# "kind": "batch.sdewan.akraino.org/v1alpha1, Kind=CniInterface"
# "resource": {"group":"batch.sdewan.akraino.org","version":"v1alpha1","resource":"cniinterfaces"}
OPERATOR=cnfop
ORGANIZATION=icn:nfc:sdewan
K8SUSER=`grep -n1 "users:" ~/.kube/config |grep -o "name:.*" | cut -d " " -f 2`
cat > sdewan-cnf-role-binding.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: icn:cnf-operator
  # namespace: kube-system
rules:
- apiGroups:
  - "batch.sdewan.akraino.org"
  resources:
  - cniinterfaces
  verbs: ["*"]
  # resourceNames: [cniinterfaces-eth0]
  
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: icn:cnf-operator-binding
  # namespace: kube-system
roleRef:
  apiGroup: ""
  kind: Role
  name: icn:cnf-operator
subjects:
- apiGroup: ""
  kind: User
  name: $OPERATOR
- kind: Group
  name: $ORGANIZATION
  apiGroup: rbac.authorization.k8s.io
EOF

kubectl --context=$LASTNAME apply -f sdewan-cnf-role-binding.yaml
rm sdewan-cnf-role-binding.yaml
kubectl --context=$LASTNAME describe  Role icn:cnf-operator 
kubectl --context=$LASTNAME describe  Rolebinding icn:cnf-operator-binding

kubectl --context=$LASTNAME delete Role icn:cnf-operator 
kubectl --context=$LASTNAME delete Rolebinding icn:cnf-operator-binding

• switch context

kubectl config use-context $LASTNAME

• create CRT and USER/GROUP by kubectl

CSRDATA=$(cat ${OPERATOR}.csr | base64 | tr -d '\n')

# Ref: 
# https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/
# https://github.com/rancher/k3s/issues/1764
# https://thenewstack.io/a-practical-approach-to-understanding-kubernetes-authentication/
# https://dev.to/focusedlabs/provisioning-users-and-groups-for-kubernetes-4251
# https://docs.bitnami.com/tutorials/configure-rbac-in-your-kubernetes-cluster/

cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: $OPERATOR
spec:
  username: $OPERATOR
  groups:
  - system:authenticated
  request: ${CSRDATA}
  usages:
  - digital signature
  - key encipherment
  - client auth
EOF

kubectl certificate approve $OPERATOR
kubectl get csr $OPERATOR -o jsonpath={.status.certificate} | base64 --decode > $OPERATOR.pem

USERNAME=$OPERATOR
kubectl create namespace $USERNAME

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: $USERNAME-namespace
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]
EOF

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: access-$USERNAME-namespace
  namespace: $USERNAME
subjects:
- kind: User
  name: $USERNAME
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: $USERNAME-namespace
  apiGroup: rbac.authorization.k8s.io
EOF

• others

REF:

kubespray deploy k8s

使用Kubespray自动化部署Kubernetes 1.13.1

使用Kubespray安装k8s集群

Kubernetes指南 kubespray

使用 Kubespray 在基础设施或云平台上安装 Kubernetes

openwrt

uHTTPd org/cn  

uHTTPd webserver (EN)   

uHTTPd github

Rest API supported in OpenWrt  

Run Custom Lua Script as CGI with uhttpd  

ubus (OpenWrt micro bus architecture)   

OpenWrt forum: Lua uhttpd API  

Kss PKI CA 

Calculating the CA Certificate Hash for Kubeadm

Reconstructing the Join Command for Kubeadm

join cluster after init token expired?

kubeadm token  

kubeadm 令牌   

kubeadm join   

go 

Installing Go from source 

get_go.sh (install latest)

update-golang (github repo, wget -O- https://raw.githubusercontent.com/udhos/update-golang/master/update-golang.sh  | sudo bash)

github 

go-latest (get project latest version in github)

go-github (github v3 API)

REST API v3

Shell - Get latest release from GitHub（ Raw）

ovn

How to use Open Virtual Networking with Kubernetes  

如何将OVN虚拟网络连接到外部网络？

OVN代替OVS后怎样定制虚拟网络实现虚拟机与外网通信  

SDN控制器之OVN实验三：从OVN虚拟网络访问物理网络  

OVN 简介

YAML

YAML 语言教程   

K8S

Getting started 

污点和容忍度  

获取正在运行容器的 Shell  

kubectl 概述   

flannel （github)

network addons  

LUA 语言 （简易教程） 

Lua 教程| 菜鸟教程   

Lua 教程_w3cschool - 编程狮   

other

Here_document 

Get final URL after curl is redirected  

Submitting ONAP Code Reviews through the Nordix Gerrit

50 `sed` Command Examples  

kustomize 

Configuring Kubeflow with kfctl and kustomize  

Declarative Management of Kubernetes Objects Using Kustomize  

Install the kustomize CLI from source without cloning the repo   

ksonnet （github）

 godoc

sigs.k8s.io/controller-runtime/pkg/client   (ListOptions, List,  controller/webhook examples/builtins, handler/enqueue_mapped, eventhandler)

k8s.io/client-go/kubernetes   type Clientset

k8s.io/api/core/v1    /  k8s.io/api/apps/v1

k8s.io/apimachinery/pkg/apis/meta/v1   

k8s.io/apimachinery/pkg/fields#Set   

sigs.k8s.io/controller-runtime/pkg/builder#Builder.For

k8s.io/apimachinery/pkg/fields#Set

 Webhook

kubebuilder2.0学习笔记——搭建和使用 /进阶使用  (更多)

Kubebuilder 中文文档   (官方版）

Writing and testing Kubernetes webhooks using Kubebuilder v2

install  cert-manager   

使用 Kubebuilder 创建自定义 K8s AdmissionWebhooks  

通过自定义资源扩展Kubernetes 

部署 cert manager (官网）

TIPS: 

1 Filed 和 label都可以做selector，但是annotations不可以

概念： Annotations(CN 注解),  label, labels vs  annotations, Labels and Selectors,  Field Selectors, 标签和选择算符  

 

kubectl get pods --field-selector status.phase=Running

Differentiate between Kubernetes labels vs annotations

Kubernetes labels and annotations are both ways of adding metadata to Kubernetes objects. The similarities end there, however. Kubernetes labels allow you to identify, select and operate on Kubernetes objects. Annotations are non-identifying metadata and do none of these things.

Annotations allow you to add non-identifying metadata to Kubernetes objects. Examples include phone numbers of persons responsible for the object or tool information for debugging purposes. In short, annotations can hold any kind of information that is useful and can provide context to DevOps teams.

Conclusion

Kubernetes labels are a great way of identifying and organizing Kubernetes objects and resources. Kubernetes team leads and IT managers are best placed to develop and implement Kubernetes labeling plans. By ensuring labeling conventions are followed across teams, they can tame Kubernetes environments and prevent sprawl. Since Labels make it easier to operate on Kubernetes objects in bulk, adhering to labeling conventions will also make teams more productive in the long run.

你可以使用 Kubernetes 注解为对象附加任意的非标识的元数据。客户端程序（例如工具和库）能够获取这些元数据信息。

你可以使用标签或注解将元数据附加到 Kubernetes 对象。 标签可以用来选择对象和查找满足某些条件的对象集合。 相反，注解不用于标识和选择对象。 注解中的元数据，可以很小，也可以很大，可以是结构化的，也可以是非结构化的，能够包含标签不允许的字符。

注解和标签一样，是键/值对。

 2 debug

insatll 

# https://github.com/go-delve/delve/blob/master/Documentation/usage/dlv.md
# https://github.com/go-delve/delve/blob/master/Documentation/installation/linux/install.md
go get github.com/go-delve/delve/cmd/dlv

hack

diff --git a/Makefile b/Makefile
index 5da22b5..f571f03 100644
--- a/Makefile
+++ b/Makefile
@@ -23,6 +23,7 @@ manager: generate fmt vet

 # Run against the configured Kubernetes cluster in ~/.kube/config
 run: generate fmt vet manifests
+       # dlv debug ./main.go
        go run ./main.go

 # Install CRDs into a cluster

 3. simple "Reconcile" Pseudocode  example:

# https://github.com/kubernetes/kubernetes/issues/84430
func (r *ReconcileObject) Reconcile(req reconcileRequest) {
  myObj = r.client.Get(context.TODO(), "my Namespace", "my object")
  myObj.Foo = "bar"
  sleep(30)

  r.client.Update(context.TODO(), myObj)
}

 4. Owns VS Watches 

 In icn-sdwan projects， it uses Watches for serveral related resources and customize the enqueue handler / event filter.  Manual for Watches/Owns from builder doc.

And it tells us:

func (*Builder) Owns ¶ Uses

func (blder *Builder) Owns(object runtime.Object, opts ...OwnsOption) *Builder

Owns defines types of Objects being *generated* by the ControllerManagedBy, and configures the ControllerManagedBy to respond to create / delete / update events by *reconciling the owner object*. This is the equivalent of calling Watches(&source.Kind{Type: <ForType-forInput>}, &handler.EnqueueRequestForOwner{OwnerType: apiType, IsController: true})

func (*Builder) Watches ¶ Uses

func (blder *Builder) Watches(src source.Source, eventhandler handler.EventHandler, opts ...WatchesOption) *Builder

Watches exposes the lower-level ControllerManagedBy Watches functions through the builder. Consider using Owns or For instead of Watches directly. Specified predicates are registered only for given source.

文档建议使用Owns.

 

 handle doc  about customize EnqueueRequestsFromMapFunc （源代码）： 

func EnqueueRequestsFromMapFunc ¶ Uses  

func EnqueueRequestsFromMapFunc(mapFN func(MapObject) []reconcile.Request) EventHandler

EnqueueRequestsFromMapFunc enqueues Requests by running a transformation function that outputs a collection of reconcile.Requests on each Event. The reconcile.Requests may be for an arbitrary set of objects defined by some user specified transformation of the source Event. (e.g. trigger Reconciler for a set of objects in response to a cluster resize event caused by adding or deleting a Node)

EnqueueRequestsFromMapFunc is frequently used to fan-out updates from one object to one or more other objects of a differing type.

For UpdateEvents which contain both a new and old object, the transformation function is run on both objects and both sets of Requests are enqueue.

 5. 理解 func set

理解go的function types

golang中 type func() 用法分析

EnqueueRequestsFromMapFunc 内部实现，使用了functions type, 根据/handler/enqueue_mapped.go的源码，实现了一个简化的例子：

package main

import (
    "fmt"
)

type NamespacedName struct {
    Namespace string
    Name      string
}

type Request struct {
    // NamespacedName is the name and namespace of the object to reconcile.
    NamespacedName
}

type Object interface {
    GetObjectKind(MapObject) []Request
}

type MapObject struct {
    Object Object 
}

type ObjectIns struct {
    Name string
}

func (o ObjectIns ) GetObjectKind(i MapObject) []Request {
    var enqueueRequest []Request
    return enqueueRequest 
}


// mapper maps an object to a collection of keys to be enqueued
type mapper interface {
    // Map maps an object
    Map(MapObject) []Request
}

type EventHandler interface {
    Create(string, string)
}
type enqueueRequestsFromMapFunc struct {
    // Mapper transforms the argument into a slice of keys to be reconciled
    ToRequests mapper
}
func (e *enqueueRequestsFromMapFunc) Create(evt string, q string) {
    fmt.Println(evt, q)
}


// var _ EventHandler = &enqueueRequestsFromMapFunc{}

type toRequestsFunc func(MapObject) []Request

func EnqueueRequestsFromMapFunc(mapFN func(MapObject) []Request) EventHandler {
    return &enqueueRequestsFromMapFunc{
        ToRequests: toRequestsFunc(mapFN),
    }
}

func (m toRequestsFunc) Map(i MapObject) []Request {
    return m(i)
}
func (m toRequestsFunc) Say() {
    fmt.Println("Function sets, say hello")
}

func GetToRequestsFunc(r string, crliststruct string) func(h MapObject) []Request {

    return func(h MapObject) []Request {
        fmt.Println("GetToRequestsFunc r parameter", r)
        fmt.Println("GetToRequestsFunc crliststruct  parameter", crliststruct )
        var enqueueRequest []Request
        req := Request{
            NamespacedName: NamespacedName{
                Name:      "meta.GetName",
                Namespace: "meta.GetNamespace()",
            }}
        enqueueRequest = append(enqueueRequest, req)

        return enqueueRequest
    }
}

func main() {
    fmt.Println("Hello, playground")
    oi := ObjectIns{Name: "ObjectIns"}
    mo := MapObject{Object: oi}
    a := GetToRequestsFunc("requert", "test")
    fmt.Println("=======================")
    fmt.Println(a(mo))
    fmt.Println("=======================")
    fmt.Println(&a)
    TR := toRequestsFunc(a)
    fmt.Println(TR)
    TR.Say()
    fmt.Println("=======================")
    TR.Map(mo) // == a(mo)
    fmt.Println("=======================")

    eq := &enqueueRequestsFromMapFunc{ToRequests: toRequestsFunc(a)}
    fmt.Println(eq)
    // &EnqueueRequestsFromMapFunc{TR: "xxxx"}
}

and try it in  https://play.golang.org/

 6. 关键数据结构 和方法

apis/meta/v1#OwnerReference

pkg/handler#EnqueueRequestForOwner     

pkg/reconcile    

pkg/types#NamespacedName   

pkg/controller/controllerutil#Object，pkg/runtime#Object   

controller-runtime/pkg/event#GenericEvent    

controller-runtime/pkg/controller#example-Controller   

controller-runtime/pkg/predicate#Predicate  

controller-runtime/pkg/handler#example-EnqueueRequestsFromMapFunc  

7. kubectl 格式化输出

JSONPath 支持     

8. 获取struct的名字

Golang- Getting struct attribute name

package main

import (
    "fmt"
    "reflect"
)

type MultiQuestions struct {
    QuestionId   int64
    QuestionType string
    QuestionText string
}

func (q *MultiQuestions) StructAttrName() string{
    v := reflect.ValueOf(q).Elem()
    t := v.Type()
    for i := 0; i < t.NumField(); i++ {
        fmt.Println(reflect.TypeOf(q).Elem().Field(i).Name)
    }
    return reflect.TypeOf(q).Elem().Field(0).Name
}

func main() {
    fmt.Println((&MultiQuestions{}).StructAttrName())
}

 9. EnqueueRequestsFromMapFunc系列函数和 shared_informer的关系（L18-23）

pkg/handler/enqueue_mapped.go 实现了怎么从将request 放入到workqueue中。

E0921 17:51:40.304838   26318 runtime.go:78] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
goroutine 309 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1441b80, 0x173f1a0)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/runtime/runtime.go:48 +0x89
panic(0x1441b80, 0x173f1a0)
        /usr/local/go/src/runtime/panic.go:969 +0x175
sdewan.akraino.org/sdewan/controllers.createCniInterfaces(0x0, 0x0, 0x0, 0x0, 0xc000040f80, 0xd, 0x0, 0x0, 0xc000040f99, 0x7, ...)
        /home/vagrant/sdwan_setup/sdewan/controllers/cniinterface_controller.go:88 +0xa6
sdewan.akraino.org/sdewan/controllers.GetToRequestsFunc2.func1(0x7fe810117918, 0xc000601000, 0x175c5a0, 0xc000601000, 0x0, 0x0, 0x40be1f)
        /home/vagrant/sdwan_setup/sdewan/controllers/cniinterface_controller.go:348 +0x16f
sigs.k8s.io/controller-runtime/pkg/handler.ToRequestsFunc.Map(0xc00061afc0, 0x7fe810117918, 0xc000601000, 0x175c5a0, 0xc000601000, 0x15b4da0, 0x7fe810117918, 0xc000986120)
        /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/handler/enqueue_mapped.go:104 +0x4e
sigs.k8s.io/controller-runtime/pkg/handler.(*EnqueueRequestsFromMapFunc).mapAndEnqueue(0xc000632c80, 0x1798060, 0xc000986120, 0x7fe810117918, 0xc000601000, 0x175c5a0, 0xc0006010
00)
        /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/handler/enqueue_mapped.go:67 +0x7c
sigs.k8s.io/controller-runtime/pkg/handler.(*EnqueueRequestsFromMapFunc).Create(0xc000632c80, 0x7fe810117918, 0xc000601000, 0x175c5a0, 0xc000601000, 0x1798060, 0xc000986120)
        /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/handler/enqueue_mapped.go:47 +0x67
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnAdd(0x1786f00, 0xc000632c80, 0x1798060, 0xc000986120, 0xc000632d50, 0x1, 0x1, 0x15b4da0, 0xc000601000)
        /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/source/internal/eventsource.go:74 +0x1a5
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /home/vagrant/go/pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/shared_informer.go:744 +0xc2
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0003cf760)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000365f60, 0x1753b40, 0xc0009340f0, 0xc00018e001, 0xc000936000)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0003cf760, 0x3b9aca00, 0x0, 0x1, 0xc000936000)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133 +0xe5
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000730180)
        /home/vagrant/go/pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/shared_informer.go:738 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000170140, 0xc0004c89f0)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:71 +0x65
panic: assignment to entry in nil map [recovered]
        panic: assignment to entry in nil map

 10. predicate Create/Update 与 shared_informer的关系（L7-10）

How to dump goroutine stacktraces?   

runtime/debug.Stack(0x36, 0x0, 0x0)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9f
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x25
sdewan.akraino.org/sdewan/controllers.glob..func1(0x7fd35829a0d8, 0xc000384e80, 0x175c5a0, 0xc000384e80, 0x175c5a0)
        /home/vagrant/sdwan_setup/sdewan/controllers/cniinterface_controller.go:396 +0xbf
sigs.k8s.io/controller-runtime/pkg/predicate.Funcs.Create(...)
        /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/predicate/predicate.go:63
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnAdd(0x1786f00, 0xc00020d690, 0x1798060, 0xc000986220, 0xc00020d720, 0x1, 0x1, 0x15b4da0, 0xc000384e80)
        /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/source/internal/eventsource.go:68 +0x130
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /home/vagrant/go/pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/shared_informer.go:744 +0xc2
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000280760)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000153f60, 0x1753b40, 0xc00037c0f0, 0xc000393201, 0xc0000d94a0)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000280760, 0x3b9aca00, 0x0, 0x1, 0xc0000d94a0)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133 +0xe5
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000410000)
        /home/vagrant/go/pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/shared_informer.go:738 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0001da140, 0xc00038e510)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:71 +0x65

 11. reconcile 与  与 shared_informer的关系（L6-12）

goroutine 343 [running]:
runtime/debug.Stack(0xc00053a900, 0x0, 0x0)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9f
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x25
sdewan.akraino.org/sdewan/controllers.(*CniInterfaceReconciler).Reconcile(0xc0006c7470, 0xc0000eb179, 0x7, 0xc00002a840, 0x13, 0x2a473adbda, 0xc0004506c0, 0xc000450638, 0xc000450630)
        /home/vagrant/sdwan_setup/sdewan/controllers/cniinterface_controller.go:202 +0x107
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000232000, 0x14901e0, 0xc00033a5a0, 0x1567300)
        /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000232000, 0x203000)
        /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000232000)

        /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0003d64d0)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0003d64d0, 0x1753b40, 0xc0003de360, 0xc00038c001, 0xc00010e240)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0003d64d0, 0x3b9aca00, 0x0, 0xc000337701, 0xc00010e240)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133 +0xe5
k8s.io/apimachinery/pkg/util/wait.Until(0xc0003d64d0, 0x3b9aca00, 0xc00010e240)
        /home/vagrant/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
        /home/vagrant/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:193 +0x32d

 12.  kubernetes cluster inited by kubeadm does not start automatically after rebooting os #110

node 重启之后需要，After sudo swapoff -a , I try systemctl enable kubelet again.

13. missing .rnd  

Purpose of RANDFILE in OpenSSL?

Can't load ./.rnd into RNG #7754 (fixed in this issue)

14. 增加用户

Certificate Signing Requests (org)      

Provisioning Users and Groups for Kubernetes   (step by step, create user by kubectl)

A Practical Approach to Understanding Kubernetes Authentication    

CertificateSigningRequest / user creation   

为Kubernetes集群添加用户   

15.  RBAC

Configure RBAC in your Kubernetes Cluster (step by step)   

Authenticating  (org)

QUESTION: How to create the k8s users and groups to map to? #139  

Using RBAC Authorization  (org/ CN 使用 RBAC 鉴权)   

Users and RBAC authorizations in Kubernetes （提供了几个很有用的tools，不过有个小错误： s/access_matrix/access-matrix

# Installation of KREW
# Krew is a tool that makes it easy to use kubectl plugins. Krew helps you discover plugins, install and manage them on your machine. It is similar to tools like apt, dnf or brew. Krew is only compatible with kubectl v1.12 and above.

set -x; cd "$(mktemp -d)" &&
curl -fsSLO "https://storage.googleapis.com/krew/v0.2.1/krew.{tar.gz,yaml}" &&
tar zxvf krew.tar.gz &&
./krew-"$(uname | tr '[:upper:]' '[:lower:]')_amd64" install \
    --manifest=krew.yaml --archive=krew.tar.gz

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

kubectl krew install access_matrix

kubectl access-matrix -n my-project-dev --as  $OPERATOR  # username

go get -v github.com/aquasecurity/kubectl-who-can
kubectl-who-can list pods -n default

kubectl krew install rbac-lookup
kubectl-rbac_lookup $OPERATOR  # username

kubectl-rbac_lookup --kind user
kubectl-rbac_lookup --kind group

 

16. service account

为 Pod 配置服务账户

管理 Service Accounts   

管理服务帐号（Service Accounts）  

Access Clusters Using the Kubernetes API   

在kubernetes 集群（POD)内访问k8s API服务  

kubernetes认证和serviceaccount  

example

How To Create Kubernetes Service Account For API Access:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: api-service-account
EOF

kubectl create serviceaccount api-service-account

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: api-access
rules:
  -
    apiGroups:
      - ""
      - apps
      - autoscaling
      - batch
      - extensions
      - policy
      - rbac.authorization.k8s.io
    resources:
      - componentstatuses
      - configmaps
      - daemonsets
      - deployments
      - events
      - endpoints
      - horizontalpodautoscalers
      - ingress
      - jobs
      - limitranges
      - namespaces
      - nodes
      - pods
      - persistentvolumes
      - persistentvolumeclaims
      - resourcequotas
      - replicasets
      - replicationcontrollers
      - serviceaccounts
      - services
    verbs: ["*"]
  - nonResourceURLs: ["*"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: api-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: api-access
subjects:
- kind: ServiceAccount
  name: api-service-account
  namespace: default
EOF

kubectl describe serviceaccount api-service-account
kubectl get serviceaccount api-service-account  -o json | jq -Mr '.secrets[].name'
SVCSERVICE=$(kubectl get serviceaccount api-service-account  -o json | jq -Mr '.secrets[].name')
kubectl get secrets ${SVCSERVICE} -o json | jq -Mr '.data.token' | base64 -d
TOKEN=`kubectl get secrets ${SVCSERVICE} -o json | jq -Mr '.data.token' | base64 -d`
kubectl get endpoints -o custom-columns=MATELABLE:.metadata.labels


SERVERIP=$(kubectl get endpoints kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}')
SERVERPORT=$(kubectl get endpoints kubernetes -o jsonpath='{.subsets[0].ports[0].port}')
curl -k  https://$SERVERIP:$SERVERPORT/api/v1/namespaces 
curl -k  https://$SERVERIP:$SERVERPORT/api/v1/namespaces -H "Authorization: Bearer $TOKEN"
curl -k -X GET https://$SERVERIP:$SERVERPORT/api/v1/namespaces/default/pods -H "Authorization: Bearer $TOKEN" |python3 -m json.tool

kubectl delete ClusterRoleBinding api-access
kubectl delete ClusterRole api-access
kubectl delete serviceaccount api-service-account

k8s创建服务账号——Service Account   

k8s API使用  

k8s 名词解释：Service Account （包括： 使用Vagrant， 各种部署例子： Spark例子）

16. controller examples in github

indexer  GetFieldIndexer().IndexField  

Reconcile icn provisioning_controller /  pod_controller

flannel_migration_utils   

kubernetes   

17. admission webhook 

动态准入控制   (org)

api: k8s.io/api/admissionregistration/  v1beta1

最简单的webhook，Creating Your Own Admission Controller， 纯手工写的， 起了一个server，配置了证书，监听某个地址，返回v1beta1.AdmissionReview对象。

the AdmissionReview consists of AdmissionRequest and AdmissionResponse objects ，详情请参考 Some Admission Webhook Basics  

 

 

 

 

serve 将 http请求的body反序列化为AdmissionReview， 传给了mutate 和 validating 处理函数, 具体实现的一个例子(深入理解 Kubernetes Admission Webhook)介绍及代码。

红帽 Custom Admission Controllers yaml文件元素具体解读，官方配置解读  

经典例子： admission-webhook-example（blog） 和 kube-mutating-webhook-tutorial

但是我们这个kubebuilder帮我们生成的脚手架代码基于controller-runtime的framework，比没有用上面的framwork， 我们想获取跟多的用户信息，很是受限。

// log is for logging in this package.
var cniinterfacelog = logf.Log.WithName("cniinterface-resource")

func (r *CniInterface) SetupWebhookWithManager(mgr ctrl.Manager) error {
        return ctrl.NewWebhookManagedBy(mgr).
                For(r).
                Complete()
}
      

改代码的官方例子，example-WebhookBuilder， 

controller-runtime的NewWebhookManagedBy 返回WebhookBuilder， 实现了 func (*WebhookBuilder) For ¶ Uses 和 func (*WebhookBuilder) Complete ¶ Uses 方法。

WebhookBuilder  的For （传入的参数要实现Validator 接口）方法返回了WebhookBuilder，Complete方法通过registerWebhooks实现了注册了三个webhook， registerDefaultingWebhook， registerValidatingWebhook， registerConversionWebhook。这三个函数最终通过controllerManager 的GetWebhookServer实现了server的Register. 我们看 registerValidatingWebhook流程，通过ValidatingWebhookFor来生成validatingHandler。 validatingHandler的Handle会通过Validator来调用Create，Update和Delete方法。

没有办法，需要自己来定制webhook，重新注册handler。 

 

Manager 提供了GetWebhookServer 方法， 返回webhook.Server， Server提供了Register 方法, 注册handler和对应的url路径。

webhook的Admission的Webhoo就是一个Handler （type HandlerFunc ¶ Uses，参考官方example）， 两个参数context.Context 和 type Request ¶ Uses 其返回值是 type Response ¶ Uses。

• type Request  就是  admissionv1beta1.AdmissionRequest
• type Response 主要结构就是前面介绍的 admissionv1beta1.AdmissionResponse