Kubernetes源码阅读笔记——Controller Manager（之二）
上一篇文章中，我们看到了Controller Manager的基本运行逻辑，但是还有一些问题没有解决，我们将在本篇文章中进行分析。
一、ListAndWatch
首先是Informer。上一篇中写道，启动Informer本质上是调用了controller的reflector的Run方法。下面我们进入reflector的Run方法看看：
k8s.io/cient-go/tools/cache/reflector.go func (r *Reflector) Run(stopCh <-chan struct{}) { glog.V(3).Infof("Starting reflector %v (%s) from %s", r.expectedType, r.resyncPeriod, r.name) wait.Until(func() { if err := r.ListAndWatch(stopCh); err != nil { utilruntime.HandleError(err) } }, r.period, stopCh) }
可以看到，方法通过调用wait.Until，每过period时间段就执行一次ListAndWatch方法。
进入ListAndWatch方法：
k8s.io/client-go/tools/cache/reflector.go // ListAndWatch first lists all items and get the resource version at the moment of call, // and then use the resource version to watch. // It returns error if ListAndWatch didn't even try to initialize watch. func (r *Reflector) ListAndWatch(stopCh <-chan struct{}) error { glog.V(3).Infof("Listing and watching %v from %s", r.expectedType, r.name) var resourceVersion string // Explicitly set "0" as resource version - it's fine for the List() // to be served from cache and potentially be delayed relative to // etcd contents. Reflector framework will catch up via Watch() eventually. options := metav1.ListOptions{ResourceVersion: "0"} r.metrics.numberOfLists.Inc() start := r.clock.Now() list, err := r.listerWatcher.List(options) if err != nil { return fmt.Errorf("%s: Failed to list %v: %v", r.name, r.expectedType, err) } r.metrics.listDuration.Observe(time.Since(start).Seconds()) listMetaInterface, err := meta.ListAccessor(list) if err != nil { return fmt.Errorf("%s: Unable to understand list result %#v: %v", r.name, list, err) } resourceVersion = listMetaInterface.GetResourceVersion() items, err := meta.ExtractList(list) if err != nil { return fmt.Errorf("%s: Unable to understand list result %#v (%v)", r.name, list, err) } r.metrics.numberOfItemsInList.Observe(float64(len(items))) if err := r.syncWith(items, resourceVersion); err != nil { return fmt.Errorf("%s: Unable to sync list result: %v", r.name, err) } r.setLastSyncResourceVersion(resourceVersion) resyncerrc := make(chan error, 1) cancelCh := make(chan struct{}) defer close(cancelCh) go func() { resyncCh, cleanup := r.resyncChan() defer func() { cleanup() // Call the last one written into cleanup }() for { select { case <-resyncCh: case <-stopCh: return case <-cancelCh: return } if r.ShouldResync == nil || r.ShouldResync() { glog.V(4).Infof("%s: forcing resync", r.name) if err := r.store.Resync(); err != nil { resyncerrc <- err return } } cleanup() resyncCh, cleanup = r.resyncChan() } }() for { // give the stopCh a chance to stop the loop, even in case of continue statements further down on errors select { case <-stopCh: return nil default: } timeoutSeconds := int64(minWatchTimeout.Seconds() * (rand.Float64() + 1.0)) options = metav1.ListOptions{ ResourceVersion: resourceVersion, // We want to avoid situations of hanging watchers. Stop any wachers that do not // receive any events within the timeout window. TimeoutSeconds: &timeoutSeconds, } r.metrics.numberOfWatches.Inc() w, err := r.listerWatcher.Watch(options) if err != nil { switch err { case io.EOF: // watch closed normally case io.ErrUnexpectedEOF: glog.V(1).Infof("%s: Watch for %v closed with unexpected EOF: %v", r.name, r.expectedType, err) default: utilruntime.HandleError(fmt.Errorf("%s: Failed to watch %v: %v", r.name, r.expectedType, err)) } // If this is "connection refused" error, it means that most likely apiserver is not responsive. // It doesn't make sense to re-list all objects because most likely we will be able to restart // watch where we ended. // If that's the case wait and resend watch request. if urlError, ok := err.(*url.Error); ok { if opError, ok := urlError.Err.(*net.OpError); ok { if errno, ok := opError.Err.(syscall.Errno); ok && errno == syscall.ECONNREFUSED { time.Sleep(time.Second) continue } } } return nil } if err := r.watchHandler(w, &resourceVersion, resyncerrc, stopCh); err != nil { if err != errorStopRequested { glog.Warningf("%s: watch of %v ended with: %v", r.name, r.expectedType, err) } return nil } } }
这个方法的注释已经写明。首先，用默认的ResourceVersion “0” 调用List方法，列出资源，并调用ListAccessor方法进行验证。
其次，调用GetResourceVersion方法取得资源的实际的ResourceVersion，并调用syncWith方法进行同步。
再次，用go func()后面的一大段代码处理reflector缓存的同步。
最后，在一个for循环里调用Watch方法，对资源的变化进行watch，并调用watchHandler方法对变化进行相应处理。
Watch方法本质上是在调用ListWatch的WatchFunc。这一字段是在创建具体的Controller对象时向informer添加的，后面会详细介绍。总之，Watch的本质与用户通过client-go工具连接API Server是一样的。
进入watchHandler方法：
k8s.io/client-go/tools/cache/reflector.go // watchHandler watches w and keeps *resourceVersion up to date. func (r *Reflector) watchHandler(w watch.Interface, resourceVersion *string, errc chan error, stopCh <-chan struct{}) error { start := r.clock.Now() eventCount := 0 defer w.Stop() // update metrics ... loop: for { select { case <-stopCh: return errorStopRequested case err := <-errc: return err case event, ok := <-w.ResultChan(): if !ok { break loop } if event.Type == watch.Error { return apierrs.FromObject(event.Object) } if e, a := r.expectedType, reflect.TypeOf(event.Object); e != nil && e != a { utilruntime.HandleError(fmt.Errorf("%s: expected type %v, but watch event object had type %v", r.name, e, a)) continue } meta, err := meta.Accessor(event.Object) if err != nil { utilruntime.HandleError(fmt.Errorf("%s: unable to understand watch event %#v", r.name, event)) continue } newResourceVersion := meta.GetResourceVersion() switch event.Type { case watch.Added: err := r.store.Add(event.Object) if err != nil { utilruntime.HandleError(fmt.Errorf("%s: unable to add watch event object (%#v) to store: %v", r.name, event.Object, err)) } case watch.Modified: err := r.store.Update(event.Object) if err != nil { utilruntime.HandleError(fmt.Errorf("%s: unable to update watch event object (%#v) to store: %v", r.name, event.Object, err)) } case watch.Deleted: // TODO: Will any consumers need access to the "last known // state", which is passed in event.Object? If so, may need // to change this. err := r.store.Delete(event.Object) if err != nil { utilruntime.HandleError(fmt.Errorf("%s: unable to delete watch event object (%#v) from store: %v", r.name, event.Object, err)) } default: utilruntime.HandleError(fmt.Errorf("%s: unable to understand watch event %#v", r.name, event)) } *resourceVersion = newResourceVersion r.setLastSyncResourceVersion(newResourceVersion) eventCount++ } } watchDuration := r.clock.Now().Sub(start) ... return nil }
这个方法中，最重要的是中间的case选择。对于watch到的结果，按照Added、Modified、Deleted分别调用store的Add、Update、Delete方法，在FIFO队列中进行更新。这个FIFO队列就是之前在执行informer的Run方法时所创建的。
Informer的大致逻辑就是这样，通过list-watch机制，与API Server建立连接，监听资源的变化，在缓存中进行更新。
二、StartControllers
上一篇文章中提到，Controller Manager通过在StartControllers方法中调用启动函数来启动所有的Controller。我们先来看看StartControllers方法：
cmd/kube-controller-manager/app/controllermanager.go func StartControllers(ctx ControllerContext, startSATokenController InitFunc, controllers map[string]InitFunc, unsecuredMux *mux.PathRecorderMux) error { ... for controllerName, initFn := range controllers { if !ctx.IsControllerEnabled(controllerName) { klog.Warningf("%q is disabled", controllerName) continue } time.Sleep(wait.Jitter(ctx.ComponentConfig.Generic.ControllerStartInterval.Duration, ControllerStartJitter)) klog.V(1).Infof("Starting %q", controllerName) debugHandler, started, err := initFn(ctx) if err != nil { klog.Errorf("Error starting %q", controllerName) return err } if !started { klog.Warningf("Skipping %q", controllerName) continue } ... return nil }
我们看到，方法通过对controllers中每个元素执行各自的initFn来执行启动函数，因为每个Controller都是以[string]function的格式保存在map中的，所以直接执行自身的function即可。
下面我们进入一个Controller的启动函数看看。以deployment为例。从NewControllerInitializers中找到controllers["deployment"]=startDeploymentController，进入startDeploymentController，可以看到方法位于app/apps.go中：
cmd/kube-controller-manager/app/apps.go func startDeploymentController(ctx ControllerContext) (http.Handler, bool, error) { if !ctx.AvailableResources[schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"}] { return nil, false, nil } dc, err := deployment.NewDeploymentController( ctx.InformerFactory.Apps().V1().Deployments(), ctx.InformerFactory.Apps().V1().ReplicaSets(), ctx.InformerFactory.Core().V1().Pods(), ctx.ClientBuilder.ClientOrDie("deployment-controller"), ) if err != nil { return nil, true, fmt.Errorf("error creating Deployment controller: %v", err) } go dc.Run(int(ctx.ComponentConfig.DeploymentController.ConcurrentDeploymentSyncs), ctx.Stop) return nil, true, nil }
我们看到，方法调用了NewDeploymentController方法创建一个Deployment，并通过GO协程调用Run方法运行它。
我们先来看一下NewDeploymentController方法。
三、NewDeploymentController
pkg/controller/deployment/deployment_controller.go func NewDeploymentController(dInformer appsinformers.DeploymentInformer, rsInformer appsinformers.ReplicaSetInformer, podInformer coreinformers.PodInformer, client clientset.Interface) (*DeploymentController, error) { eventBroadcaster := record.NewBroadcaster() eventBroadcaster.StartLogging(klog.Infof) eventBroadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: client.CoreV1().Events("")}) if client != nil && client.CoreV1().RESTClient().GetRateLimiter() != nil { if err := metrics.RegisterMetricAndTrackRateLimiterUsage("deployment_controller", client.CoreV1().RESTClient().GetRateLimiter()); err != nil { return nil, err } } dc := &DeploymentController{ client: client, eventRecorder: eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "deployment-controller"}), queue: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "deployment"), } dc.rsControl = controller.RealRSControl{ KubeClient: client, Recorder: dc.eventRecorder, } dInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: dc.addDeployment, UpdateFunc: dc.updateDeployment, // This will enter the sync loop and no-op, because the deployment has been deleted from the store. DeleteFunc: dc.deleteDeployment, }) rsInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: dc.addReplicaSet, UpdateFunc: dc.updateReplicaSet, DeleteFunc: dc.deleteReplicaSet, }) podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ DeleteFunc: dc.deletePod, }) dc.syncHandler = dc.syncDeployment dc.enqueueDeployment = dc.enqueue dc.dLister = dInformer.Lister() dc.rsLister = rsInformer.Lister() dc.podLister = podInformer.Lister() dc.dListerSynced = dInformer.Informer().HasSynced dc.rsListerSynced = rsInformer.Informer().HasSynced dc.podListerSynced = podInformer.Informer().HasSynced return dc, nil }
我们看到，这个方法大致做了三件事。
（1）创建了一个Broadcaster，用于做kubernetes中event资源相关的处理。
（2）创建DeploymentController
我们知道，Deployment并不直接操作集群中的pod，而是通过操作ReplicaSet来间接操作pod，因此我们可以看到，DeploymentController结构体中也包含rsControl这一字段，用于操作ReplicaSet。而queue字段则维护了一个deployment的队列，用于将需要更新状态的deployment元素存入，后面会讲到。
（3）添加回调函数
Informer的AddEventHandler方法为deployment controller的所有informer添加了回调函数，对deployment和ReplicaSet的添加、更新、删除进行相应的处理。
下面我们详细分析一下添加回调函数操作，这里分成三部分说，一是Informer的具体创建，二是函数被调用的逻辑，三是回调函数本身。
四、Informer()
在Deployment Controller中包含了deployment、replicaset、pod三个informer，都是各自的Informer()方法创建各自的informer。
以方法中dInformer.Informer()方法为例。方法本质上调用了InformerFor方法：
k8s.io/client-go/informers/apps/v1/deployment.go
func (f *deploymentInformer) Informer() cache.SharedIndexInformer {
return f.factory.InformerFor(&apps_v1.Deployment{}, f.defaultInformer)
}
InformerFor方法比较直观，就是调用defaultInformer方法，将生成的informer存入factory的informer字段中。
defaultInformer方法则是调用了NewFilteredDeploymentInformer方法：
k8s.io/client-go/informers/apps/v1/deployment.go
func NewFilteredDeploymentInformer(client kubernetes.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers, tweakListOptions internalinterfaces.TweakListOptionsFunc) cache.SharedIndexInformer { return cache.NewSharedIndexInformer( &cache.ListWatch{ ListFunc: func(options meta_v1.ListOptions) (runtime.Object, error) { if tweakListOptions != nil { tweakListOptions(&options) } return client.AppsV1().Deployments(namespace).List(options) }, WatchFunc: func(options meta_v1.ListOptions) (watch.Interface, error) { if tweakListOptions != nil { tweakListOptions(&options) } return client.AppsV1().Deployments(namespace).Watch(options) }, }, &apps_v1.Deployment{}, resyncPeriod, indexers, ) }
这个方法直接就返回了一个NewSharedIndexInformer结构体。可以看到，这个结构体内部，定义了这个Informer的List和Watch函数，与一般用户通过client-go连接API Server并无二致。
五、AddEventHandler
首先看调用的逻辑。AddEventHandler方法除了为Informer添加新的回调函数外，还同时向这些新回调函数如何分发消息，使这些回调函数被调用。AddEventHandler方法只有一行，即调用AddEventHandlerWithResyncPeriod方法。
进入AddEventHandlerWithResyncPeriod方法：
k8s.io/client-go/tools/cache/shared_informer.go func (s *sharedIndexInformer) AddEventHandlerWithResyncPeriod(handler ResourceEventHandler, resyncPeriod time.Duration) { s.startedLock.Lock() defer s.startedLock.Unlock() if s.stopped { glog.V(2).Infof("Handler %v was not added to shared informer because it has stopped already", handler) return } if resyncPeriod > 0 { ... } listener := newProcessListener(handler, resyncPeriod, determineResyncPeriod(resyncPeriod, s.resyncCheckPeriod), s.clock.Now(), initialBufferSize) if !s.started { s.processor.addListener(listener) return } // in order to safely join, we have to // 1. stop sending add/update/delete notifications // 2. do a list against the store // 3. send synthetic "Add" events to the new handler // 4. unblock s.blockDeltas.Lock() defer s.blockDeltas.Unlock() s.processor.addListener(listener) for _, item := range s.indexer.List() { listener.add(addNotification{newObj: item}) } }
我们看到，这里引入了一个listener的概念。listener是informer用于获取Thread Safe Store元素并处理的机制。通过调用addListener方法，为informer添加并运行listener。
随后，又遍历Thread Safe Store中的元素，将所有的消息都发送给新添加的EventHandler。这里的目的是让新EventHandler先处理一遍Thread Safe Store中已有的元素，不然它们就只能接收到新消息，没有机会处理旧消息了。
六、addListener
listener的添加和运行都通过addListener方法进行。进入addListener方法：
k8s.io/client-go/tools/cache/shared_informer.go
func (p *sharedProcessor) addListener(listener *processorListener) {
p.listenersLock.Lock()
defer p.listenersLock.Unlock()
p.addListenerLocked(listener)
if p.listenersStarted {
p.wg.Start(listener.run)
p.wg.Start(listener.pop)
}
}
addListener方法通过goroutine的方式，调用listener的run和pop方法，运行listener。
分别进入run和pop方法：
k8s.io/client-go/tools/cache/shared_informer.go func (p *processorListener) run() { stopCh := make(chan struct{}) wait.Until(func() { // this gives us a few quick retries before a long pause and then a few more quick retries err := wait.ExponentialBackoff(retry.DefaultRetry, func() (bool, error) { for next := range p.nextCh { switch notification := next.(type) { case updateNotification: p.handler.OnUpdate(notification.oldObj, notification.newObj) case addNotification: p.handler.OnAdd(notification.newObj) case deleteNotification: p.handler.OnDelete(notification.oldObj) default: utilruntime.HandleError(fmt.Errorf("unrecognized notification: %#v", next)) } } // the only way to get here is if the p.nextCh is empty and closed return true, nil }) // the only way to get here is if the p.nextCh is empty and closed if err == nil { close(stopCh) } }, 1*time.Minute, stopCh) }
k8s.io/client-go/tools/cache/shared_informer.go func (p *processorListener) pop() { defer utilruntime.HandleCrash() defer close(p.nextCh) // Tell .run() to stop var nextCh chan<- interface{} var notification interface{} for { select { case nextCh <- notification: // Notification dispatched var ok bool notification, ok = p.pendingNotifications.ReadOne() if !ok { // Nothing to pop nextCh = nil // Disable this select case } case notificationToAdd, ok := <-p.addCh: if !ok { return } if notification == nil { // No notification to pop (and pendingNotifications is empty) // Optimize the case - skip adding to pendingNotifications notification = notificationToAdd nextCh = p.nextCh } else { // There is already a notification waiting to be dispatched p.pendingNotifications.WriteOne(notificationToAdd) } } } }
pop方法用到好几个channel，本质上仍是从channel中取出一个notification，并通过run方法进行处理。可以看到，在run方法中有对于notification的类型选择，并分别调用OnUpdate、OnAdd、OnDelete方法进行处理，而这三个方法则分别调用了前面AddEventHandler方法中传进来的回调函数，即：
pkg/controller/deployment/deployment_controller.go
func NewDeploymentController(dInformer appsinformers.DeploymentInformer, rsInformer appsinformers.ReplicaSetInformer, podInformer coreinformers.PodInformer, client clientset.Interface) (*DeploymentController, error) { ... dInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: dc.addDeployment, UpdateFunc: dc.updateDeployment, DeleteFunc: dc.deleteDeployment, }) rsInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: dc.addReplicaSet, UpdateFunc: dc.updateReplicaSet, DeleteFunc: dc.deleteReplicaSet, }) podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ DeleteFunc: dc.deletePod, }) ... }
那么这些notification从何而来呢？答案是，通过listener的add方法添加，即前述AddEventHandlerWithResyncPeriod方法的最后一句：
k8s.io/client-go/tools/cache/shared_informer.go
func (s *sharedIndexInformer) AddEventHandlerWithResyncPeriod(handler ResourceEventHandler, resyncPeriod time.Duration) {
...
for _, item := range s.indexer.List() {
listener.add(addNotification{newObj: item})
}
}
add方法只有一行，就是将信息传入listener的channel中。而信息则是通过indexer的List方法，从Thread Safe Store中获得。此外，上一篇文章中也提到过，informer会不断从FIFO中pop元素，进行处理后向listener发送信息。所以对于EventHandler来说，一方面在刚注册时，从Thread Safe Store中一次性获得所有旧的消息，一方面从FIFO队列中一个一个地获得消息。
至此，informer的逻辑就很清晰了。首先通过ListAndWatch方法，将watch到的资源状态的变化存入FIFO队列。之后，在启动controller后，通过factory生成相应的informer，并通过AddEventHandler方法添加相应的回调函数。AddEventHandler方法会为informer添加listener，将Thread Safe Store队列中的内容取出，根据其状态调用相应的回调函数进行处理。同时FIFO队列中的元素也会一个一个pop出来，供informer消费。
剩下的内容，将在下一篇文章中分析。下一篇文章链接https://www.cnblogs.com/00986014w/p/10570488.html
