的工作原理本文源码基于Kubernetesv1.24.0，容器运行Containerd1.5。从源码分析kubectlport-forward的工作原理。通过对port-forward流程的分析，梳理kubectl->api-server->kubelet->containerruntime的交互，了解cri的工作原理。kubectl-port-forwardkubectl只是创建一个pod：kubectlrunpipy--imageflomesh/pipy:latest-ndefault添加参数-v9执行kubectlforward时打印日志。kubectl端口转发pipy8080-v9...I080721:45:58.45798614495round_trippers.go:466]curl-v-XPOST-H“用户代理：kubectl/v1.24.3(darwin/arm64)kubernetes/aef86a9"-H"X-Stream-Protocol-Version:portforward.k8s.io"'https://192.168.1.12:6443/api/v1/namespaces/default/pods/pipy/portforward'I080721:45:58.48401314495round_trippers.go:553]POSThttps://192.168.1.12:6443/api/v1/namespaces/default/pods/pipy/portforward101在26毫秒内切换协议I080721:45:58.48402914495round_trippers.go:570]HTTP统计:DNSLookup0msDial0msTLSHandshake0msDuration26msI080721:45:58.48403514495round_trippers.go:577]ResponseHeaders:I080721:45:58.48404014495round_trippers.go:580]升级：SPDY/3.11I0457:58.48404414495round_trippers.go:580]X-Stream-Protocol-Version:portforward.k8s.ioI080721:45:58.48404714495round_trippers.go:580]日期：2022年8月7日星期日13:45:58GMTI080721:45:58.48405114495round_trippers.go:580]Connection:UpgradeForwardingfrom127.0.0.1:8080->8080Forwardingfrom[::1]:8080->8080从日志中我们可以看到请求的地址是/api/v1/namespaces/default/pods/pipy/portforward，其中portforward是pod资源的子资源，这里使用的协议是spdy。此时kubectl会监听本地端口，并使用pod子资源portforward的url创建到api-server的连接。当本地端口有连接访问时，kubectl会在两个连接之间不断复制数据。参考源码：staging/src/k8s.io/kubectl/pkg/cmd/portforward/portforward.go:389[1]staging/src/k8s.io/client-go/tools/portforward/portforward.go:242[2]staging/src/k8s.io/client-go/tools/portforward/portforward.go:330[3]api-serverpodexec、attach和portforward这三个子资源，将表示对这三个资源的操作由对应节点的kubetletserver进行处理。api-server收到访问pod子资源portforward的请求后，通过pod及其节点的信息获取访问该节点上kubelet服务器的url。然后将访问pod的portforward请求代理到kubelet服务器。参考源码pkg/registry/core/pod/rest/subresources.go:185[4]kubeletportforward请求来到pod所在节点的kubeletserver。在kubeletserver中，有几个用于调试的端点，portforward就是其中之一：/run/{podNamespace}/{podID}/{containerName}/exec/{podNamespace}/{podID}/{containerName}/attach/{podNamespace}/{podID}/{containerName}/portforward/{podNamespace}/{podID}/containerLogs/{podNamespace}/{podID}/{containerName}/runningpods/kubeletserver收到请求后，会先发送通过RuntimeServiceClient（/runtime.v1alpha2.RuntimeService/PortForward）向容器运行时接口发起gRCP请求，获取容器运行时streamingserver处理的pordforward请求的url。kubeletserver获取到portforwardstreamingurl后，会将请求代理到该url。参考源码pkg/kubelet/server/server.go:463[5]pkg/kubelet/server/server.go:873[6]pkg/kubelet/cri/streaming/portforward/portforward.go:46[7]pkg/kubelet/cri/streaming/server.go:111[8]cri这里以Containerd为例。Containerd在启动的时候会启动runtimeservice和imageservice。前者负责容器相关操作，后者负责镜像相关操作。kubelet通过调用运行时服务的gRPC接口获取端口转发的流式url。除了这两个gRPC服务外，还加载了一系列插件。这些插件之一是cri服务。cri服务将启动流媒体服务器。此服务器将响应/exec、/attach和/portforward流请求。portforward支持linux和windows两种操作系统：sandbox_portforward_linux.go和sandbox_portforward_windows.go。在linux上，在pod所在的网络命名空间中使用地址localhost创建到目标端口的连接。然后在流媒体服务器的连接和连接之间复制数据，完成数据传输。在Windows上，到目标端口的连接由wincat.exe使用地址127.0.0.1创建。参考源码pkg/cri/streaming/server.go:149[9]pkg/cri/server/streaming.go:69[10]pkg/cri/server/service.go:138[11]pkg/cri/server/sandbox_portforward_linux.go:34[12]结合源码分析总结了port-foward的工作原理，相信对cri的工作方式有了一定的了解。本文以容器运行时Containerd为例。虽然不同的容器运行时实现了cri，但是实现的细节会有所不同。例如port-forward的实现，在Kubernetesv1.23.0（1.24中移除）[13]中的dockershim中，使用nsenter进入pod所在的networknamespace，通过socat完成端口转发。参考资料[1]staging/src/k8s.io/kubectl/pkg/cmd/portforward/portforward.go:389:https://github.com/kubernetes/kubernetes/tree/release-1.24/staging/src/k8s.io/kubectl/pkg/cmd/portforward/portforward.go#L389[2]staging/src/k8s.io/client-go/tools/portforward/portforward.go:242:https://github.com/kubernetes/kubernetes/tree/release-1.24/staging/src/k8s.io/client-go/tools/portforward/portforward.go#L242[3]staging/src/k8s.io/client-go/tools/portforward/portforward.go:330:https://github.com/kubernetes/kubernetes/tree/release-1.24/staging/src/k8s.io/client-go/tools/portforward/portforward.go#L330[4]pkg/registry/core/pod/rest/subresources.go:185:https://github.com/kubernetes/kubernetes/tree/release-1.24/pkg/registry/core/pod/rest/subresources.go#L185[5]pkg/kubelet/server/server.go:463:https://github.com/kubernetes/kubernetes/tree/release-1.24/pkg/kubelet/server/server.go#L463[6]pkg/kubelet/server/server.go:873:https://github.com/kubernetes/kubernetes/tree/release-1.24/pkg/kubelet/server/server.go#L873[7]pkg/kubelet/cri/streaming/portforward/portforward.go:46:https://github.com/kubernetes/kubernetes/tree/release-1.24/pkg/kubelet/cri/streaming/portforward/portforward.go#L46[8]pkg/kubelet/cri/streaming/server.go:111:https://github.com/kubernetes/kubernetes/tree/release-1.24/pkg/kubelet/cri/streaming/server.go#L111[9]pkg/cri/streaming/server.go:149:https://github.com/containerd/containerd/tree/release/1.5/pkg/cri/streaming/server.go#L149[10]pkg/cri/server/streaming.go:69:https://github.com/containerd/containerd/tree/release/1.5/pkg/cri/server/streaming.go#L69[11]pkg/cri/server/service.go:138:https://github.com/containerd/containerd/tree/release/1.5/pkg/cri/server/service.go#L138[12]pkg/cri/server/sandbox_portforward_linux.go:34:https://github.com/containerd/containerd/tree/release/1.5/pkg/cri/server/sandbox_portforward_linux.go#L34[13]Kubernetesv1.23.0版本中的dockershim（1.24中被移除）：https://github.com/kubernetes/kubernetes/blob/release-1.23/pkg/kubelet/dockershim/docker_streaming_others.go#L43
