Kubernetes Architectural Roadmap

Shared with the community
Status: First draft
Last update: 4/20/2017
Authors: Brian Grant, Tim Hockin, and Clayton Coleman
Intended audience: Kubernetes contributors

Table of Contents

• Kubernetes Architectural Roadmap
  • Summary/TL;DR
  • Background
  • System Layers
    • The Nucleus: API and Execution
      • The API and cluster control plane
      • Execution
    • The Application Layer: Deployment and Routing
    • The Governance Layer: Automation and Policy Enforcement
    • The Interface Layer: Libraries and Tools
    • The Ecosystem
  • Managing the matrix
  • Layering of the system as it relates to security
  • Next Steps

Summary/TL;DR#

This document describes the ongoing architectural development of the Kubernetes system, and the
motivations behind it. System developers wanting to extend or customize Kubernetes should use this document as a guide to inform where and how best to implement these enhancements. Application developers wanting to develop large, portable and/or future-proof Kubernetes applications may refer to this document for guidance on which parts of Kubernetes they can rely on being present now and in the future.

本文档介绍了 Kubernetes 系统正在进行的架构开发及其背后的动机。想要扩展或自定义 Kubernetes 的系统开发人员应使用本文档作为指导手册，以了解在何处、如何最好地实施这些增强功能。希望开发大型、可移植和 / 或面向未来的 Kubernetes 应用程序的开发人员可以参考本文档，了解他们现在和将来可以依赖 Kubernetes 的哪些部分。

The layers of the architecture are named and described (see the diagram below). Distinctions are drawn between what exists today and what we plan to provide in future, and why.
Succinctly, the layers comprise:

1. The Nucleus which provides standardized API and execution machinery, including basic REST mechanics, security, individual Pod, container, network interface and storage volume management, all of which are extensible via well-defined interfaces.  The Nucleus is non-optional and expected to be the most stable part of the system.
2. The Application Management Layer which provides basic deployment and routing, including self-healing, scaling, service discovery, load balancing and traffic routing.  This is often referred to as orchestration and the service fabric. Default implementations of all functions are provided, but conformant replacements are permitted.
3. The Governance Layer which provides higher level automation and policy enforcement, including single- and multi-tenancy, metrics, intelligent autoscaling and provisioning, and schemes for authorization, quota, network, and storage policy expression and enforcement. These are optional, and achievable via other solutions.
4. The Interface Layer which provides commonly used libraries, tools, UI's and systems used to interact with the Kubernetes API.
5. The Ecosystem which includes everything else associated with Kubernetes, and is not really "part of" Kubernetes at all.  This is where most of the development happens, and includes CI/CD, middleware, logging, monitoring, data processing, PaaS, serverless/FaaS systems, workflow, container runtimes, image registries, node and cloud provider management, and many others.

该体系结构的层级结构如图所示。现有的东西和我们计划在未来提供的（以及为什么提供）等东西是有区别的。
简洁地说，这些层包括:

1. 内核，提供了标准 API 和执行机制，包括基本的 REST 机制、安全、独立 Pod、容器、网络接口和存储卷管理，借助良好定义的接口，这些都是可扩展的。内核是必须的，期望成为整个系统最稳定的部分。
2. 应用管理层，提供基本的部署和路由，包括自我修复、扩缩容、服务发现、负载均衡和流量路由。这通常被称为服务编排。所有功能都提供了默认实现，但允许用符合要求的组件进行替换。
3. 治理层，提供更高级别的自动化和策略实施，包括单租户和多租户、指标、智能自动伸缩和供应，以及授权、配额、网络和存储策略表达和实施方案。这些都是可选的，可通过其他解决方案来实现。
4. 接口层，提供通用库，工具，UI 和用于与 Kubernetes API 交互的系统。
5. 生态系统，包括与 Kubernetes 有关的一切，它们实际上不是 Kubernetes 的一部分。这是进行大多数开发的地方，包括 CI/CD，中间件，日志，监控，数据处理，PaaS，serverless/FaaS 系统，工作流，容器运行时，镜像仓库，节点以及云供应商管理等。

背景 Background#

Kubernetes 是一个用于部署和管理容器的平台。了解关于 Kubernetes 的任务、范围以及设计的更多信息，参见 Kubernetes 是什么和架构概览。 后者还描述了当前将系统细分为组件 / 流程的情况。

Kubernetes is a platform for deploying and managing containers. For more information about the mission, scope, and design of Kubernetes, see What Is Kubernetes and the architectural overview. The latter also describes the current breakdown of the system into components/processes.

Kubernetes 的贡献者需要知道他们在系统的不同部分添加新特性时可以依赖什么功能。

Contributors to Kubernetes need to know what functionality they can rely upon when adding new features to different parts of the system.

此外，像 Kubernetes 这样的平台面临的问题之一是定义什么是 “内”，什么是 “外”。虽然 Kubernetes 必须提供一些基本功能，用户在运行容器化应用程序或构建扩展时可以依赖这些功能，但 Kubernete 不能也不应该尝试解决用户遇到的所有问题。更为困难的是，与数据库、负载平衡器或消息传递系统等其他类型的基础设施服务不同，“内置” 功能几乎没有明显的自然边界。因此，理性的人们可能会在界限的确切位置或指导决策的原则上产生分歧。

Additionally, one of the problems that faces platforms like Kubernetes is to define what is "in" and what is “out”. While Kubernetes must offer some base functionality on which users can rely when running their containerized applications or building their extensions, Kubernetes cannot and should not try to solve every problem that users have. Adding to the difficulty is that, unlike some other types of infrastructure services such as databases, load balancers, or messaging systems, there are few obvious, natural boundaries for the “built-in” functionality. Consequently, reasonable minds can disagree on exactly where the boundaries lie or what principles guide the decisions.

本文档受到社区类似努力（文件已不存在）的启发，旨在阐明 Kubernetes 架构 SIG（Specific Interesting Group）的意图。它目前是雄心勃勃的，旨在成为当前和未来发展的蓝图。NIY 在本文件的开头标记了截至更新日期尚未实施的项目。演示版本

This document, which was inspired by similar efforts from the community, aims to clarify the intentions of the Kubernetes’s architecture SIG. It is currently somewhat aspirational, and is intended to be a blueprint for ongoing and future development. NIY marks items not yet implemented as of the lated updated date at the head of this document. Presentation version

系统层 System Layers#

正如 Linux 有内核、核心系统库和可选的用户界面工具一样，Kubernetes 也有功能和工具的 “层”。理解这些层对于 Kubernetes 功能的开发人员来说非常重要，可以治澳哪些跨概念依赖应该被允许，哪些不应该被允许。
Kubernetes API、概念和功能可以分为以下几层。

Just as Linux has a kernel, core system libraries, and optional userland tools, Kubernetes also has "layers" of functionality and tools. An understanding of these layers is important for developers of Kubernetes functionality to determine which cross-concept dependencies should be allowed and which should not.
Kubernetes APIs, concepts, and functionality can be sorted into the following layers

内核：API 和执行 The Nucleus: API and Execution#

基本 API 和执行机制。
这些 API 和功能由上游 Kubernetes 代码库实现，包括构建系统高阶层所需的最基本的特性和概念集。这些部件都经过了详细的规定和记录，每个容器化应用都会使用它们。开发人员可以安全地假设他们总是存在的。
他们最终应该变得稳定和 “无聊”。然而，Linux 在其 25 年的生命周期中不断发展，在过去 10 年中添加了许多重大功能，包括 NPTL（需要重建所有应用程序）和启用容器（这些功能正在改变所有应用程序的运行方式）等功能。Kubernetes 也需要一段时间才能稳定下来。

Essential API and execution machinery.
These APIs and functions, implemented by the upstream Kubernetes codebase, comprise the bare minimum set of features and concepts needed to build up the higher-order layers of the system.  These pieces are thoroughly specified and documented, and every containerized application will use them. Developers can safely assume they are present.
They should eventually become stable and "boring". However, Linux has continuously evolved over its 25-year lifetime and major changes, including NPTL (which required rebuilding all applications) and features that enable containers (which are changing how all applications are run), have been added over the past 10 years. It will take some time for Kubernetes to stabilize, as well.

API 和集群控制面 The API and cluster control plane#

Kubernetes 集群提供了一组类似的 REST API，由 Kubernetes API 服务器暴露，主要支持（大部分）持久资源上的 CRUD 操作。这些 API 充当其控制平面的中枢。

Kubernetes clusters provide a collection of similar REST APIs, exposed by the Kubernetes API server, supporting primarily CRUD operations on (mostly) persistent resources. These APIs serve as the hub of its control plane.

遵循 Kubernetes API 约定（路径约定、标准元数据等）的 REST API 能受益于共享 API 的服务（授权、身份验证、审计日志记录），通用客户端代码可以与之交互（ CLI 和 UI 可发现性、UI 和 CLI 中的通用状态报告、编排工具的通用等待约定、监视、标签选择）。

REST APIs that follow Kubernetes API conventions (path conventions, standard metadata, …) are automatically able to benefit from shared API services (authorization, authentication, audit logging) and generic client code can interact with them (CLI and UI discoverability, generic status reporting in UIs and CLIs, generic waiting conventions for orchestration tools, watching, label selection).

系统的最底层还需要支持必要的扩展机制，以添加高层提供的功能。此外，该层必须适用于单用途集群和高度租用的集群。核心应该提供足够的灵活性，以便更高级别的 API 可以在不损害集群安全模型的情况下引入新的作用域（资源集）。

The lowest layer of the system also needs to support extension mechanisms necessary to add the functionality provided by the higher layers. Additionally, this layer must be suitable for use both in single-purpose clusters and highly tenanted clusters. The nucleus should provide sufficient flexibility that higher-level APIs could introduce new scopes (sets of resources) without compromising the security model of the cluster.

Kubernetes 不能在没有基本 API 机制和语义的情况下工作，包括：

Kubernetes cannot function without this basic API machinery and semantics, including:

• 身份验证：身份验证方案是一项关键功能，必须得到服务器和客户端的同意。API 服务器支持基本身份验证（用户名 / 密码）（注意：我们可能最终会弃用基本身份验证。）、X.509 客户端证书、OpenID Connect 令牌和承载令牌，其中任何一个（但不是全部）都可能被禁用。客户端应支持 kubeconfig 支持的所有表单。第三方认证系统可以实现 TokenReview API 并配置认证 webhook 来调用它，尽管选择非标准认证机制可能会限制可用客户端的数量。
  • TokenReview API（与 hook 模式相同）支持外部身份验证检查，例如 Kubelet
  • 通过 “服务账号” 提供的 Pod 识别
    • ServiceAccount API，包括通过控制器创建的默认服务账号 Secret 和通过准入控制器注入的。

• Authentication: The authentication scheme is a critical function that must be agreed upon by both the server and clients. The API server supports basic auth (username/password) (NOTE: We will likely deprecate basic auth eventually.), X.509 client certificates, OpenID Connect tokens, and bearer tokens, any of which (but not all) may be disabled. Clients should support all forms supported by kubeconfig. Third-party authentication systems may implement the TokenReview API and configure the authentication webhook to call it, though choice of a non-standard authentication mechanism may limit the number of usable clients.
  • The TokenReview API (same schema as the hook) enables external authentication checks, such as by Kubelet
  • Pod identity is provided by "service accounts"
    • The ServiceAccount API, including default ServiceAccount secret creation via a controller and injection via an admission controller.

• 鉴权：第三方鉴权系统可以实现 SubjectAccessReview API 并配置鉴权 webook 并调用它。
  • SubjectAccessReview（与 hook 相同的模式）、LocalSubjectAccess Review 和 SelfSubjectAccountReview API 支持外部权限检查，如 Kubelet 和其他控制器

• Authorization: Third-party authorization systems may implement the SubjectAccessReview API and configure the authorization webhook to call it.
  • The SubjectAccessReview (same schema as the hook), LocalSubjectAccessReview, and SelfSubjectAccessReview APIs enable external permission checks, such as by Kubelet and other controllers

• REST 语义、监视、持久性和一致性保证、API 版本控制、默认设置和验证
  • NIY：需要解决的 API 缺陷：
    • Confusing defaulting behavior
    • Lack of guarantees
    • Orchestration support
    • Support for event-driven automation
    • Clean teardown

• REST semantics, watch, durability and consistency guarantees, API versioning, defaulting, and validation
  • NIY: API deficiencies that need to be addressed:
    • Confusing defaulting behavior
    • Lack of guarantees
    • Orchestration support
    • Support for event-driven automation
    • Clean teardown

• NIY：内置接纳控制语义、同步接纳控制挂钩和异步资源初始化 —— 分发供应商、系统集成商和集群管理员需要能够实施额外的策略和自动化

• NIY，包括 API 聚合，以注册其他 API，找出支持哪些 API，并获取支持的操作、有效负载和结果模式的详细信息

• NIY API（或其继任者），以支持第三方存储和扩展 API

• NIY：一个可扩展和 HA 兼容的 / componentstatus API 替代品，用于确定集群是否已完全启动并正确运行：ExternalServiceProvider（组件注册）

• NIY: Built-in admission-control semantics, synchronous admission-control hooks, and asynchronous resource initialization -- it needs to be possible for distribution vendors, system integrators, and cluster administrators to impose additional policies and automation

• NIY: API registration and discovery, including API aggregation, to register additional APIs, to find out which APIs are supported, and to get the details of supported operations, payloads, and result schemas

• NIY: ThirdPartyResource and ThirdPartyResourceData APIs (or their successors), to support third-party storage and extension APIs

• NIY: An extensible and HA-compatible replacement for the /componentstatuses API to determine whether the cluster is fully turned up and operating correctly: ExternalServiceProvider (component registration)

• 端点 API（及其未来发展），用于组件注册、API 服务器端点的自发布、HA 会合以及应用层目标发现

• 命名空间 API，它是确定用户资源范围和命名空间生命周期（例如，批量删除）的方法

• 事件 API，它是报告重要事件（如状态更改和错误）以及事件垃圾收集的方法

• The Endpoints API (and future evolutions thereof), which is needed for component registration, self-publication of API server endpoints, and HA rendezvous, as well as application-layer target discovery

• The Namespace API, which is the means of scoping user resources, and namespace lifecycle (e.g., bulk deletion)

• The Event API, which is the means of reporting significant occurrences, such as status changes and errors, and Event garbage collection

• NIY：级联删除垃圾收集器、终结和孤立

• NIY：我们需要一个内置的插件管理器（不像静态 pod 清单，而是在集群级别），这样我们可以自动向集群添加自托管组件和动态配置，这样我们就可以从运行集群中的现有组件中排除功能。它的核心将是一个基于拉的声明性协调器，由当前的插件管理器提供，并在白盒应用程序管理文档中进行了描述。一旦我们在 API 中应用了支持，这将更加容易。

  • 附加组件应该是作为集群的一部分进行管理的集群服务，并提供与集群提供的多租户相同程度的多租约。
  • 它们可以（但不是必须）在 kube 系统命名空间中运行，但需要选择所选的命名空间，使其不会与用户的命名空间冲突。

• API 服务器充当集群的网关。根据定义，API 服务器必须可由集群外部的客户端访问，而节点（当然是 pod）可能无法访问。客户端使用/proxy 和/portforward API 对 API 服务器进行身份验证，并将其用作到节点和 pod（和服务）的堡垒和代理/隧道。

• 待定：CertificateSigningRequest API，用于生成凭证，特别是生成 Kubelet 凭证。理想情况下，此核 API 服务器将只支持最低要求的 API，并且将通过聚合、挂钩、初始值设定项和其他扩展机制添加附加功能。

• NIY: Cascading-deletion garbage collector, finalization, and orphaning

• NIY: We need a built-in add-on manager (not unlike static pod manifests, but at the cluster level) so that we can automatically add self-hosted components and dynamic configuration to the cluster, and so we can factor out functionality from existing components in running clusters. At its core would be a pull-based declarative reconciler, as provided by the current add-on manager and as described in the whitebox app management doc. This would be easier once we have apply support in the API.

  • Add-ons should be cluster services that are managed as part of the cluster and that provide the same degree of multi-tenancy as that provided by the cluster.
  • They may, but are not required to, run in the kube-system namespace, but the chosen namespace needs to be chosen such that it won't conflict with users' namespaces.

• The API server acts as the gateway to the cluster. By definition, the API server must be accessible by clients from outside the cluster, whereas the nodes, and certainly pods, may not be. Clients authenticate the API server and also use it as a bastion and proxy/tunnel to nodes and pods (and services), using /proxy and /portforward APIs.

• TBD: The CertificateSigningRequest API, to enable credential generation, in particular to mint Kubelet credentials Ideally, this nuclear API server would only support the minimum required APIs, and additional functionality would be added via aggregation, hooks, initializers, and other extension mechanisms.

请注意，集中式异步控制器（如垃圾收集）目前由一个称为控制器管理器的单独进程运行。
/healthz 和/metrics 端点可以用于集群管理机制，但不被视为受支持的 API 表面的一部分，并且通常不应由客户端使用以检测集群存在。应改用/version 终结点。
Note that the centralized asynchronous controllers, such as garbage collection, are currently run by a separate process, called the Controller Manager.
/healthz and /metrics endpoints may be used for cluster management mechanisms, but are not considered part of the supported API surface, and should not be used by clients generally in order to detect cluster presence. The /version endpoint should be used instead.

API Server 依赖于以下几个外部组件：

• 持久化状态存储（etcd，或其他等效的；可能多实例）

API Server 可能依赖于：

• 证书认证
• 身份提供者
• TokenReview API 实现
• SubjectAccessReview API 实现者

The API server depends on the following external components:

• Persistent state store (etcd, or equivalent; perhaps multiple instances)

The API server may depend on:

• Certificate authority
• Identity provider
• TokenReview API implementer
• SubjectAccessReview API implementer

执行 Execution#

Kubernetes 中最重要、最突出的控制器是 Kubelet，它是驱动容器执行层的 Pod 和 Node API 的主要实现者。如果没有这些 API，Kubernetes 将只是一个由键值存储支持的面向 CRUD 的 REST 应用程序框架（也许 API 机制最终将作为一个独立的项目分离出来）。
Kubernetes 执行独立的应用程序容器作为其默认的本机执行模式。Kubernetes 提供了可以承载多个容器和存储卷的 Pod 作为其基本执行原语。
The most important and most prominent controller in Kubernetes is the Kubelet, which is the primary implementer of the Pod and Node APIs that drive the container execution layer. Without these APIs, Kubernetes would just be a CRUD-oriented REST application framework backed by a key-value store (and perhaps the API machinery will eventually be spun out as an independent project).
Kubernetes executes isolated application containers as its default, native mode of execution. Kubernetes provides Pods that can host multiple containers and storage volumes as its fundamental execution primitive.
Kubelet API 表面和语义包括：

• Pod API，Kubernetes 执行原语，包括：
  • 基于 Pod API 策略的基于 Pod 可行性的接纳控制（资源请求、节点选择器、节点 / Pod 亲和性和反亲和性、污点和容忍度）。API 准入控制可能会拒绝 pod 或向其添加额外的调度约束，但 Kubelet 是 pod 可以在给定节点上运行和不能在给定节点中运行的最终仲裁者，而不是调度器或守护程序集。
  • 容器和卷语义及生命周期
  • Pod IP 地址分配（每个 pod 都需要的可路由 IP 地址）
  • 将 Pod 绑定到特定安全范围的机制（即 ServiceAccount）
  • 卷资源
    • emptyDir
    • hostPath
    • secret
    • configMap
    • downwardAPI
    • NIY: Container and image volumes (and deprecate gitRepo) 容器和镜像卷（以及遗弃的 gitRepo ）
    • NIY: Claims against local storage, so that complex templating or separate configs are not needed for dev vs. prod application manifests 针对本地存储的声明，因此开发与生产应用程序清单不需要复杂的模板或单独的配置
    • flexVolume (which should replace built-in cloud-provider-specific volumes) flexVolume（应取代内置云提供商特定卷）
  • 子资源：binding、status、exec、logs、attach、portforward、proxy
  • NIY：API 资源可用性和 bootstrap 检查点
  • 容器映像和日志生命周期
  • Secret API，以及支持第三方秘密管理的机制
  • ConfigMap API，用于组件配置和 Pod 参考
• 节点 API，Pod 的宿主机
  • 在某些配置中，只能对群集管理员可见
• 节点和 Pod 网络及其控制器（路由控制器）
• 节点资源清册、运行状况和可达性（节点控制器）
  • 云提供商特定的节点清单功能应拆分为提供商特定的控制器。
• 已终止 Pod 垃圾收集
• 卷控制器
  • 云提供商特定的连接 / 分离逻辑应拆分为提供商特定的控制器。需要一种从 API 中提取特定于提供商的卷源的方法。
• PersistentVolume API
  • NIY：至少有本地存储支持，如上所述
• PersistentVolumeClaim API

同样，集中式异步功能（如终止的 pod 垃圾收集）由控制器管理器执行。
The Kubelet API surface and semantics include:

• The Pod API, the Kubernetes execution primitive, including:
  • Pod feasibility-based admission control based on policies in the Pod API (resource requests, node selector, node/pod affinity and anti-affinity, taints and tolerations). API admission control may reject pods or add additional scheduling constraints to them, but Kubelet is the final arbiter of what pods can and cannot run on a given node, not the schedulers or DaemonSets.
  • Container and volume semantics and lifecycle
  • Pod IP address allocation (a routable IP address per pod is required)
  • A mechanism (i.e., ServiceAccount) to tie a Pod to a specific security scope
  • Volume sources:
    • emptyDir
    • hostPath
    • secret
    • configMap
    • downwardAPI
    • NIY: Container and image volumes (and deprecate gitRepo)
    • NIY: Claims against local storage, so that complex templating or separate configs are not needed for dev vs. prod application manifests
    • flexVolume (which should replace built-in cloud-provider-specific volumes)
  • Subresources: binding, status, exec, logs, attach, portforward, proxy
• NIY: Checkpointing of API resources for availability and bootstrapping
• Container image and log lifecycles
• The Secret API, and mechanisms to enable third-party secret management
• The ConfigMap API, for component configuration as well as Pod references
• The Node API, hosts for Pods
  • May only be visible to cluster administrators in some configurations
• Node and pod networks and their controllers (route controller)
• Node inventory, health, and reachability (node controller)
  • Cloud-provider-specific node inventory functions should be split into a provider-specific controller.
• Terminated-pod garbage collection
• Volume controller
  • Cloud-provider-specific attach/detach logic should be split into a provider-specific controller. Need a way to extract provider-specific volume sources from the API.
• The PersistentVolume API
  • NIY: At least backed by local storage, as mentioned above
• The PersistentVolumeClaim API

Again, centralized asynchronous functions, such as terminated-pod garbage collection, are performed by the Controller Manager.
控制器管理器和 Kubelet 目前调用 “云提供商” 接口来查询基础设施层的信息和/或管理基础设施资源。然而，我们正在努力将这些接触点（问题）提取到外部组件中。预期的模型是，不可满足的应用程序/容器/OS 级请求（例如 Pods、PersistentVolumeClaims）用作外部 “动态供应” 系统的信号，该系统将使基础设施可用以满足这些请求，并使用基础设施资源（例如 Nodes、PersitentVolumes）在 Kubernetes 中表示这些请求，这样 Kubernetes 可以将请求和基础设施资源绑定在一起。
The Controller Manager and Kubelet currently call out to a "cloud provider" interface to query information from the infrastructure layer and/or to manage infrastructure resources. However, we’re working to extract those touchpoints (issue) into external components. The intended model is that unsatisfiable application/container/OS-level requests (e.g., Pods, PersistentVolumeClaims) serve as a signal to external “dynamic provisioning” systems, which would make infrastructure available to satisfy those requests and represent them in Kubernetes using infrastructure resources (e.g., Nodes, PersistentVolumes), so that Kubernetes could bind the requests and infrastructure resources together.

Kubelet 依赖一下几个外部组件

• 镜像仓库
• CRI 实现
• CNI 实现
• FlexVolume 实现（CVI）CSI

The Kubelet depends on the following external components:

• Image registry
• Container Runtime Interface implementation
• Container Network Interface implementation
• FlexVolume implementations ("CVI" in the diagram)

还可能依赖：

• NIY： 云提供商节点插件，用于提供节点标识、拓扑等。
• NIY：第三方秘密管理系统（如 Vault）
• NIY：凭证生成和轮换控制器

And may depend on:

• NIY: Cloud-provider node plug-in, to provide node identity, topology, etc.
• NIY: Third-party secret management system (e.g., Vault)
• NIY: Credential generation and rotation controller

可接受的层级违规：

• 显式服务链接
• API server 的 Kubernetes 服务

Accepted layering violations:

• Explicit service links
• The kubernetes service for the API server

The Application Layer: Deployment and Routing#

应用程序管理和组合层，提供自我修复、扩展、应用程序生命周期管理、服务发现、负载平衡和路由，也称为编排和服务结构。
Kubernetes 的任何发行版都需要这些 API 和功能。Kubernetes 应该为这些 API 提供默认实现，但允许替换这些功能的实现（前提是可通过一致性测试）。如果没有这些，大多数容器化应用程序将无法运行，并且很少（如果有的话）发布的示例会起作用。绝大多数容器化应用程序将使用其中一个或多个。
Kubernetes 的 API 提供了类似 IaaS 的以容器为中心的原语以及生命周期控制器，以支持所有主要工作负载类别的协调（自我修复、扩展、更新、终止）。这些应用程序管理、组合、发现和路由 API 和功能包括：
The application management and composition layer, providing self-healing, scaling, application lifecycle management, service discovery, load balancing, and routing -- also known as orchestration and the service fabric.
These APIs and functions are REQUIRED for any distribution of Kubernetes. Kubernetes should provide default implementations for these APIs, but replacements of the implementations of any or all of these functions are permitted, provided the conformance tests pass. Without these, most containerized applications will not run, and few, if any, published examples will work. The vast majority of containerized applications will use one or more of these.
Kubernetes’s API provides IaaS-like container-centric primitives and also lifecycle controllers to support orchestration (self-healing, scaling, updates, termination) of all major categories of workloads. These application management, composition, discovery, and routing APIs and functions include:

• 默认调度器，实现 Pod API 中的调度策略：资源请求，节点选择器，节点和 pod 亲和性 / 反亲和性，污点和容忍度。调度器作为独立的进程运行，也可以在集群之外
• NIY：一个重调度器，用于主动和被动地删除已调度的 pod，以便它们可以被替换并重新调度到其他节点。
• 持续运行的应用程序：这些应用程序类型都应该支持通过声明性更新、级联删除和孤立 / 采用来进行卷展（和回滚）。除了 DaemonSet 之外，所有的都应该支持水平缩放。
  • Deployment API，它协调无状态应用程序的更新，包括子资源（状态、扩展、回滚）
    • ReplicaSet API，适用于简单的可替换 / 无状态应用程序，尤其是特定版本的 Deployment pod 模板，包括子资源（状态、规模）
  • DaemonSet API，用于群集服务，包括子资源（状态）
  • StatefulSet API，用于有状态应用程序，包括子资源（状态、规模）
  • PodTemplate API，由 DaemonSet 和 StatefulSet 用于记录更改历史
• 批量终止程序：这些应用程序应包括对终止作业的自动剔除（NIY）的支持。
  • Job API（GC 讨论）
  • CronJob API
• 发现、LB、路由
  • Service API，包括集群 IP 的分配、服务分配图上的修复、通过 kube 代理或等效代理的负载平衡以及服务的自动端点生成、维护和删除。NIY，但如果支持，则一致性测试必须通过。如果 / 当添加它们时，只有当分发支持 LoadBalancer 服务时，才应支持 LoadBalancer 和 LoadBalacerClaim API。
  • Ingress API，包括内部 L7
  • Service DNS，应该是官方 Kubernetes 模式的 DNS
• A default scheduler, which implements the scheduling policies in the Pod API: resource requests, nodeSelector, node and pod affinity/anti-affinity, taints and tolerations. The scheduler runs as a separate process, on or outside the cluster.
• NIY: A rescheduler, to reactively and proactively delete scheduled pods so that they can be replaced and rescheduled to other nodes.
• Continuously running applications: These application types should all support rollouts (and rollbacks) via declarative updates, cascading deletion, and orphaning/adoption. Other than DaemonSet, all should support horizontal scaling.
  • The Deployment API, which orchestrates updates of stateless applications, including subresources (status, scale, rollback)
    • The ReplicaSet API, for simple fungible/stateless applications, especially specific versions of Deployment pod templates, including subresources (status, scale)
  • The DaemonSet API, for cluster services, including subresources (status)
  • The StatefulSet API, for stateful applications, including subresources (status, scale)
  • The PodTemplate API, used by DaemonSet and StatefulSet to record change history
• Terminating batch applications: These should include support for automatic culling of terminated jobs (NIY).
  • The Job API (GC discussion)
  • The CronJob API
• Discovery, load balancing, and routing
  • The Service API, including allocation of cluster IPs, repair on service allocation maps, load balancing via kube-proxy or equivalent, and automatic Endpoints generation, maintenance, and deletion for services. NIY: LoadBalancer service support is
    OPTIONAL, but conformance tests must pass if it is supported. If/when they are added, support for LoadBalancer and LoadBalancerClaim APIs should be present if and only if the distribution supports LoadBalancer services.
  • The Ingress API, including internal L7 (NIY)
  • Service DNS. DNS, using the official Kubernetes schema, is required.

应用层还可能依赖于：

• 身份 provider（集群验证或应用验证）
• NIY：云供应商控制器实现
• Ingress 控制器
• 调度器 / 重调度器
• 替换 DNS 服务
• 替换 kube-proxy
• 辅助工作负载控制器，特别是用于扩展部署策略

The application layer may depend on:

• Identity provider (to-cluster identities and/or to-application identities)
• NIY: Cloud-provider controller implementation
• Ingress controller(s)
• Replacement and/or additional schedulers and/or reschedulers
• Replacement DNS service
• Replacement for kube-proxy
• Replacement and/or auxiliary workload controllers, especially for extended rollout strategies

The Governance Layer: Automation and Policy Enforcement#

政策执行和更高级别的自动化。
这些 API 和功能对于运行应用程序来说应该是可选的，并且应该可以通过其他解决方案实现。
每个受支持的 API/功能应适用于大部分企业操作、安全和/或治理场景。
需要能够配置和发现集群的默认策略（可能类似于 Openshift 的新项目模板机制，但支持多个策略模板，例如系统名称空间和用户名称空间），以至少支持以下用例：

• Is this a
  • 单租户 / 单用户集群
  • 多个经验证的租户集群
  • 生产 vs 开发集群
  • 高租户 playground 集群
  • 将计算 / 应用程序服务转售给他人的细分集群
• Do I care about limiting：
  • 资源使用
  • 内部节点细分
  • 终端用户
  • 管理
  • DoS

Policy enforcement and higher-level automation.
These APIs and functions should be optional for running applications, and should be achievable via other solutions.
Each supported API/function should be applicable to a large fraction of enterprise operations, security, and/or governance scenarios.
It needs to be possible to configure and discover default policies for the cluster (perhaps similar to Openshift’s new project template mechanism, but supporting multiple policy templates, such as for system namespaces vs. user ones), to support at least the following use cases:

• Is this a: (source: multi-tenancy working doc)
  • Single tenant / single user cluster
  • Multiple trusted tenant cluster
  • Production vs. dev cluster
  • Highly tenanted playground cluster
  • Segmented cluster for reselling compute / app services to others
• Do I care about limiting:
  • Resource usage
  • Internal segmentation of nodes
  • End users
  • Admins
  • DoS

自动化 API 和功能：

• 指标 API (需要 水平 / 垂直扩容，调度 TBD)
• HorizontalPodAutoscaler API
• 集群自动扩容和节点供给
• PodDisruptionBudget API
• 至少一个卷类型的动态卷供给
  • StorageClass API，至少实现默认卷类型
• 动态 LB 供给
• NIY： PodPreset API
• service broker/catalog API
• Template and TemplateInstance API

Automation APIs and functions:

• The Metrics APIs (needed for H/V autoscaling, scheduling TBD)
• The HorizontalPodAutoscaler API
• NIY: The vertical pod autoscaling API(s)
• Cluster autoscaling and/or node provisioning
• The PodDisruptionBudget API
• Dynamic volume provisioning, for at least one volume source type
  • The StorageClass API, implemented at least for the default volume type
• Dynamic load-balancer provisioning
• NIY: The PodPreset API
• NIY: The service broker/catalog APIs
• NIY: The Template and TemplateInstance APIs

策略 API 和功能：

• 鉴权：ABAC 和 RBAC 授权策略方案。
  • 如果使用 RBAC，则使用许多 API 进行配置：角色、角色绑定、集群角色、集群角色绑定
• The LimitRange API
• The ResourceQuota API
• The PodSecurityPolicy API
• The ImageReview API
• The NetworkPolicy API

Policy APIs and functions:

• Authorization: The ABAC and RBAC authorization policy schemes.
  • RBAC, if used, is configured using a number of APIs: Role, RoleBinding, ClusterRole, ClusterRoleBinding
• The LimitRange API
• The ResourceQuota API
• The PodSecurityPolicy API
• The ImageReview API
• The NetworkPolicy API

管理层可能依赖于：

• 网络策略执行机制
• 更换和 / 或额外的水平和垂直 pod 自动扩缩容
• 集群自动缩放器和 / 或节点供应器
• 动态卷调配器
• 动态负载平衡器配置程序
• 指标监控 pipeline，或其替代品
• 服务 broker

The management layer may depend on:

• Network policy enforcement mechanism
• Replacement and/or additional horizontal and vertical pod autoscalers
• Cluster autoscaler and/or node provisioner
• Dynamic volume provisioners
• Dynamic load-balancer provisioners
• Metrics monitoring pipeline, or a replacement for it
• Service brokers

The Interface Layer: Libraries and Tools#

这些机制建议用于发行版，也可供用户独立下载和安装。它们包括 Kubernetes 官方项目开发的常用库、工具、系统和 UI，尽管也可以使用其他工具来完成相同的任务。它们可用于已发布的示例。
一些 Kubernetes 拥有的 GitHub org 下开发的常用库、工具、系统和 UI。

• kubectl —— 我们认为 kubectl 是许多客户端工具之一，而不是特权工具。我们的目标是通过将常用的非平凡功能移动到 API 中，使 kubectl 更轻巧。这对于促进 Kubernetes 版本的正确操作、促进 API 扩展性、保持以 API 为中心的 Kubernets 生态系统模型以及简化其他客户端（尤其是非 Go 客户端）来说是必要的。
• 客户端库（例如，client-go，client-python）
• 集群联合（API Server，controller，kubefed）
• Dashboard
• Helm

这些组件可能依赖于：

• Kubectl 拓展 (可通过 help 发现)
• Helm 拓展 (可通过 help 发现)

These mechanisms are suggested for distributions, and also are available for download and installation independently by users. They include commonly used libraries, tools, systems, and UIs developed by official Kubernetes projects, though other tools may be used to accomplish the same tasks. They may be used by published examples.
Commonly used libraries, tools, systems, and UIs developed under some Kubernetes-owned GitHub org.

• Kubectl -- We see kubectl as one of many client tools, rather than as a privileged one. Our aim is to make kubectl thinner, by moving commonly used non-trivial functionality into the API. This is necessary in order to facilitate correct operation across Kubernetes releases, to facilitate API extensibility, to preserve the API-centric Kubernetes ecosystem model, and to simplify other clients, especially non-Go clients.
• Client libraries (e.g., client-go, client-python)
• Cluster federation (API server, controllers, kubefed)
• Dashboard
• Helm

These components may depend on:

• Kubectl extensions (discoverable via help)
• Helm extensions (discoverable via help)

The Ecosystem#

这些东西实际上不是 Kubernetes 的一部分。
这里有很多领域我们已经定义在 Kubernetes 的清晰切分的边界中
These things are not really "part of" Kubernetes at all.
There are a number of areas where we have already defined clear-cut boundaries for Kubernetes.
虽然 Kubernetes 必须提供部署和管理容器化应用程序所需的功能，但作为一般规则，我们在补充 Kubernete 通用编排功能的领域保留用户选择权，特别是那些拥有自己的竞争优势的领域，这些领域由满足不同需求和偏好的众多解决方案组成。Kubernetes 可以为此类解决方案提供插件 API，或者可以公开可以由多个后端实现的通用 API，或者公开此类解决方案可以针对的 API。有时，功能可以在没有显式接口的情况下与 Kubernetes 轻松组合。
While Kubernetes must offer functionality commonly needed to deploy and manage containerized applications, as a general rule, we preserve user choice in areas complementing Kubernetes’s general-purpose orchestration functionality, especially areas that have their own competitive landscapes comprised of numerous solutions satisfying diverse needs and preferences. Kubernetes may provide plug-in APIs for such solutions, or may expose general-purpose APIs that could be implemented by multiple backends, or expose APIs that such solutions can target. Sometimes, the functionality can compose cleanly with Kubernetes without explicit interfaces.
此外，要被视为 Kubernetes 的一部分，组件需要遵循 Kubernets 的设计惯例。例如，其主要接口是特定于域的语言（例如，Puppet、Open Policy Agent）的系统与以 Kubernetes API 为中心的方法不兼容，与 Kubernets 一起使用完全可以，但不会被认为是 Kubernete 的一部分。类似地，设计用于支持多个平台的解决方案可能不会遵循 Kubernetes API 约定，因此不会被视为 Kubernets 的一部分。
Additionally, to be considered part of Kubernetes, a component would need to follow Kubernetes design conventions. For instance, systems whose primary interfaces are domain-specific languages (e.g., Puppet, Open Policy Agent) aren’t compatible with the Kubernetes API-centric approach, and are perfectly fine to use with Kubernetes, but wouldn’t be considered to be part of Kubernetes. Similarly, solutions designed to support multiple platforms likely wouldn’t follow Kubernetes API conventions, and therefore wouldn’t be considered to be part of Kubernetes.

• 内部容器镜像：Kubernetes 不关心容器内部是什么样的 —— 如果它位于镜像内部，那么它就位于 Kubernets 外部。例如，这包括特定于语言的应用程序框架。

• Kubernetes 之上

  • CI/CD：它不部署源代码，也不构建应用程序。持续集成（CI）和持续部署工作流是不同用户和项目有自己需求和偏好的领域，因此我们旨在促进 Kubernetes 上的 CI/CD 工作流分层，但不规定它们应该如何工作。
  • 应用程序中间件：Kubernetes 不提供像 MQ、DB 这样的中间件。可能的话，提供一般机制，例如服务 broker 集成，方便供给发现并访问这样的组件，这样的组件只是运行在 Kubernetes 上而已。
  • 日志和监控：kubernetes 不提供日志聚合、综合应用监控，也不提供遥测分析和警报系统，尽管这些机制对生产集群至关重要。
  • 数据处理平台：Spark 和 Hadoop 已经是众所周知的例子了，但还有很多这样的系统
  • 应用特定操作：Kubernetes 支持常见类别应用程序的工作负载管理，但不支持特定应用程序的工作负载管理。
  • PasS：Kubernetes 为众多专注、固执己见的 PaaSes（包括 DIY）提供了基础平台。
  • FasS：与 PasS 类似，但是 FaaS 还侵入了容器和特定语言的应用程序框架。
  • 工作流编排：“工作流” 是一个非常广泛、多样的领域，其解决方案通常针对特定用例（数据流图、数据驱动处理、部署管道、事件驱动自动化、业务流程执行、iPaaS）以及特定输入和事件源而定制，并且通常需要任意代码来评估条件、操作和 / 或故障处理。
  • 配置 DSL：DSL 不利于对更高级别的 API 和工具进行分层，它们通常具有有限的可表达性、可测试性、熟悉性和文档化，它们促进了复杂的配置生成，它们倾向于损害互操作性和可组合性，它们使依赖性管理复杂化，并且经常使用颠覆性的抽象和封装。
  • Kompose：Kompose 是一个项目支持的适配器工具，它有助于从 Docker Compose 迁移到 Kubernetes，并支持简单的用例，但不遵循 Kubernete 约定，基于手动维护的 DSL。
  • ChatOps：也是适配工具，用于多种聊天服务。

• Inside container images: Kubernetes is not opinionated about the contents of container images -- if it lives inside the container image, it lives outside Kubernetes. This includes, for example, language-specific application frameworks.

• On top of Kubernetes

  • Continuous integration and deployment: Kubernetes is unopinionated in the source-to-image space. It does not deploy source code and does not build your application. Continuous Integration (CI) and continuous deployment workflows are areas where different users and projects have their own requirements and preferences, so we aim to facilitate layering CI/CD workflows on Kubernetes but don't dictate how they should work.
  • Application middleware: Kubernetes does not provide application middleware, such as message queues and SQL databases, as built-in infrastructure. It may, however, provide general-purpose mechanisms, such as service-broker integration,
    to make it easier to provision, discover, and access such components. Ideally, such components would just run on Kubernetes.
  • Logging and monitoring: Kubernetes does not provide logging aggregation, comprehensive application monitoring, nor telemetry analysis and alerting systems, though such mechanisms are essential to production clusters.
  • Data-processing platforms: Spark and Hadoop are well known examples, but there are many such systems.
  • Application-specific operators: Kubernetes supports workload management for common categories of applications, but not for specific applications.
  • Platform as a Service: Kubernetes provides a foundation for a multitude of focused, opinionated PaaSes, including DIY ones.
  • Functions as a Service: Similar to PaaS, but FaaS additionally encroaches into containers and language-specific application frameworks.
  • Workflow orchestration: "Workflow" is a very broad, diverse area, with solutions typically tailored to specific use cases (data-flow graphs, data-driven processing, deployment pipelines, event-driven automation, business-process execution, iPaaS) and specific input and event sources, and often requires arbitrary code to evaluate conditions, actions, and/or failure handling.
  • Configuration DSLs: Domain-specific languages do not facilitate layering higher-level APIs and tools, they usually have limited expressibility, testability, familiarity, and documentation, they promote complex configuration generation, they tend to compromise interoperability and composability, they complicate dependency management, and uses often subvert abstraction and encapsulation.
  • Kompose: Kompose is a project-supported adaptor tool that facilitates migration to Kubernetes from Docker Compose and enables simple use cases, but doesn’t follow Kubernetes conventions and is based on a manually maintained DSL.
  • ChatOps: Also adaptor tools, for the multitude of chat services.

• Kubernetes 之下

  • 容器运行时：Kubernetes 不提供自己的容器运行时，但提供了一个接口，用于插入您选择的容器运行时。
  • 镜像仓库，Kubernetes 拉取镜像到节点上
  • 集群状态存储：Etcd
  • 网络：与容器运行时一样，我们支持一个促进可插拔性的接口（CNI）。
  • 文件存储：本地文件系统和网络存储。
  • 节点管理：Kubernetes 既不提供也不采用任何全面的机器配置、维护、管理或自我修复系统，这些系统通常在不同的公共 / 私有云中以不同的方式处理，适用于不同的操作系统，适用于可变与不变的基础设施，适用于已经在 Kubernete 集群之外使用工具的商店等。
  • 云供应商：IaaS 供应和管理。
  • 集群创建和管理：社区已经开发了许多工具，如 minikube、kubeadm、bootkube、kube-aws、kops、kargo、kubernetes 等。从工具的多样性可以看出，集群部署和管理（如升级）没有一个通用的解决方案。有一系列可能的解决方案，每种方案都有不同的权衡。也就是说，通用的构建块（例如，安全的 Kubelet 注册）和方法（特别是自托管）将减少此类工具所需的定制编排量。

• Underlying Kubernetes

  • Container runtime: Kubernetes does not provide its own container runtime, but provides an interface for plugging in the container runtime of your choice.
  • Image registry: Kubernetes pulls container images to the nodes.
  • Cluster state store: Etcd
  • Network: As with the container runtime, we support an interface (CNI) that facilitates pluggability.
  • File storage: Local filesystems and network-attached storage.
  • Node management: Kubernetes neither provides nor adopts any comprehensive machine configuration, maintenance, management, or self-healing systems, which typically are handled differently in different public/private clouds, for different operating systems, for mutable vs. immutable infrastructure, for shops already using tools outside of their Kubernetes clusters, etc.
  • Cloud provider: IaaS provisioning and management.
  • Cluster creation and management: The community has developed numerous tools, such as minikube, kubeadm, bootkube, kube-aws, kops, kargo, kubernetes-anywhere, and so on. As can be seen from the diversity of tools, there is no one-size-fits-all solution for cluster deployment and management (e.g., upgrades). There's a spectrum of possible solutions, each with different tradeoffs. That said, common building blocks (e.g., secure Kubelet registration) and approaches (in particular, self-hosting) would reduce the amount of custom orchestration needed in such tools.

我们希望看到生态系统构建和 / 或集成解决方案来满足这些需求。
最终，大多数 Kubernetes 开发项目都应该属于生态系统。
We would like to see the ecosystem build and/or integrate solutions to fill these needs.
Eventually, most Kubernetes development should fall in the ecosystem.

Managing the matrix#

选项、可配置默认值、扩展、插件、加载项、特定于提供程序的功能、版本偏差、功能发现和依赖项管理。
Options, Configurable defaults, Extensions, Plug-ins, Add-ons, Provider-specific functionality, Version skew, Feature discovery, and Dependency management.
Kubernetes 不仅仅是一个开源工具包，它通常被用作一个可运行的、易于运行的或可随时运行的集群或服务。我们希望大多数用户和用例能够使用库存上游版本。这意味着 Kubernetes 需要足够的可扩展性，而不需要重建来处理此类用例。
Kubernetes is not just an open-source toolkit, but is typically consumed as a running, easy-to-run, or ready-to-run cluster or service. We would like most users and use cases to be able to use stock upstream releases. This means Kubernetes needs sufficient extensibility without rebuilding to handle such use cases.
虽然可扩展性方面的差距是代码 fork 的主要驱动因素，而上游集群生命周期管理解决方案方面的差距目前是 Kubernetes 发行版激增的主要驱动力，但可选特性（例如，alpha API、特定于提供商的 API）、可配置性、可插拔性和可扩展性的存在使得这一概念不可避免。
While gaps in extensibility are the primary drivers of code forks and gaps in upstream cluster lifecycle management solutions are currently the primary drivers of the proliferation of Kubernetes distributions, the existence of optional features (e.g., alpha APIs, provider-specific APIs), configurability, pluggability, and extensibility make the concept inevitable.
然而，为了让用户能够部署和管理他们的应用程序，让开发人员能够在任意 Kubernetes 集群上 / 为其构建 Kubernete 扩展，他们必须能够对 Kubernets 的集群或分布提供什么进行假设。当功能超出这些基本假设时，需要有一种方法来发现哪些功能可用，并表达使用的功能需求（依赖关系）。
集群组件（包括附加组件）应通过组件注册 API 注册，并通过 / componentstatus 发现。
启用的内置 API、聚合 API 和注册的第三方资源应该可以通过发现和 OpenAPI（swagger.json）端点发现。如上所述，云提供商对 LoadBalancer 类型服务的支持应取决于 LoadBaancer API 是否存在。
扩展及其选项应通过 FooClass 资源注册，类似于 StorageClass，但要使用参数描述、类型（例如，整数与字符串）、用于验证的约束（例如，range 或 regexp）和默认值，并引用扩展 API 中的 fooClassName。这些 API 还应配置 / 公开相关功能，如动态卷调配（由非空的 storageclass.provisioner 字段指示），以及标识负责的控制器。我们需要至少为调度器类、入口控制器类、灵活卷类和计算资源类（例如 GPU、其他加速器）添加这样的 API。
假设我们将现有的网络连接卷源转换为灵活卷，这种方法将覆盖卷源。未来，API 应该只提供通用抽象，即使与 LoadBalancer 服务一样，抽象不是在所有环境中实现的（即，API 不需要满足最低的公约数）。
However, to make it possible for users to deploy and manage their applications and for developers to build Kubernetes extensions on/for arbitrary Kubernetes clusters, they must be able to make assumptions about what a cluster or distribution of Kubernetes provides. Where functionality falls out of these base assumptions, there needs to be a way to discover what functionality is available and to express functionality requirements (dependencies) for usage.
Cluster components, including add-ons, should be registered via the component registration API and discovered via /componentstatuses.
Enabled built-in APIs, aggregated APIs, and registered third-party resources should be discoverable via the discovery and OpenAPI (swagger.json) endpoints. As mentioned above, cloud-provider support for LoadBalancer-type services should be determined by whether the LoadBalancer API is present.
Extensions and their options should be registered via FooClass resources, similar to StorageClass, but with parameter descriptions, types (e.g., integer vs string), constraints (e.g., range or regexp) for validation, and default values, with a reference to fooClassName from the extended API. These APIs should also configure/expose the presence of related features, such as dynamic volume provisioning (indicated by a non-empty storageclass.provisioner field), as well as identifying the responsible controller. We need to add such APIs for at least scheduler classes, ingress controller classes, flex volume classes, and compute resource classes (e.g., GPUs, other accelerators).
Assuming we transitioned existing network-attached volume sources to flex volumes, this approach would cover volume sources. In the future, the API should provide only general-purpose abstractions, even if, as with LoadBalancer services, the abstractions are not implemented in all environments (i.e., the API does not need to cater to the lowest common denominator).

NIY：我们还需要建立注册和发现以下内容的机制：
● 准入控制插件和挂钩（包括内置 API）
● 身份验证插件
● 授权插件和挂钩
● 初始化器和终结器
● 计划程序扩展
● 节点标签和群集拓扑（拓扑类？）
NIY: We also need to develop mechanisms for registering and discovering the following:

• Admission-control plugins and hooks (including for built-in APIs)
• Authentication plugins
• Authorization plugins and hooks
• Initializers and finalizers
• Scheduler extensions
• Node labels and cluster topology (topology classes?)

NIY：单个 API 和细粒度特性的激活 / 停用可通过以下机制解决：

• 正在将所有组件的配置从命令行标志转换为版本化配置。
• 我们打算将大部分配置数据存储在 ConfigMaps 中，以便于动态重新配置、逐步展开和内观。
• 所有 / 多个组件的通用配置应分解为其自己的配置对象。这应包括特征门机制。
• 应该为语义上有意义的设置添加 API，例如删除无响应节点上的 pod 之前的默认等待时间。

NIY：对于依赖于多个组件（包括 HA 集群中同一组件的副本）升级的功能，版本偏差操作的问题应该通过以下方式解决：
1. 为所有新的此类特征创建标志门，
2. 总是在功能出现的第一个次要版本中默认禁用这些功能，
3. 提供配置补丁以启用功能，以及
4. 在下一个次要版本中默认启用它们。
NIY: Activation/deactivation of both individual APIs and finer-grain features could be addressed by the following mechanisms:

• The configuration for all components is being converted from command-line flags to versioned configuration.
• We intend to store most of that configuration data in ConfigMaps, to facilitate dynamic reconfiguration, progressive rollouts, and introspectability.
• Configuration common to all/multiple components should be factored out into its own configuration object(s). This should include the feature-gate mechanism.
• An API should be added for semantically meaningful settings, such as the default length of time to wait before deleting pods on unresponsive nodes.

NIY: The problem of version-skewed operation, for features dependent on upgrades of multiple components (including replicas of the same component in HA clusters), should be addressed by:

1. Creating flag gates for all new such features,
2. Always disabling the features by default in the first minor release in which they appear,
3. Providing configuration patches to enable the features, and
4. Enabling them by default in the next minor release.

NIY：我们还需要一个机制来警告过期节点，和/或潜在地阻止主升级（而不是补丁版本），直到/除非节点已经升级。
NIY：字段级版本控制将有助于解决新的和 / 或 alpha API 字段的批量激活、防止写得不好的过时客户端对新字段的攻击、以及在没有全面 API 定义的情况下发展非 alpha API 的问题。
Kubernetes API 服务器会自动忽略不支持的资源字段和查询参数，但不会忽略未知/未注册的 API（请注意，应禁用未实现/未激活的 API）。这有助于在多个版本的集群中重用配置，但通常会导致意外。Kubectl 支持从服务器使用 Swagger/OpenAPI 规范进行可选验证。这种可选验证应由服务器（NIY）提供。此外，为了方便用户，共享资源清单应该指定 Kubernetes 所需的最低版本，这可能会被 kubectl 和其他客户端验证。
此外，不可满足的 Pod 调度约束和 PersistentVolumeClaim 标准未得到满足，这可以作为自动供应器的需求信号，但也使系统更容易出错。应该可以使用 FooClass 风格的 API 配置拒绝不可满足的请求，如上所述（NIY）。
服务目录机制（NIY）应该能够断言应用程序级服务的存在，例如 S3 兼容的集群存储。
NIY: We additionally need a mechanism to warn about out of date nodes, and/or potentially prevent master upgrades (other than to patch releases) until/unless the nodes have been upgraded.
NIY: Field-level versioning would facilitate solutions to bulk activation of new and/or alpha API fields, prevention of clobbering of new fields by poorly written out-of-date clients, and evolution of non-alpha APIs without a proliferation of full-fledged API definitions.
The Kubernetes API server silently ignores unsupported resource fields and query parameters, but not unknown/unregistered APIs (note that unimplemented/inactive APIs should be disabled). This can facilitate the reuse of configuration across clusters of multiple releases, but more often leads to surprises. Kubectl supports optional validation using the Swagger/OpenAPI specification from the server. Such optional validation should be provided by the server (NIY). Additionally, shared resource manifests should specify the minimum required Kubernetes release, for user convenience, which could potentially be verified by kubectl and other clients.
Additionally, unsatisfiable Pod scheduling constraints and PersistentVolumeClaim criteria silently go unmet, which can useful as demand signals to automatic provisioners, but also makes the system more error prone. It should be possible to configure rejection of unsatisfiable requests, using FooClass-style APIs, as described above (NIY).
The Service Catalog mechanism (NIY) should make it possible to assert the existence of application-level services, such as S3-compatible cluster storage.

Layering of the system as it relates to security#

为了正确地保护 Kubernetes 集群并实现安全扩展，系统组件需要定义并同意几个基本概念。从安全的角度来看，最好将 Kubernetes 看作一系列环，每一层都授予连续层的功能。
1 用于内核 API 的一个或多个数据存储系统（etcd）
2. 内核 API
3. 高度可信资源的 API（系统策略）
4. 在集群范围或更小范围内的委托信任 API 和控制器（用户授权访问 API / 控制器以代表他们执行操作）
5. 在不同范围内运行的不受信任 / 作用域的 API 和控制器以及用户工作负载
In order to properly secure a Kubernetes cluster and enable safe extension, a few fundamental concepts need to be defined and agreed on by the components of the system. It’s best to think of Kubernetes as a series of rings from a security perspective, with each layer granting the successive layer capabilities to act.

1. One or more data storage systems (etcd) for the nuclear APIs
2. The nuclear APIs
3. APIs for highly trusted resources (system policies)
4. Delegated trust APIs and controllers (users grant access to the API / controller to perform actions on their behalf) either at the cluster scope or smaller scopes
5. Untrusted / scoped APIs and controllers and user workloads that run at various scopes

当一个较低的层依赖于一个较高的层时，它会破坏安全模型，使防御系统变得更加复杂 —— 管理员可以选择这样做以获得操作的简单性，但这必须是有意识的选择。一个简单的例子是 etcd：任何可以向 etcd 写入数据的组件现在都是整个集群的根，任何可以破坏高度可信资源的参与者几乎都可以升级。对于每一层流程（etcd->apiservers + controller->nuclear security extensions->delegated extension->user workload），将上面的层划分为不同的机器集是很有用的，即使有些在实践中可能会崩溃。
如果上面描述的层定义了同心圆，那么也应该存在重叠或独立的圆 - 例如，管理员可以选择一个替代的秘密存储解决方案，集群工作负载可以访问该解决方案，但平台不能隐式访问该解决方案。这些圆的交点往往是运行工作负载的机器，节点的权限不得超过正常运行所需的权限。
最后，通过扩展在任何层添加新功能都应遵循传达该操作影响的最佳实践。
When a lower layer depends on a higher layer, it collapses the security model and makes defending the system more complicated - an administrator may choose to do so to gain operational simplicity, but that must be a conscious choice. A simple example is etcd: any component that can write data to etcd is now root on the entire cluster, and any actor that can corrupt a highly trusted resource can almost certainly escalate. It is useful to divide the layers above into separate sets of machines for each layer of processes (etcd -> apiservers + controllers -> nuclear security extensions -> delegated extensions -> user workloads), even if some may be collapsed in practice.
If the layers described above define concentric circles, then it should also be possible for overlapping or independent circles to exist - for instance, administrators may choose an alternative secret storage solution that cluster workloads have access to yet the platform does not implicitly have access to. The point of intersection for these circles tends to be the machines that run the workloads, and nodes must have no more privileges than those required for proper function.
Finally, adding a new capability via extension at any layer should follow best practices for communicating the impact of that action.

当通过扩展将功能添加到系统中时，它有什么用途？

• 使系统更安全
• 启用新的 “生产质量” API 供集群中的每个人使用
• 跨集群的子集自动执行通用任务
• 运行托管工作负载，为消费者提供 api（spark、数据库等）
• 这些可分为三大类：
  • 集群所需的（因此必须靠近核心运行，并在出现故障时导致操作权衡）
  • 向所有群集用户公开（必须正确租用）
  • 暴露于集群用户的子集（运行方式更像传统的 “应用程序” 工作负载）

如果管理员在扩展过程中很容易被欺骗安装新的集群级安全规则，那么分层就会受到破坏，系统也会受到攻击。
When a capability is added to the system via extension, what purpose does it have?

• Make the system more secure
• Enable a new "production quality" API for consumption by everyone in the cluster
• Automate a common task across a subset of the cluster
• Run a hosted workload that offers apis to consumers (spark, a database, etcd)
• These fall into three major groups:
  • Required for the cluster (and hence must run close to the core, and cause operational tradeoffs in the presence of failure)
  • Exposed to all cluster users (must be properly tenanted)
  • Exposed to a subset of cluster users (runs more like traditional "app" workloads)

If an administrator can easily be tricked into installing a new cluster level security rule during extension, then the layering is compromised and the system is vulnerable.

Next Steps#

除了完成上面描述和 / 或暗示的技术机制之外，我们还需要将本文件中的原则应用于一组更为集中的文件，以回答具体的实际问题。以下是一些建议的文件及其回答的问题：
In addition to completing the technical mechanisms described and/or implied above, we need to apply the principles in this document to a set of more focused documents that answer specific practical questions. Here are some suggested documents and the questions they answer:

• 扩展 API 开发人员的 Kubernetes API 惯例
  • 受众：计划构建类似 Kubernetes 的 API 扩展（TPR 或聚合 API 等）的人
  • 回答问题：
    • 我应该遵循什么惯例？（元数据、状态等）
    • 遵循这些惯例，我可以得到什么集成？
    • 我可以省略什么和做什么
  • 回答此问题的文档：
    • 针对 CLI 和 UI 开发人员的 Kubernetes API 惯例
  • 受众：使用 kubectl、仪表板或其他 CLI 或 UI 的人。
  • 回答问题：
    • 如何向用户显示哪些对象是从属对象并且通常应该隐藏
• Kubernetes 分发 / 服务的必需和可选行为
  • 另请参见认证问题
  • 我刚刚实现了 Kubernetes API 的托管版本。
    • 我必须公开哪些 API 组、版本和种类？
    • 我可以提供我自己的日志记录、身份验证、审计集成等吗？
    • 如果它只有 pod 但没有服务，我可以叫它 Kubernetes 吗？
  • 我正在打包一个精心策划的 Kubernetes 发行版并出售。
    • 为了称之为 Kubernetes，我必须公开哪些 API 组、版本和种类。
• 应用程序开发人员的假设 / 约定
  • 我想在 prem 和几个云上编写可移植配置。
    • 我应该假设什么样的 API 类型和行为始终存在并完全抽象。我应该在什么时候尝试检测一个特性（通过询问用户他们有什么云提供商，或者在将来有一个特性发现 API）。
• Kubernetes 安全模型
  • 受众：主持人、分销商、定制集群构建者
• Kubernetes API Conventions For Extension API Developers
  • Audience: someone planning to build a Kubernetes-like API extension (TPR or Aggregated API, etc…)
  • Answers Questions:
    • What conventions should I follow? (metadata, status, etc)
    • What integrations do I get from following those conventions?
    • What can I omit and Does
  • Document that answers this question:
• Kubernetes API Conventions For CLI and UI Developers
  • Audience: someone working on kubectl, dashboard, or another CLI or UI.
  • Answers Questions:
    • How can I show a user which objects subordinate and should normally be hidden
• Required and Optional Behaviors for Kubernetes Distributions/Services
  • See also the certification issue
  • I just implemented a Hosted version of the Kubernetes API.
    • Which API groups, versions and Kinds do I have to expose?
    • Can I provide my own logging, auth, auditing integrations and so on?
    • Can I call it Kubernetes if it just has pods but no services?
  • I'm packaging up a curated Kubernetes distro and selling it.
    • Which API groups, versions and Kinds do I have to expose in order to call it Kubernetes.
• Assumptions/conventions for application developers
  • I want to write portable config across on-prem and several clouds.
    • What API types and behaviors should I assume are always present and fully abstracted.  When should I try to detect a feature (by asking the user what cloud provider they have, or in the future a feature discovery API).
• Kubernetes Security Models
  • Audience: hosters, distributors, custom cluster builders