Fricky's Blog

Flink K8s Operator收敛至namespace级的部署方案

2023/08/31 flink-k8s-operator

基于角色访问控制模型（RBAC）

为了能在k8s上对operator和flink作业进行部署，社区定义了两类角色(roles)。 角色flink-operator职责是管理flinkdeployments、为flink作业创建并管理JobManager和其他资源（比如说services）。 角色flink职责是帮助flink作业的JobManager去创建并管理TaskManagers和ConfigMaps。

如图详细描述了两类角色的职责范围。 flink-k8s-operator rbac

角色flink-operator在安装部署时默认是集群级别角色(clusterrole)，所以在一个k8s集群上只需一个operator实例即可无视namespace进行flink作业部署。在一些特定限制更苛刻的环境下，比如说只允许namespace级别的角色时，还可通过使用配置watchNamespaces完成部署。

由于默认配置给的是clusterrole权限，被k8s管理员要求收敛至namespace级别，为满足线上要求而给的整改方案。整体就两步，先通过helm本地部署收敛权限的values.yaml，然后创建flink k8s operator的RBAC权限账户和角色

1. namespace级的values.yaml

# 这个是要监听的namespaces列表，之后增加也要在这填
watchNamespaces: ["fuyiqi"]

image:
  repository: fuyiqi/flink-kubernetes-operator
  pullPolicy: IfNotPresent
  tag: 1.5.0
  # If image digest is set then it takes precedence and the image tag will be ignored
  # digest: ""

imagePullSecrets: []

# Replicas must be 1 unless operator leader election is configured
replicas: 1

# Strategy type must be Recreate unless leader election is configured
strategy:
  type: Recreate

rbac:
  create: false
  # kubernetes.rest-service.exposed.type: NodePort requires
  # list permission for nodes at the cluster scope.
  # Set create to true if you are using NodePort type.
  nodesRule:
    create: false # 默认不创建
  operatorRole:
    create: false
    name: "fuyiqi-flink-operator-role"
  operatorRoleBinding:
    create: false # 默认不创建
    name: "fuyiqi-flink-operator-role-binding"
  jobRole:
    create: false # 默认不创建
    name: "fuyiqi-flink"
  jobRoleBinding:
    create: false # 默认不创建
    name: "fuyiqi-flink-role-binding"

operatorPod:
  priorityClassName: null
  annotations: {}
  labels: {}
  env:
  # - name: ""
  #   value: ""
  # dnsPolicy: ""
  # dnsConfig: {}
  # Node labels for operator pod assignment
  # https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
  nodeSelector: {}
  # Node tolerations for operator pod assignment
  # https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
  tolerations: []
  # Topology spread constrains
  # https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/
  topologySpreadConstraints: []
  resources:
    resources:
      limits:
        cpu: "1000m"
        memory: "2048Mi"
      requests:
        cpu: "250m"
        memory: "1024Mi"
  webhook:
    resources: {}

operatorServiceAccount:
  create: false # 默认不创建
  annotations: {}
  name: "fuyiqi-flink-operator-sa"

jobServiceAccount:
  create: false # 默认不创建
  annotations:
    "helm.sh/resource-policy": keep
  name: "fuyiqi-flink-sa"

operatorVolumeMounts:
  create: false
  data:
    - name: flink-artifacts
      mountPath: /opt/flink/artifacts

operatorVolumes:
  create: false
  data:
    - name: flink-artifacts
      hostPath:
        path: /tmp/flink/artifacts
        type: DirectoryOrCreate
#    - name: flink-artifacts
#      persistentVolumeClaim:
#        claimName: flink-artifacts

podSecurityContext:
  runAsUser: 9999
  runAsGroup: 9999
  # fsGroup: 9999

operatorSecurityContext: {}

webhookSecurityContext: {}

webhook:
  create: false
  # validator:
  #   create: true
  # mutator:
  #   create: true
  keystore:
    useDefaultPassword: false
  # passwordSecretRef:
  #   name: jks-password-secret
  #   key: password-key

defaultConfiguration:
  # If set to true, creates ConfigMaps/VolumeMounts. If set to false, no configuration will be created.
  # All below fields will be ignored if create is set to false.
  create: true
  # If set to true,
  #      (1) loads the built-in default configuration
  #      (2) appends the below flink-conf and logging configuration overrides
  # If set to false, loads just the overrides as in (2).
  # This option has not effect, if create is equal to false.
  append: true
  flink-conf.yaml: |+
    # Flink Config Overrides
    kubernetes.operator.metrics.reporter.slf4j.factory.class: org.apache.flink.metrics.slf4j.Slf4jReporterFactory
    kubernetes.operator.metrics.reporter.slf4j.interval: 5 MINUTE
    kubernetes.operator.reconcile.interval: 15 s
    kubernetes.operator.observer.progress-check.interval: 5 s
    # 强烈建议在生产上开启此配置
    kubernetes.operator.leader-election.enabled: true
    kubernetes.operator.leader-election.lease-name: flink-operator-lease
  log4j-operator.properties: |+
    # Flink Operator Logging Overrides
    # rootLogger.level = DEBUG
    # logger.operator.name= org.apache.flink.kubernetes.operator
    # logger.operator.level = DEBUG
  log4j-console.properties: |+
    # Flink Deployment Logging Overrides
    # rootLogger.level = DEBUG

# (Optional) Exposes metrics port on the container if defined
metrics:
  port:

nameOverride: ""
fullnameOverride: ""

# Set the jvm start up options for webhook and operator
jvmArgs:
  webhook: ""
  operator: ""

# 这个配置我没有用到
#operatorHealth:
#  port: 8085
#  livenessProbe:
#    periodSeconds: 10
#    initialDelaySeconds: 30
#  startupProbe:
#    failureThreshold: 1
#    periodSeconds: 1

命名规则是namespace-roleName/saName-component

---
# 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fuyiqi-flink-operator-sa
  namespace: fuyiqi
  labels:
    app.kubernetes.io/name: flink-kubernetes-operator
    app.kubernetes.io/version: "1.5.0"
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: flink-kubernetes-operator-1.5.0
---
#
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fuyiqi-flink-sa
  namespace: fuyiqi
  labels:
    app.kubernetes.io/name: flink-kubernetes-operator
    app.kubernetes.io/version: "1.5.0"
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: flink-kubernetes-operator-1.5.0
  annotations:
    helm.sh/resource-policy: keep

---
#
---

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: fuyiqi-flink-operator-role
  namespace: fuyiqi
  labels:
    app.kubernetes.io/name: flink-kubernetes-operator
    app.kubernetes.io/version: "1.5.0"
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: flink-kubernetes-operator-1.5.0
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - services
      - events
      - configmaps
      - secrets
    verbs:
      - "*"
  - apiGroups:
      - apps
    resources:
      - deployments
      - deployments/finalizers
      - replicasets
    verbs:
      - "*"
  - apiGroups:
      - extensions
    resources:
      - deployments
      - ingresses
    verbs:
      - "*"
  - apiGroups:
      - flink.apache.org
    resources:
      - flinkdeployments
      - flinkdeployments/status
      - flinksessionjobs
      - flinksessionjobs/status
    verbs:
      - "*"
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - "*"
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - "*"
---
#
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: fuyiqi-flink-role
  namespace: fuyiqi
  labels:
    app.kubernetes.io/name: flink-kubernetes-operator
    app.kubernetes.io/version: "1.5.0"
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: flink-kubernetes-operator-1.5.0
  annotations:
    "helm.sh/resource-policy": keep
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - configmaps
    verbs:
      - '*'
  - apiGroups:
      - apps
    resources:
      - deployments
      - deployments/finalizers
    verbs:
      - '*'
---
#
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: fuyiqi-flink-operator-role-binding
  namespace: fuyiqi
  labels:
    app.kubernetes.io/name: flink-kubernetes-operator
    app.kubernetes.io/version: "1.5.0"
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: flink-kubernetes-operator-1.5.0
roleRef:
  kind: Role
  name: fuyiqi-flink-operator-role
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: fuyiqi-flink-operator-sa
    namespace: fuyiqi

---
#
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: fuyiqi-flink-role-binding
  namespace: fuyiqi
  labels:
    app.kubernetes.io/name: flink-kubernetes-operator
    app.kubernetes.io/version: "1.5.0"
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: flink-kubernetes-operator-1.5.0
  annotations:
    "helm.sh/resource-policy": keep
roleRef:
  kind: Role
  name: fuyiqi-flink-role
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: fuyiqi-flink-sa
    namespace: fuyiqi

Search

    Table of Contents