nginx-ingress 转发TCP和UDP请求

如题，ingress做为k8s体系提供的暴露service endpoint的一种方式，提供了集群外访问集群内服务的方式，同样通过负载均衡器也能实现，ingress默认情况下只处理http和https请求，但是对于要暴露的如mysql、redis这样的TCP协议类型的服务就不适用了，不过这个限制对于有一些ingress controller是可以突破的，这里就说明nginx ingress controller如何通过配置来达到暴露集群内tcp或UDP服务的需求。

首先nginx controller是基于nginx开发的，nginx本身现在已经提供了转发UDP请求、TCP请求的能力，因此nginx ingress controller也是可以的。看下如何设置。

首先官方文档的描述如下：

Ingress does not support TCP or UDP services. For this reason this Ingress controller uses the flags --tcp-services-configmap and --udp-services-configmap to point to an existing config map where the key is the external port to use and the value indicates the service to expose using the format: <namespace/service name>:<service port>:[PROXY]:[PROXY]

也就是说需要通过增加—tcp-services-configmap and —udp-services-configmap 这两个flag来将设置指向一个configmap来实现配置，那如何修改呢？对于裸机实列来说通过修改nginx ingress controller的Deployment对象可以达到目的，代码如下：

    apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  minReadySeconds: 0
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
        - --election-id=ingress-controller-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --tcp-services-configmap=$(POD_NAMESPACE)/ingress-nginx-tcp
        --udp-services-configmap=$(POD_NAMESPACE)/ingress-nginx-tcp
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: LD_PRELOAD
          value: /usr/local/lib/libmimalloc.so
        image: registry.cn-shanghai.aliyuncs.com/macco/ingress-nginx-controller:v1.3.0
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /wait-shutdown
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 8443
          name: webhook
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 90Mi
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
          runAsUser: 101
        volumeMounts:
        - mountPath: /usr/local/certificates/
          name: webhook-cert
          readOnly: true
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
      - name: webhook-cert
        secret:
          secretName: ingress-nginx-admission
-

下面以tcp为例，描述configmap如何写

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-tcp
  namespace: ingress-nginx
data:
  5100: "xunzhao/commonapi-debug-service:5100"
  5008: "xunzhao/fileapi-debug-service:5008"
  5007: "xunzhao/user-api-debug-service:5007"
  5005: "xunzhao/business-api-debug-service:5005"
  5010: "xunzhao/financial-api-debug-service:5010"
  5006: "xunzhao/sync-client-debug-serice:5006"
  

可以看到是通过端口号作为key，namespace/servciename:port 作为value来实现的。

同时对于nginx ingress controller需要在自己的服务暴露相关的端口

apiVersion: v1
kind: Service
metadata:
  annotations:
    lb.kubesphere.io/v1alpha1: openelb
    protocol.openelb.kubesphere.io/v1alpha1: layer2
    eip.openelb.kubesphere.io/v1alpha2: eip-pool
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  #externalTrafficPolicy: Local
  #ports:
  #  - name: http
  #    port: 80
  #    targetPort: 80
  #  - name: https
  #    port: 443
  #    targetPort: 443
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  - name: porxy-5100
    port: 5100
    protocol: TCP 
    targetPort: 5100
  - name: porxy-5008
    port: 5008
    protocol: TCP 
    targetPort: 5008
  - name: porxy-5005
    port: 5005
    protocol: TCP 
    targetPort: 5005
  - name: porxy-5007
    port: 5007
    protocol: TCP 
    targetPort: 5007
  - name: porxy-5006
    port: 5006
    protocol: TCP 
    targetPort: 5006
  - name: porxy-5010
    port: 5010
    protocol: TCP 
    targetPort: 5010
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer
  

上述例子结合openelb 将nginx conroller暴露在负载均衡器上实现二层到7层的通讯并暴露tcp相关接口。需要主要的集群的firewalld需要关闭或者在所有实例上添加相关端口放行。