如何给nginx-ingress进行日志落盘

1. nginx-ingress-controller的日志

nginx-ingress-controller的日志包括三个部分：

• controller日志： 输出到stdout，通过启动参数中的–log_dir可已配置输出到文件，重定向到文件后会自动轮转，但不会自动清理
• accesslog：输出到stdout，通过nginx-configuration中的字段可以配置输出到哪个文件。输出到文件后不会自动轮转或清理
• errorlog：输出到stderr，配置方式与accesslog类似。

2. 给controller日志落盘

• 给nginx-ingress-controller挂一个hostpath： /data/log/nginx/ 映射到容器里的/var/log/nginx/ ，
• 给nginx-ingress-controller配置log-dir和logtostderr参数，将日志重定向到/var/log/nginx/中。

controller的日志需要做定时清理。由于controller的日志是通过klog(k8s.io/klog)输出的，会进行日志滚动，所以我们通过脚本定时清理一定时间之前的日志文件即可。

3. 给nginx日志落盘

修改configmap： nginx-configuration。配置accesslog和errorlog的输出路径，替换默认的stdout和stderr。输出路径我们可以与controller一致，便于查找。

accesslog和errorlog都只有一个日志文件，我们可以使用logrotate进行日志轮转，将输出到宿主机上的日志进行轮转和清理。配置如：

$ cat /etc/logrotate.d/nginx.log  /data/log/nginx/access.log {    su root list    rotate 7    daily    maxsize 50M    copytruncate    missingok    create 0644 www-data root  }

官方提供的模板中，nginx-ingress-controller默认都是以33这个用户登录启动容器的，因此挂载hostpath路径时存在权限问题。我们需要手动在机器上执行chown -R 33:33 /data/log/nginx.

4. 自动化

nginx日志落盘中，第2、3两点均需要人工运维，有什么解决办法吗？

问题的关键是：有什么办法可以在nginx-ingress-controller容器启动之前加一个hook，将宿主机的指定目录执行chown呢？

可以用initContainer。initcontainer必须在containers中的容器运行前运行完毕并成功退出。再说第二点，我们注意到nginx-ingress-controller的基础镜像中就自带了logrotate，那么问题就简单了，我们将写好的logrotate配置文件以configmap的形式挂载到容器中就可以了。

完整的yaml文件如下：

apiVersion: v1  kind: Namespace  metadata:    name: ingress-nginx    labels:      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx    ---  # Source: ingress-nginx/templates/controller-serviceaccount.yaml  apiVersion: v1  kind: ServiceAccount  metadata:    labels:      helm.sh/chart: ingress-nginx-2.0.3      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/version: 0.32.0      app.kubernetes.io/managed-by: Helm      app.kubernetes.io/component: controller    name: ingress-nginx    namespace: ingress-nginx  ---  # Source: ingress-nginx/templates/controller-configmap.yaml  apiVersion: v1  kind: ConfigMap  metadata:    labels:      helm.sh/chart: ingress-nginx-2.0.3      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/version: 0.32.0      app.kubernetes.io/managed-by: Helm      app.kubernetes.io/component: controller    name: ingress-nginx-controller    namespace: ingress-nginx  data:    client_max_body_size: "100m"    proxy_body_size: "100m"    access-log-path: /var/log/nginx/access.log    error-log-path: /var/log/nginx/erroes.log  ---  # 创建一个configmap，配置nginx日志的轮转策略，对应的是nginx日志在容器内的日志文件  apiVersion: v1  data:   nginx.log: |    /var/log/nginx/access.log {      rotate 7      daily      maxsize 200M      minsize 10M      copytruncate      missingok      create 0644 root root    }    /var/log/nginx/error.log {      rotate 7      daily      maxsize 200M      minsize 10M      copytruncate      missingok      create 0644 root root    }  kind: ConfigMap  metadata:    name: nginx-ingress-logrotate    namespace: ingress-nginx  ---  # Source: ingress-nginx/templates/clusterrole.yaml  apiVersion: rbac.authorization.k8s.io/v1  kind: ClusterRole  metadata:    labels:      helm.sh/chart: ingress-nginx-2.0.3      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/version: 0.32.0      app.kubernetes.io/managed-by: Helm    name: ingress-nginx    namespace: ingress-nginx  rules:    - apiGroups:        - ''      resources:        - configmaps        - endpoints        - nodes        - pods        - secrets      verbs:        - list        - watch    - apiGroups:        - ''      resources:        - nodes      verbs:        - get    - apiGroups:        - ''      resources:        - services      verbs:        - get        - list        - update        - watch    - apiGroups:        - extensions        - networking.k8s.io   # k8s 1.14+      resources:        - ingresses      verbs:        - get        - list        - watch    - apiGroups:        - ''      resources:        - events      verbs:        - create        - patch    - apiGroups:        - extensions        - networking.k8s.io   # k8s 1.14+      resources:        - ingresses/status      verbs:        - update    - apiGroups:        - networking.k8s.io   # k8s 1.14+      resources:        - ingressclasses      verbs:        - get        - list        - watch  ---  # Source: ingress-nginx/templates/clusterrolebinding.yaml  apiVersion: rbac.authorization.k8s.io/v1  kind: ClusterRoleBinding  metadata:    labels:      helm.sh/chart: ingress-nginx-2.0.3      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/version: 0.32.0      app.kubernetes.io/managed-by: Helm    name: ingress-nginx    namespace: ingress-nginx  roleRef:    apiGroup: rbac.authorization.k8s.io    kind: ClusterRole    name: ingress-nginx  subjects:    - kind: ServiceAccount      name: ingress-nginx      namespace: ingress-nginx  ---  # Source: ingress-nginx/templates/controller-role.yaml  apiVersion: rbac.authorization.k8s.io/v1  kind: Role  metadata:    labels:      helm.sh/chart: ingress-nginx-2.0.3      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/version: 0.32.0      app.kubernetes.io/managed-by: Helm      app.kubernetes.io/component: controller    name: ingress-nginx    namespace: ingress-nginx  rules:    - apiGroups:        - ''      resources:        - namespaces      verbs:        - get    - apiGroups:        - ''      resources:        - configmaps        - pods        - secrets        - endpoints      verbs:        - get        - list        - watch    - apiGroups:        - ''      resources:        - services      verbs:        - get        - list        - update        - watch    - apiGroups:        - extensions        - networking.k8s.io   # k8s 1.14+      resources:        - ingresses      verbs:        - get        - list        - watch    - apiGroups:        - extensions        - networking.k8s.io   # k8s 1.14+      resources:        - ingresses/status      verbs:        - update    - apiGroups:        - networking.k8s.io   # k8s 1.14+      resources:        - ingressclasses      verbs:        - get        - list        - watch    - apiGroups:        - ''      resources:        - configmaps      resourceNames:        - ingress-controller-leader-nginx      verbs:        - get        - update    - apiGroups:        - ''      resources:        - configmaps      verbs:        - create    - apiGroups:        - ''      resources:        - endpoints      verbs:        - create        - get        - update    - apiGroups:        - ''      resources:        - events      verbs:        - create        - patch  ---  # Source: ingress-nginx/templates/controller-rolebinding.yaml  apiVersion: rbac.authorization.k8s.io/v1  kind: RoleBinding  metadata:    labels:      helm.sh/chart: ingress-nginx-2.0.3      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/version: 0.32.0      app.kubernetes.io/managed-by: Helm      app.kubernetes.io/component: controller    name: ingress-nginx    namespace: ingress-nginx  roleRef:    apiGroup: rbac.authorization.k8s.io    kind: Role    name: ingress-nginx  subjects:    - kind: ServiceAccount      name: ingress-nginx      namespace: ingress-nginx  ---  # Source: ingress-nginx/templates/controller-service-webhook.yaml  apiVersion: v1  kind: Service  metadata:    labels:      helm.sh/chart: ingress-nginx-2.0.3      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/version: 0.32.0      app.kubernetes.io/managed-by: Helm      app.kubernetes.io/component: controller    name: ingress-nginx-controller-admission    namespace: ingress-nginx  spec:    type: ClusterIP    ports:      - name: https-webhook        port: 443        targetPort: webhook    selector:      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/component: controller  ---  # Source: ingress-nginx/templates/controller-service.yaml  apiVersion: v1  kind: Service  metadata:    labels:      helm.sh/chart: ingress-nginx-2.0.3      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/version: 0.32.0      app.kubernetes.io/managed-by: Helm      app.kubernetes.io/component: controller    name: ingress-nginx-controller    namespace: ingress-nginx  spec:    type: LoadBalancer    externalTrafficPolicy: Local    ports:      - name: http        port: 80        protocol: TCP        targetPort: http      - name: https        port: 443        protocol: TCP        targetPort: https    selector:      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/component: controller  ---  # Source: ingress-nginx/templates/controller-deployment.yaml  apiVersion: apps/v1  kind: Deployment  metadata:    labels:      helm.sh/chart: ingress-nginx-2.0.3      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/version: 0.32.0      app.kubernetes.io/managed-by: Helm      app.kubernetes.io/component: controller    name: ingress-nginx-controller    namespace: ingress-nginx  spec:    selector:      matchLabels:        app.kubernetes.io/name: ingress-nginx        app.kubernetes.io/instance: ingress-nginx        app.kubernetes.io/component: controller    revisionHistoryLimit: 10    minReadySeconds: 0    replicas: 1    template:      metadata:        labels:          app.kubernetes.io/name: ingress-nginx          app.kubernetes.io/instance: ingress-nginx          app.kubernetes.io/component: controller      spec:        dnsPolicy: ClusterFirst        hostNetwork: true        tolerations:        - operator: "Exists"        nodeSelector:          kubernetes.io/hostname: k8s-master-134        initContainers:        - name: adddirperm          image: busybox          command:          - /bin/sh          - -c          - chown -R ${USER_ID}:${USER_ID} ${LOG_DIR}          env:          - name: LOG_DIR            value: /var/log/nginx          - name: USER_ID            value: "101"          volumeMounts:          - name: logdir            mountPath: /var/log/nginx        containers:          - name: controller            image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0            imagePullPolicy: IfNotPresent            lifecycle:              preStop:                exec:                  command:                    - /wait-shutdown            args:              - /nginx-ingress-controller              - --publish-service=ingress-nginx/ingress-nginx-controller              - --election-      resources:        - secrets      verbs:        - get        - create  ---  # Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml  apiVersion: rbac.authorization.k8s.io/v1  kind: RoleBinding  metadata:    name: ingress-nginx-admission    annotations:      helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade      helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded    labels:      helm.sh/chart: ingress-nginx-2.0.3      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/version: 0.32.0      app.kubernetes.io/managed-by: Helm      app.kubernetes.io/component: admission-webhook    namespace: ingress-nginx  roleRef:    apiGroup: rbac.authorization.k8s.io    kind: Role    name: ingress-nginx-admission  subjects:    - kind: ServiceAccount      name: ingress-nginx-admission      namespace: ingress-nginx  ---  # Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml  apiVersion: v1  kind: ServiceAccount  metadata:    name: ingress-nginx-admission    annotations:      helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade      helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded    labels:      helm.sh/chart: ingress-nginx-2.0.3      app.kubernetes.io/name: ingress-nginx      app.kubernetes.io/instance: ingress-nginx      app.kubernetes.io/version: 0.32.0      app.kubernetes.io/managed-by: Helm      app.kubernetes.io/component: admission-webhook    namespace: ingress-nginx

原文出处：coolops -> https://www.coolops.cn/posts/kubernetes-update-nginx-ingress-log/