nginx容器 报错 nginx: [emerg] mkdir() “varcachenginxclient

nginx容器 报错 nginx: [emerg] mkdir() “/var/cache/nginx/client 现象

之前用docker运行了一个nginx镜像没有任何报错,但是用k8s启动镜像的时候就报错"nginx: [emerg] mkdir() “/var/cache/nginx/client_temp” failed (13: Permission denied)" 而且是在特定的命名空间下面才会出现这个错误.正常的docker版本是17.03.3-ce不正常的docker版本是Docker 19.03.4 后者用的是Overlay2存储驱动.

思考

根据报错信息可以明显看出是用户权限的问题, 之前也遇到过类似的nginx的权限问题但那是由于SELinux 的设置导致的,关闭SELinux后就恢复正常了 设置方法可参考 “CentOS 7.X 关闭SELinux”
发现k8s启动还是失败, 之前还看到了一篇博客Unable to run NGINX Docker due to “13: Permission denied” 通过执行下面的命令把container_t 加到SELinux中,但是还是失败了

semanage permissive -a container_t
semodule -l | grep permissive

其他

另外我还通过为 Pod 或容器配置安全性上下文的方式来试着解决这个问题,yaml关于安全上下文的配置是

  securityContext:
    fsGroup: 1000
    runAsGroup: 1000
    runAsUser: 1000
    runAsNonRoot: true

配置方式参考 https://kubernetes.io/zh/docs/tasks/configure-pod-container/security-context/ 但是配置之后仍然没有生效, 关于这个bug的详细讨论可以查看 Bug 1850148 - Executing mkdir commands inside pods results in Permission denied
或者 “why-i-am-getting-read-only-file-system-error-from-nginx-in-my-container”

最后

无奈之下最后只能直接做个非root用户启动的nginx镜像了, 按照 https://github.com/nginxinc/docker-nginx-unprivileged 项目制作自己的镜像
先查看 自己启动pod的用户id和组id,可以通过id <用户名>的命令查看,例如:

[deploy@host ~]$ id deploy
uid=1000(deploy) gid=1000(deploy) 组=1000(deploy),980(docker) 

你需要将项目中的Dockerfile 里面的UID和GID 分别修改成自己用户对应的id,我的用户id和组id都是1000
我还加了一行使用阿里云镜像的设置,不然构建镜像会特别慢,你还可以自己添加一些自定义设置, 需要注意的是镜像暴露的是8080端口而不是80端口, 非root用户不能直接启动80端口

Dockerfile:

#
# NOTE: THIS DOCKERFILE IS GENERATED VIA "update.sh"
#
# PLEASE DO NOT EDIT IT DIRECTLY.
#
ARG IMAGE=alpine:3.13
FROM $IMAGE

LABEL maintainer="NGINX Docker Maintainers "

ENV NGINX_VERSION 1.20.1
ENV NJS_VERSION   0.5.3
ENV PKG_RELEASE   1

ARG UID=1000
ARG GID=1000

RUN set -x 
    && sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories 
# create nginx user/group first, to be consistent throughout docker variants
    && addgroup -g $GID -S nginx 
    && adduser -S -D -H -u $UID -h /var/cache/nginx -s /sbin/nologin -G nginx -g nginx nginx 
    && apkArch="$(cat /etc/apk/arch)" 
    && nginxPackages=" 
        nginx=${NGINX_VERSION}-r${PKG_RELEASE} 
        nginx-module-xslt=${NGINX_VERSION}-r${PKG_RELEASE} 
        nginx-module-geoip=${NGINX_VERSION}-r${PKG_RELEASE} 
        nginx-module-image-filter=${NGINX_VERSION}-r${PKG_RELEASE} 
        nginx-module-njs=${NGINX_VERSION}.${NJS_VERSION}-r${PKG_RELEASE} 
    " 
    && case "$apkArch" in 
        x86_64|aarch64) 
# arches officially built by upstream
            set -x 
            && KEY_SHA512="e7fa8303923d9b95db37a77ad46c68fd4755ff935d0a534d26eba83de193c76166c68bfe7f65471bf8881004ef4aa6df3e34689c305662750c0172fca5d8552a *stdin" 
            && apk add --no-cache --virtual .cert-deps 
                openssl 
            && wget -O /tmp/nginx_signing.rsa.pub https://nginx.org/keys/nginx_signing.rsa.pub 
            && if [ "$(openssl rsa -pubin -in /tmp/nginx_signing.rsa.pub -text -noout | openssl sha512 -r)" = "$KEY_SHA512" ]; then 
                echo "key verification succeeded!"; 
                mv /tmp/nginx_signing.rsa.pub /etc/apk/keys/; 
            else 
                echo "key verification failed!"; 
                exit 1; 
            fi 
            && apk del .cert-deps 
            && apk add -X "https://nginx.org/packages/alpine/v$(egrep -o '^[0-9]+.[0-9]+' /etc/alpine-release)/main" --no-cache $nginxPackages 
            ;; 
        *) 
# we're on an architecture upstream doesn't officially build for
# let's build binaries from the published packaging sources
            set -x 
            && tempDir="$(mktemp -d)" 
            && chown nobody:nobody $tempDir 
            && apk add --no-cache --virtual .build-deps 
                gcc 
                libc-dev 
                make 
                openssl-dev 
                pcre-dev 
                zlib-dev 
                linux-headers 
                libxslt-dev 
                gd-dev 
                geoip-dev 
                perl-dev 
                libedit-dev 
                mercurial 
                bash 
                alpine-sdk 
                findutils 
            && su nobody -s /bin/sh -c " 
                export HOME=${tempDir} 
                && cd ${tempDir} 
                && hg clone https://hg.nginx.org/pkg-oss 
                && cd pkg-oss 
                && hg up ${NGINX_VERSION}-${PKG_RELEASE} 
                && cd alpine 
                && make all 
                && apk index -o ${tempDir}/packages/alpine/${apkArch}/APKINDEX.tar.gz ${tempDir}/packages/alpine/${apkArch}/*.apk 
                && abuild-sign -k ${tempDir}/.abuild/abuild-key.rsa ${tempDir}/packages/alpine/${apkArch}/APKINDEX.tar.gz 
                " 
            && cp ${tempDir}/.abuild/abuild-key.rsa.pub /etc/apk/keys/ 
            && apk del .build-deps 
            && apk add -X ${tempDir}/packages/alpine/ --no-cache $nginxPackages 
            ;; 
    esac 
# if we have leftovers from building, let's purge them (including extra, unnecessary build deps)
    && if [ -n "$tempDir" ]; then rm -rf "$tempDir"; fi 
    && if [ -n "/etc/apk/keys/abuild-key.rsa.pub" ]; then rm -f /etc/apk/keys/abuild-key.rsa.pub; fi 
    && if [ -n "/etc/apk/keys/nginx_signing.rsa.pub" ]; then rm -f /etc/apk/keys/nginx_signing.rsa.pub; fi 
# Bring in gettext so we can get `envsubst`, then throw
# the rest away. To do this, we need to install `gettext`
# then move `envsubst` out of the way so `gettext` can
# be deleted completely, then move `envsubst` back.
    && apk add --no-cache --virtual .gettext gettext 
    && mv /usr/bin/envsubst /tmp/ 
    
    && runDeps="$( 
        scanelf --needed --nobanner /tmp/envsubst 
            | awk '{ gsub(/,/, "nso:", $2); print "so:" $2 }' 
            | sort -u 
            | xargs -r apk info --installed 
            | sort -u 
    )" 
    && apk add --no-cache $runDeps 
    && apk del .gettext 
    && mv /tmp/envsubst /usr/local/bin/ 
# Bring in tzdata so users could set the timezones through the environment
# variables
    && apk add --no-cache tzdata 
# Bring in curl and ca-certificates to make registering on DNS SD easier
    && apk add --no-cache curl ca-certificates 
# forward request and error logs to docker log collector
    && ln -sf /dev/stdout /var/log/nginx/access.log 
    && ln -sf /dev/stderr /var/log/nginx/error.log 
# create a docker-entrypoint.d directory
    && mkdir /docker-entrypoint.d

# implement changes required to run NGINX as an unprivileged user
RUN sed -i 's,listen       80;,listen       8080;,' /etc/nginx/conf.d/default.conf 
    && sed -i '/user  nginx;/d' /etc/nginx/nginx.conf 
    && sed -i 's,/var/run/nginx.pid,/tmp/nginx.pid,' /etc/nginx/nginx.conf 
    && sed -i "/^http {/a     proxy_temp_path /tmp/proxy_temp;n    client_body_temp_path /tmp/client_temp;n    fastcgi_temp_path /tmp/fastcgi_temp;n    uwsgi_temp_path /tmp/uwsgi_temp;n    scgi_temp_path /tmp/scgi_temp;n" /etc/nginx/nginx.conf 
# nginx user must own the cache and etc directory to write cache and tweak the nginx config
    && chown -R $UID:0 /var/cache/nginx 
    && chmod -R g+w /var/cache/nginx 
    && chown -R $UID:0 /etc/nginx 
    && chmod -R g+w /etc/nginx

COPY docker-entrypoint.sh /
COPY 10-listen-on-ipv6-by-default.sh /docker-entrypoint.d
COPY 20-envsubst-on-templates.sh /docker-entrypoint.d
COPY 30-tune-worker-processes.sh /docker-entrypoint.d
RUN  chmod 755 /docker-entrypoint.sh 
     && chmod 755 /docker-entrypoint.d/*.sh

ENTRYPOINT ["/docker-entrypoint.sh"]

EXPOSE 8080

STOPSIGNAL SIGQUIT

USER $UID

CMD ["nginx", "-g", "daemon off;"]

10-listen-on-ipv6-by-default.sh:

#!/bin/sh
# vim:sw=4:ts=4:et

set -e

ME=$(basename $0)
DEFAULT_CONF_FILE="etc/nginx/conf.d/default.conf"

# check if we have ipv6 available
if [ ! -f "/proc/net/if_inet6" ]; then
    echo >&3 "$ME: info: ipv6 not available"
    exit 0
fi

if [ ! -f "/$DEFAULT_CONF_FILE" ]; then
    echo >&3 "$ME: info: /$DEFAULT_CONF_FILE is not a file or does not exist"
    exit 0
fi

# check if the file can be modified, e.g. not on a r/o filesystem
touch /$DEFAULT_CONF_FILE 2>/dev/null || { echo >&3 "$ME: info: can not modify /$DEFAULT_CONF_FILE (read-only file system?)"; exit 0; }

# check if the file is already modified, e.g. on a container restart
grep -q "listen  [::]:8080;" /$DEFAULT_CONF_FILE && { echo >&3 "$ME: info: IPv6 listen already enabled"; exit 0; }

if [ -f "/etc/os-release" ]; then
    . /etc/os-release
else
    echo >&3 "$ME: info: can not guess the operating system"
    exit 0
fi

echo >&3 "$ME: info: Getting the checksum of /$DEFAULT_CONF_FILE"

case "$ID" in
    "debian")
        CHECKSUM=$(dpkg-query --show --showformat='${Conffiles}n' nginx | grep $DEFAULT_CONF_FILE | cut -d' ' -f 3)
        echo "$CHECKSUM  /$DEFAULT_CONF_FILE" | md5sum -c - >/dev/null 2>&1 || {
            echo >&3 "$ME: info: /$DEFAULT_CONF_FILE differs from the packaged version"
            exit 0
        }
        ;;
    "alpine")
        CHECKSUM=$(apk manifest nginx 2>/dev/null| grep $DEFAULT_CONF_FILE | cut -d' ' -f 1 | cut -d ':' -f 2)
        echo "$CHECKSUM  /$DEFAULT_CONF_FILE" | sha1sum -c - >/dev/null 2>&1 || {
            echo >&3 "$ME: info: /$DEFAULT_CONF_FILE differs from the packaged version"
            exit 0
        }
        ;;
    *)
        echo >&3 "$ME: info: Unsupported distribution"
        exit 0
        ;;
esac

# enable ipv6 on default.conf listen sockets
sed -i -E 's,listen       8080;,listen       8080;n    listen  [::]:8080;,' /$DEFAULT_CONF_FILE

echo >&3 "$ME: info: Enabled listen on IPv6 in /$DEFAULT_CONF_FILE"

exit 0

20-envsubst-on-templates.sh:

#!/bin/sh

set -e

ME=$(basename $0)

auto_envsubst() {
  local template_dir="${NGINX_ENVSUBST_TEMPLATE_DIR:-/etc/nginx/templates}"
  local suffix="${NGINX_ENVSUBST_TEMPLATE_SUFFIX:-.template}"
  local output_dir="${NGINX_ENVSUBST_OUTPUT_DIR:-/etc/nginx/conf.d}"

  local template defined_envs relative_path output_path subdir
  defined_envs=$(printf '${%s} ' $(env | cut -d= -f1))
  [ -d "$template_dir" ] || return 0
  if [ ! -w "$output_dir" ]; then
    echo >&3 "$ME: ERROR: $template_dir exists, but $output_dir is not writable"
    return 0
  fi
  find "$template_dir" -follow -type f -name "*$suffix" -print | while read -r template; do
    relative_path="${template#$template_dir/}"
    output_path="$output_dir/${relative_path%$suffix}"
    subdir=$(dirname "$relative_path")
    # create a subdirectory where the template file exists
    mkdir -p "$output_dir/$subdir"
    echo >&3 "$ME: Running envsubst on $template to $output_path"
    envsubst "$defined_envs" < "$template" > "$output_path"
  done
}

auto_envsubst

exit 0

30-tune-worker-processes.sh:

#!/bin/sh
# vim:sw=2:ts=2:sts=2:et

set -eu

LC_ALL=C
ME=$( basename "$0" )
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

[ "${NGINX_ENTRYPOINT_WORKER_PROCESSES_AUTOTUNE:-}" ] || exit 0

touch /etc/nginx/nginx.conf 2>/dev/null || { echo >&2 "$ME: error: can not modify /etc/nginx/nginx.conf (read-only file system?)"; exit 0; }

ceildiv() {
  num=$1
  div=$2
  echo $(( (num + div - 1) / div ))
}

get_cpuset() {
  cpusetroot=$1
  cpusetfile=$2
  ncpu=0
  [ -f "$cpusetroot/$cpusetfile" ] || return 1
  for token in $( tr ',' ' ' < "$cpusetroot/$cpusetfile" ); do
    case "$token" in
      *-*)
        count=$( seq $(echo "$token" | tr '-' ' ') | wc -l )
        ncpu=$(( ncpu+count ))
        ;;
      *)
        ncpu=$(( ncpu+1 ))
        ;;
    esac
  done
  echo "$ncpu"
}

get_quota() {
  cpuroot=
  ncpu=0
  [ -f "$cpuroot/cpu.cfs_quota_us" ] || return 1
  [ -f "$cpuroot/cpu.cfs_period_us" ] || return 1
  cfs_quota=$( cat "$cpuroot/cpu.cfs_quota_us" )
  cfs_period=$( cat "$cpuroot/cpu.cfs_period_us" )
  [ "$cfs_quota" = "-1" ] && return 1
  [ "$cfs_period" = "0" ] && return 1
  ncpu=$( ceildiv "$cfs_quota" "$cfs_period" )
  [ "$ncpu" -gt 0 ] || return 1
  echo "$ncpu"
}

get_quota_v2() {
  cpuroot=
  ncpu=0
  [ -f "$cpuroot/cpu.max" ] || return 1
  cfs_quota=$( cut -d' ' -f 1 < "$cpuroot/cpu.max" )
  cfs_period=$( cut -d' ' -f 2 < "$cpuroot/cpu.max" )
  [ "$cfs_quota" = "max" ] && return 1
  [ "$cfs_period" = "0" ] && return 1
  ncpu=$( ceildiv "$cfs_quota" "$cfs_period" )
  [ "$ncpu" -gt 0 ] || return 1
  echo "$ncpu"
}

get_cgroup_v1_path() {
  needle=
  found=
  foundroot=
  mountpoint=

  [ -r "/proc/self/mountinfo" ] || return 1
  [ -r "/proc/self/cgroup" ] || return 1

  while IFS= read -r line; do
    case "$needle" in
      "cpuset")
        case "$line" in
          *cpuset*)
            found=$( echo "$line" | cut -d ' ' -f 4,5 )
            break
            ;;
        esac
        ;;
      "cpu")
        case "$line" in
          *cpuset*)
            ;;
          *cpu,cpuacct*|*cpuacct,cpu|*cpuacct*|*cpu*)
            found=$( echo "$line" | cut -d ' ' -f 4,5 )
            break
            ;;
        esac
    esac
  done << __EOF__
$( grep -F -- '- cgroup ' /proc/self/mountinfo )
__EOF__

  while IFS= read -r line; do
    controller=$( echo "$line" | cut -d: -f 2 )
    case "$needle" in
      "cpuset")
        case "$controller" in
          cpuset)
            mountpoint=$( echo "$line" | cut -d: -f 3 )
            break
            ;;
        esac
        ;;
      "cpu")
        case "$controller" in
          cpu,cpuacct|cpuacct,cpu|cpuacct|cpu)
            mountpoint=$( echo "$line" | cut -d: -f 3 )
            break
            ;;
        esac
        ;;
    esac
done << __EOF__
$( grep -F -- 'cpu' /proc/self/cgroup )
__EOF__

  case "${found%% *}" in
    "/")
      foundroot="${found##* }$mountpoint"
      ;;
    "$mountpoint")
      foundroot="${found##* }"
      ;;
  esac
  echo "$foundroot"
}

get_cgroup_v2_path() {
  found=
  foundroot=
  mountpoint=

  [ -r "/proc/self/mountinfo" ] || return 1
  [ -r "/proc/self/cgroup" ] || return 1

  while IFS= read -r line; do
    found=$( echo "$line" | cut -d ' ' -f 4,5 )
  done << __EOF__
$( grep -F -- '- cgroup2 ' /proc/self/mountinfo )
__EOF__

  while IFS= read -r line; do
    mountpoint=$( echo "$line" | cut -d: -f 3 )
done << __EOF__
$( grep -F -- '0::' /proc/self/cgroup )
__EOF__

  case "${found%% *}" in
    "")
      return 1
      ;;
    "/")
      foundroot="${found##* }$mountpoint"
      ;;
    "$mountpoint")
      foundroot="${found##* }"
      ;;
  esac
  echo "$foundroot"
}

ncpu_online=$( getconf _NPROCESSORS_onLN )
ncpu_cpuset=
ncpu_quota=
ncpu_cpuset_v2=
ncpu_quota_v2=

cpuset=$( get_cgroup_v1_path "cpuset" ) && ncpu_cpuset=$( get_cpuset "$cpuset" "cpuset.effective_cpus" ) || ncpu_cpuset=$ncpu_online
cpu=$( get_cgroup_v1_path "cpu" ) && ncpu_quota=$( get_quota "$cpu" ) || ncpu_quota=$ncpu_online
cgroup_v2=$( get_cgroup_v2_path ) && ncpu_cpuset_v2=$( get_cpuset "$cgroup_v2" "cpuset.cpus.effective" ) || ncpu_cpuset_v2=$ncpu_online
cgroup_v2=$( get_cgroup_v2_path ) && ncpu_quota_v2=$( get_quota_v2 "$cgroup_v2" ) || ncpu_quota_v2=$ncpu_online

ncpu=$( printf "%sn%sn%sn%sn%sn" 
               "$ncpu_online" 
               "$ncpu_cpuset" 
               "$ncpu_quota" 
               "$ncpu_cpuset_v2" 
               "$ncpu_quota_v2" 
               | sort -n 
               | head -n 1 )

sed -i.bak -r 's/^(worker_processes)(.*)$/# Commented out by '"$ME"' on '"$(date)"'n#12n1 '"$ncpu"';/' /etc/nginx/nginx.conf

docker-entrypoint.sh:

#!/bin/sh
# vim:sw=4:ts=4:et

set -e

if [ -z "${NGINX_ENTRYPOINT_QUIET_LOGS:-}" ]; then
    exec 3>&1
else
    exec 3>/dev/null
fi

if [ "$1" = "nginx" -o "$1" = "nginx-debug" ]; then
    if /usr/bin/find "/docker-entrypoint.d/" -mindepth 1 -maxdepth 1 -type f -print -quit 2>/dev/null | read v; then
        echo >&3 "$0: /docker-entrypoint.d/ is not empty, will attempt to perform configuration"

        echo >&3 "$0: Looking for shell scripts in /docker-entrypoint.d/"
        find "/docker-entrypoint.d/" -follow -type f -print | sort -V | while read -r f; do
            case "$f" in
                *.sh)
                    if [ -x "$f" ]; then
                        echo >&3 "$0: Launching $f";
                        "$f"
                    else
                        # warn on shell scripts without exec bit
                        echo >&3 "$0: Ignoring $f, not executable";
                    fi
                    ;;
                *) echo >&3 "$0: Ignoring $f";;
            esac
        done

        echo >&3 "$0: Configuration complete; ready for start up"
    else
        echo >&3 "$0: No files found in /docker-entrypoint.d/, skipping configuration"
    fi
fi

exec "$@"

把上面几个文件放到同一个目录下面然后执行 docker build -t nginxinc / docker-nginx-unprivileged :latest 就能构建镜像了

参考

1. https://www.thegeekdiary.com/unable-to-run-nginx-docker-due-to-13-permission-denied/
2. https://github.com/nginxinc/docker-nginx-unprivileged/blob/main/stable/alpine/Dockerfile
3. https://kubernetes.io/zh/docs/tasks/configure-pod-container/security-context/
4. https://bugzilla.redhat.com/show_bug.cgi?id=1850148
5. https://discuss.kubernetes.io/t/why-i-am-getting-read-only-file-system-error-from-nginx-in-my-container/5782