[Kubernetes] CrashLoopBackOff

kubernetes에서 'nginx' 이미지가 아닌 다른 이미지를 가지고 object를 생성하면, running 상태가 되지 않았다. 기본 'ubuntu' 이미지를 가지고 실험했지만, 마찬가지였다. "kubectl logs" 에는 에러메세지가 뜨지 않았고, 다른 로그는 다음과 같았다.

# kubectl get pods

NAME                        READY     STATUS             RESTARTS   AGE

ubuntu-1-3470236784-crkvd   0/1       CrashLoopBackOff   3          1m

ubuntu-1-3470236784-wl3pk   0/1       CrashLoopBackOff   4          1m

ubuntu-1-3470236784-xd43r   0/1       CrashLoopBackOff   3          1m

# kubectl describe pods

Name: ubuntu-1-3470236784-xd43r

Namespace: default

Node: 10.0.0.95/10.0.0.95

Start Time: Fri, 10 Mar 2017 03:40:24 -0500

Labels: app=ubuntu

pod-template-hash=3470236784

Status: Running

IP: 11.1.21.3

Controllers: ReplicaSet/ubuntu-1-3470236784

Containers:

  ubuntu:

    Container ID: docker://f208797bb2997c0a859e7c5c470c2941292e386c7fc34d1a91b91c6ec4661442

    Image: ubuntu:latest

    Image ID: docker-pullable://ubuntu@sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535

    Port:

    State: Waiting

      Reason: CrashLoopBackOff

    Last State: Terminated

      Reason: Completed

      Exit Code: 0

      Started: Fri, 10 Mar 2017 03:40:47 -0500

      Finished: Fri, 10 Mar 2017 03:40:48 -0500

    Ready: False

    Restart Count: 1

    Volume Mounts: <none>

    Environment Variables: <none>

Conditions:

  Type Status

  Initialized True 

  Ready False 

  PodScheduled True 

No volumes.

QoS Class: BestEffort

Tolerations: <none>

Events:

  FirstSeen LastSeen Count From SubObjectPath Type Reason Message

  --------- -------- ----- ---- ------------- -------- ------ -------

  28s 28s 1 {default-scheduler } Normal Scheduled Successfully assigned ubuntu-1-3470236784-xd43r to 10.0.0.95

  17s 17s 1 {kubelet 10.0.0.95} spec.containers{ubuntu} Normal Created Created container with docker id 63ce059e3c36; Security:[seccomp=unconfined]

  17s 17s 1 {kubelet 10.0.0.95} spec.containers{ubuntu} Normal Started Started container with docker id 63ce059e3c36

  26s 16s 2 {kubelet 10.0.0.95} spec.containers{ubuntu} Normal Pulling pulling image "ubuntu:latest"

  28s 5s 3 {kubelet 10.0.0.95} Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.

  17s 5s 2 {kubelet 10.0.0.95} spec.containers{ubuntu} Normal Pulled Successfully pulled image "ubuntu:latest"

  5s 5s 1 {kubelet 10.0.0.95} spec.containers{ubuntu} Normal Created Created container with docker id f208797bb299; Security:[seccomp=unconfined]

  4s 4s 1 {kubelet 10.0.0.95} spec.containers{ubuntu} Normal Started Started container with docker id f208797bb299

  4s 3s 2 {kubelet 10.0.0.95} spec.containers{ubuntu} Warning BackOff Back-off restarting failed docker container

  4s 3s 2 {kubelet 10.0.0.95} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "ubuntu" with CrashLoopBackOff: "Back-off 10s restarting failed container=ubuntu pod=ubuntu-1-3470236784-xd43r_default(34513428-056d-11e7-9413-74d02b039f40)"

이유는 메인 프로세스가 exit하면서 container도 exit 하기 때문이었다. "docker run" 을 할때 "-dit" 옵션을 주지 않으면, exit 되는 것처럼. 계속해서 실행되는 'CMD'를 지정해주지 않으면(프로세스를 실행시키지 않으면), container들이 곧바로 exit되어서 running 상태가 없었던 것처럼 보여진 것이다. 

실행시킬 프로세스는 없지만, 간단한 이미지를 가지고 테스트를 하고 싶다면 다음과 같은 해결 방법이 있다.

Solution 1 - Dockerfile에 CMD 추가

CMD exec /bin/bash -c "trap : TERM INT; sleep infinity & wait"

// 'kubectl exec' 를 통해 종료명령을 보낼때까지 기다린다.

Solution 2 - yaml에 CMD 추가

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  containers:
  - name: ubuntu
    image: ubuntu:latest
    # Just spin & wait forever
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 30; done;" ]

// 계속해서 sleep을 시킨다.