Incident and alert rules

Container is unhealthy

Field checked: status.containerStatuses[].state

Opens when

• A container is in the waiting state with any reason other than ContainerCreating or PodInitializing, for example CrashLoopBackOff, ImagePullBackOff, ErrImagePull or CreateContainerConfigError.
• A waiting state without a reason also opens the incident, reported as “Waiting”.

Resolves when

• The container has been running for at least 5 minutes, measured from its start time. Gita checks again at that moment even if the Pod does not change.
• The container terminates and the Pod will not restart it: restartPolicy Never, OnFailure with exit code 0, or a Pod that has Succeeded or Failed.
• The container no longer appears in the Pod status, or the Pod is deleted.

Missing or unknown status

A restart keeps the incident open: ContainerCreating, PodInitializing, a run shorter than 5 minutes and a termination that will be restarted do not resolve it, so a CrashLoopBackOff stays one incident instead of reopening on every restart. A running Pod that declares containers but reports no container statuses is not evaluated; the current state of its findings is kept.

Gita opens one incident per container, so two failing containers in one Pod are two incidents. A terminated container does not open this rule, even with a non-zero exit code: a Pod that fails as a whole is reported by GITA-INC-002. Init containers are not evaluated.

status:
  phase: Running
  containerStatuses:
    - name: api
      state:
        waiting:
          reason: CrashLoopBackOff   # opens GITA-INC-001 for "api"

kubectl get pod <pod> -n <namespace> \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.state}{"\n"}{end}'

Pod status is unhealthy

Field checked: status.phase

Opens when

• The Pod phase is anything other than Running, Succeeded or Pending: Failed or Unknown.

Resolves when

• The phase returns to Running, Succeeded or Pending, or the Pod is deleted.

Missing or unknown status

A Pod without a phase is not evaluated; open incidents stay open.

Pending is not an incident by itself. A Pod stuck in Pending because an image cannot be pulled is reported by GITA-INC-001 through its waiting container.

status:
  phase: Failed        # opens GITA-INC-002
  reason: Evicted

kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.phase}'

Node is not ready

Field checked: status.conditions[type=Ready].status

Opens when

• The Ready condition is False: the kubelet reports that the Node cannot run Pods.
• The Ready condition is Unknown: the node controller stopped receiving status from the kubelet, usually because the Node is down or unreachable.

Resolves when

• The Ready condition becomes True, or the Node is removed from the cluster.

Missing or unknown status

A Node without a Ready condition is not evaluated; open incidents stay open.

status:
  conditions:
    - type: Ready
      status: "Unknown"   # opens GITA-INC-005
      reason: NodeStatusUnknown

kubectl get node <node> \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

Namespace is not active

Field checked: status.phase

Opens when

• The Namespace phase is anything other than Active, typically Terminating.

Resolves when

• The phase becomes Active again, or the Namespace finishes deleting.

Missing or unknown status

A Namespace without a phase is not evaluated; open incidents stay open.

A Namespace stuck in Terminating usually has resources with finalizers that cannot complete. Namespace exclusions do not apply to this rule.

status:
  phase: Terminating   # opens GITA-INC-007

kubectl get namespace <namespace> -o jsonpath='{.status.phase}'

Kubelet has insufficient disk space

Field checked: status.conditions[type=DiskPressure].status

Opens when

• The DiskPressure condition is True: available disk or inodes on the Node fell below the kubelet eviction threshold. The kubelet may evict Pods and refuse new ones.

Resolves when

• The DiskPressure condition becomes False, or the Node is removed from the cluster.

Missing or unknown status

If DiskPressure is missing or Unknown, the alert keeps its current state: an open alert stays open and no new alert opens. This happens when a Node stops reporting status, which is reported separately by GITA-INC-005.

status:
  conditions:
    - type: DiskPressure
      status: "True"   # opens the alert

kubectl get node <node> \
  -o jsonpath='{.status.conditions[?(@.type=="DiskPressure")].status}'

Kubelet has insufficient memory

Field checked: status.conditions[type=MemoryPressure].status

Opens when

• The MemoryPressure condition is True: available memory on the Node fell below the kubelet eviction threshold. The kubelet may evict Pods.

Resolves when

• The MemoryPressure condition becomes False, or the Node is removed from the cluster.

Missing or unknown status

If MemoryPressure is missing or Unknown, the alert keeps its current state: an open alert stays open and no new alert opens. This happens when a Node stops reporting status, which is reported separately by GITA-INC-005.

status:
  conditions:
    - type: MemoryPressure
      status: "True"   # opens the alert

kubectl get node <node> \
  -o jsonpath='{.status.conditions[?(@.type=="MemoryPressure")].status}'

Kubelet has insufficient PIDs

Field checked: status.conditions[type=PIDPressure].status

Opens when

• The PIDPressure condition is True: the number of available process IDs on the Node fell below the kubelet threshold. New processes may fail to start.

Resolves when

• The PIDPressure condition becomes False, or the Node is removed from the cluster.

Missing or unknown status

If PIDPressure is missing or Unknown, the alert keeps its current state: an open alert stays open and no new alert opens. This happens when a Node stops reporting status, which is reported separately by GITA-INC-005.

status:
  conditions:
    - type: PIDPressure
      status: "True"   # opens the alert

kubectl get node <node> \
  -o jsonpath='{.status.conditions[?(@.type=="PIDPressure")].status}'