[SOLVED] K8s Containers dont start ImagePullBackOff, ErrImagePull

Issue

I am trying to start up a couple of containers locally using k8s but container creation is stopped cause of ImagePullBackOff, ErrImagePull. The yaml is fine, tested it on another workstation. And i can pull images using regular docker. But it fails in k8s/minikube environment

Error container logs is

Error from server (BadRequest): container "mongo-express" in pod "mongoexpress-deployment-bd7cf697b-nc4h5" is waiting to start: trying and failing to pull image

Error in minikube dashboard is

Failed to pull image "docker.io/mongo": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 

I tried pulling the image to my local docker cache and running

eval $(minikube docker-env)

But i keep getting this error. It doesnt see local image repository and it doesnt dowload the image by itself.

I am 100% sure it has something to do with user access on Fedora. But dont have any idea what to do, and i've been trying to fix this for a couple of days :(.

Please help, thank you

Dont know if this helps: I tried using k3s. Image pull is successful, but minikube isnt compatible with it on Fedora.

Also... If i try using docker without sudo it doesnt pull images. With sudo it pulls.

OS is Fedora, and i am using docker, kubernetes, minikube, podman as driver.

- linux version
NAME="Fedora Linux"
VERSION="35 (Workstation Edition)"
- kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", 
- docker version
Version:          20.10.12
- minikube version
minikube version: v1.25.2

I am trying to start up locally this yaml file

apiVersion: v1
kind: Secret
metadata:
  name: mongodb-secret
type: Opaque
data:
  mongo-root-username: dXNlcm5hbWU=
  mongo-root-password: cGFzc3dvcmQ=
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mongodb-configmap
data:
  database_url: mongodb-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongoexpress-deployment
  labels:
    app: mongoexpress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongoexpress
  template:
    metadata:
      labels:
        app: mongoexpress
    spec:
      containers:
        - name: mongo-express
          image: mongo-express
          ports:
            - containerPort: 8081
          env:
            - name: ME_CONFIG_MONGODB_ADMINUSERNAME
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: mongo-root-username
            - name: ME_CONFIG_MONGODB_ADMINPASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: mongo-root-password
            - name: ME_CONFIG_MONGODB_SERVER
              valueFrom:
                configMapKeyRef:
                  name: mongodb-configmap
                  key: database_url
            - name: WHATEVER
              value: Someconfig
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
  labels:
    app: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
        - name: mongodb
          image: mongo
          ports:
            - containerPort: 27017
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              valueFrom:
                  secretKeyRef:
                    name: mongodb-secret
                    key: mongo-root-username
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom:
                  secretKeyRef:
                    name: mongodb-secret
                    key: mongo-root-password
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-service
spec:
  selector:
    app: mongodb
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: mongoexpress-service
spec:
  selector:
    app: mongoexpress-deployment
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 8081
      targetPort: 8081
      nodePort: 30000

Solution

Based on the comments, my suggestion is to use the docker driver, since Docker has been installed in the system and is the preferred stable driver.

minikube start --driver=docker

You can also set this as the default driver.

minikube config set driver docker
minikube start

That doesn't explain why it doesn't work with podman, though.

Answered By - The Fool

Answer Checked By - Katrina (WPSolving Volunteer)