Maria Goveas

k8s minikube rpc error

March 08, 2020

I started this udemy course on Kubernetes/Microservices to get some hands on experience on k8s and devops in general. My company uses k8s to orchestrate our miroservice containers in AWS - similar to what is being offered in this course. I installed minikube to begin local kubernetes deployment. I also have docker on my macOS so I thought I could get away without installing a specific hypervisor but needed to install Virtualbox anyway.

workloads.yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: position-simulator
spec:
  selector:
    matchLabels:
      app: position-simulator
  replicas: 1
  template:
    metadata:
      labels:
        app: position-simulator
    spec:
      containers:
      - name: queue
        image: richardchesterwood/k8s-fleetman-position-simulator:release1
        env:
        - name: SPRING_PROFILES_ACTIVE
          value: production-microservice

I executed kubectl apply -f workloads.yaml but I got an error

Failed to pull image "richardchesterwood/k8s-fleetman-position-simulator:release1": 
rpc error: code = Unknown desc = Error response from daemon: 
Get https://registry-1.docker.io/v2/: 
dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: 
read udp 192.168.64.3:60508->192.168.64.1:53: read: connection refused

Turns out my minikube’s DNS resolver was not able to lookup docker’s registry. I thought at first that the solution is pretty straightforward. Add a nameserver for resolution in /etc/resolv.conf

Step 1. minikube ssh

Step 2. sudo vi /etc/resolv.conf

Step 3. Add nameserver 8.8.8.8 Save & Exit.

However, there is a warning at the top of the file.

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

So what was my next option? I could ignore the warning and get the docker images in the same session. However, the next time I run kubectl apply my changes to resolve.conf would be reverted.

That’s when I had to retrace my steps. I realized this was probably the driver issue. You see, when I deleted my local k8s instance to increase the RAM, I forgot to specify the vm-driver. So I did minikube delete followed by minikube start --vm-driver=virtualbox.

After this, the docker images were downloaded correctly.