AWS EKS - Error: You must be logged in to the server (the server has asked for the client to provide credentials)

Today I was working on a PoC to try out the new Amazon EKS Auto Mode feature announced on re:Invent 2024.
I wanted to make a very quick PoC, so I took the least effort approach to set up my cluster.

When I wanted to start deploying the app proposed by AWS in their article about this feature, I was hit by the following error when running any kubectl command:

E1230 11:20:11.743615    4991 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E1230 11:20:12.438833    4991 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E1230 11:20:13.117818    4991 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E1230 11:20:13.804336    4991 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E1230 11:20:14.477604    4991 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)

The error message is not really meaningful and helpful, it can be caused by various factors, what I found out when I was googling for a solution, I found this SO question, but the accepted answer was not really covering my case (and it links to AWS re:Post article which is hidden by a premium support paywall).

As my setup was supposed to be really quick, I took the example from Terraform eks_cluster resource docs.

I was diving deeper into the problem and then I finally found out what was going on - my user was missing permissions to the cluster!

So I added the following configuration:

data "aws_iam_user" "jan_tyminski" {
  user_name = "diving.devops"
}

resource "aws_eks_access_entry" "jan_tyminski" {
  cluster_name  = aws_eks_cluster.my_cluster.name
  principal_arn = data.aws_iam_user.jan_tyminski.arn
  type          = "STANDARD"
}

resource "aws_eks_access_policy_association" "jan_tyminski_AmazonEKSAdminPolicy" {
  cluster_name  = aws_eks_cluster.my_cluster.name
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSAdminPolicy"
  principal_arn = aws_eks_access_entry.jan_tyminski.principal_arn

  access_scope {
    type = "cluster"
  }
}

resource "aws_eks_access_policy_association" "jan_tyminski_AmazonEKSClusterAdminPolicy" {
  cluster_name  = aws_eks_cluster.my_cluster.name
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
  principal_arn = aws_eks_access_entry.jan_tyminski.principal_arn

  access_scope {
    type = "cluster"
  }
}

diving.devops is not a real user, this is just my Instagram profile and my YouTube channel.
And this resolved my issue, I could finally start using kubectl, yay!

I have also answered the SO question with my own solution, based on this answer - I showed the Terraform way and added the AmazonEKSAdminPolicy and AmazonEKSClusterAdminPolicy there to be used with note regarding working on PoC in this scenario - so that readers are aware of this issue.

Of course this is just one of the possible scenarios for this error, that was my scenario - it may or may not work for you depending on the underlying issue you have - if that is not your case, I am linking again to the SO question - check it for more solutions.