I’ve been there. Early on, I loved the simplicity of a single state file.
One place to rule them all! But as my infrastructure grew, so did the headaches.
Here’s why I (and probably you, too) needed to split up that Terraform state - whether you’re still running everything in one file, or you’ve already split things up and are wondering if you need to go further.

The monolith: Why one state file feels good… Until it doesn’t

At first, having everything in one state file feels like freedom.
All your resources, all your environments, one command to manage them all.
But then:

• Performance Tanks: Suddenly, terraform plan is slow.
  So slow, you start checking your WiFi.
  But it’s not your connection - it’s the state file, bloated with hundreds (or thousands) of resources.

• The “Oh Crap” Moment: Ever accidentally deleted a production resource while working on dev?
  With one state file, the blast radius of a mistake is the whole infrastructure.
  Definitely not fun.

• Team Gridlock: Terraform locks the state file during changes.
  So if Victoria is updating a subnet and Leon is tweaking a security group, someone’s waiting.
  And waiting.
  And waiting…
  And… You get the point ;)

• Secrets Everywhere: That one state file?
  It’s got all your secrets.
  Anyone who can read it can see everything.
  Not ideal when you want to keep prod secrets, well, secret.

• Environments Collide: Mixing dev, staging, and prod in one state is a recipe for disaster.
  One wrong move, and your “test” change hits prod.

So, what do you do?
You start splitting.

Already split? Here’s why you might need to split again

Maybe you’ve already broken things up - a state file for networking, another for compute, maybe one per environment.
But the pain isn’t gone:

• Still Too Big: Even after splitting, some state files keep growing.
  That “networking” state now covers VPCs, subnets, gateways, peering, and more.
  Time to split again - maybe VPC in one, subnets in another.

• Team Ownership: As teams grow, so do ownership boundaries.
  The DB team doesn’t want to wait for the network team to finish their changes.
  Or maybe you have split teams and responsibilities?
  Give them their own state, let them move fast.

• Parallel Deployments: CI/CD pipelines are happiest when they can run in parallel.
  Multiple state files mean multiple pipelines, all running at once, no waiting.

• Refactoring Time: Infrastructure evolves.
  Maybe you want to turn that old-school monolith into shiny new modules.
  Moving resources into their own state files makes this possible (and safe).

• Compliance & Auditing: Sometimes, it’s not about what you want - it’s what the auditors want.
  Need to separate PCI or GDPR workloads?
  Split the state.

• Complex Dependencies: When you’re referencing resources across modules, things get messy.
  Splitting state helps keep dependencies clear and manageable.

My rule of thumb

Start simple.
But as soon as you feel the pain - slow plans, team friction, scary blast radius - split.
And don’t be afraid to split again as your infra grows.
It’s not a sign of bad design - it’s a sign that you’re scaling.
In Poland we say: “Kto to Panu tak spierdolił?”
Don’t think like that…
Something that may seem as poorly designed, worked in past and delivered a value - it just doesn’t necessarily do anymore and it’s time for a refactor.

If you’re using Terragrunt (like I sometimes do), it makes this even easier.
Each module, each environment, its own state.
Clean, fast, and safe.

TL;DR

Splitting Terraform state files isn’t just a best practice - it’s a survival tactic as your infrastructure grows.
It keeps your team moving, your secrets safe, and your weekends free from “terraform panic” moments.

Been there, done that, got the (split) state files to prove it.

You convinced me! So how do I split a Terraform state?

Prerequisites

1. Ensure your S3 bucket for holding states has versioning enabled.
   You don’t want to find out you have no backup in case of messing the state up.

2. Have a state locking mechanism to avoid external changes in the middle of process.
   If you use Atlantis, you’re probably on the safe side.
   If you orchestrate Terraform with different CI/CD solution, please ensure it will lock the states.

3. Be aware that Terraform state may contain sensitive information.
   You will download this state locally.
   Be very careful so you don’t commit local state file to the repository!
   Ensure you remove all the local state files after you finish.

Steps to move resources between Terraform states

Before you begin, a short clarification note - I use current state to refer to the state you already have before split and that will most likely still exist, just with less resources; and I use new state to refer to the state you create and move resources to it.
In case something is not clear, leave me a comment and I will improve the writings.

1. Create a new state directory in your configuration, with terraform.tf pointing to the new state location in your S3 bucket for states.
   Be sure you don’t change state path in current state!

2. [OPTIONAL] If you use Atlantis, add the new directory to the atlantis.yml file.

3. Open a PR that will lock both states and ensure nobody will interact with these states during migration.

   1. if you use Atlantis, it will create locks. Just make it clear in the PR, that nobody should take the lock down (e.g. add DO NOT TAKE LOCK to the PR title if you don’t have any mechanism that would guarantee nobody takes the lock).

   2. If you don’t use Atlantis, ensure how your CI/CD solution can lock the state you work on.

   3. If you don’t use any CI/CD for Terraform execution, then clearly communicate with all your Terraform contributors, that you are splitting this particular state and they cannot work with it unless you finish the split.

4. Move Terraform resources, modules and outputs to the new state, as needed.

5. Create a terraform_remote_state data source pointing to the new state - the same path you used in new terraform.tf

6. In the current state refer to the outputs of this new terraform_remote_state, for the resources that you moved

7. Pull both states locally with terraform state pull > state.backup, respectively in current and new state directories.
   Remember: Terraform states may contain highly sensitive information - work with caution!

8. In both directories make copies of pulled states with cp state.backup modified.state

9. Now you can move your resources between states with:

    For regular resources:
    terraform state mv -state /path/to/current/state/modified.state -state-out /path/to/new/state/networking/modified.state aws_vpc.my_vpc aws_vpc.my_vpc
   
    For modules:
    terraform state mv -state /path/to/current/state/modified.state -state-out /path/to/new/state/networking/modified.state module.my_module module.my_module
   

   Moving modules is actually awesome here, as you can move entire module, without moving it’s internal resources one by one - this simplifies a lot!

10. Push both states, in both directories, with terraform state push modified.state

11. Plan the new state - you should see no changes to resources if everything went well - but you will see new outputs to be created.
    If not, get back to 9. and move what you missed.

12. Apply the new state so it has the outputs - they are referred in the current state.

13. Plan the current state - you should see no changes if everything went well so far.
    If not, get back to 9. and move what you missed.

14. Apply the current state.

15. At this point everything is good, so merge your PR.

16. Now the process is completed.
    Moving resources produced some modified.state.*.backup in both directories - you can now remove them, as well as modified.state and state.backup files.

17. Search your repositories for other states references to the current state.
    On GitHub you can search through your entire Organization.

    1. If any state refers to the moved resources, add terraform_remote_state data source pointing to the new state, as in 5.

    2. Point relevant resources to that new terraform_remote_state data source

    3. Plan that state - you should see no changes.
       If not - verify previous steps, if you haven’t missed or mistaken anything.

    4. Apply that state.

    5. Repeat for every additional state that referred the state that was split.

18. The process is over, you have cleaned up the state 💪

If you want to dive deeper, or need help with your own Terraform breakup story, hit me up - always happy to talk about infra (or scuba stories 🤿).

You can also hire me as a consultant if you feel you need support - just drop me a message!

I was surprised when I found out…

I was about to implement an Organization wide solution with AWS Control Tower Account Factory for Terraform (AFT) that, in my case, is using Account Name for further resources creation.
As I don’t read Account Aliases often with Terraform, I googled and the first result I got was this article from StackOverflow.
I became a bit worried about how it could affect my solution, but I decided to proceed.
And - unfortunately - I confirmed this is true.

Account Name vs. Account Alias

I want to briefly describe how Account Name and Account Alias differ from each other.

As they are different things, they don’t have to be equal and it may require proper handling depending on the situation.

Account Name

The Account Name in AWS is a name that is used by AWS Organizations and it is organized within the scope of AWS Organizations.

The details regarding AWS Organizations are available only to the Root Account of the AWS Organization, therefore reading it with the aws_organizations_account data source requires configuring Terraform to access that account first to read the details.
This is not straightforward, as it requires reading two accounts at once - technically possible with separate AWS providers to connect to different accounts, but this is not necessarily an optimal approach and may not be possible in some organizations due to additional constraints.
It is also possible to overcome this by storing the Account Name in AWS SSM Parameter Store for each account in the Organization, but we would be already building a solution that requires maintenance, so it may not be suitable option.

Because Account Name exists in terms of AWS Organization, it doesn’t have to be unique across the whole AWS - it only needs to be unique in your Organization.

Contradictory to the Account Alias, Account Name has to be set, it is required.

You can read more about accounts in AWS Organizations in Managing member accounts with AWS Organizations User Guide.

Account Alias

The Account Alias is a different thing.

An AWS Account doesn’t have to be a part of an AWS Organization - so the Account Name for this account may not exist.
Yet every AWS Account can have the Account Alias.

The Account Alias is used to identify a particular AWS Account.
It is optional, so it doesn’t have to be set for an account, but if it is, it has to be unique.
The Account Alias is used to uniquely identify the account and it may be used instead of Account ID to log in to the AWS Console.
The Account Alias is easy to remember and many organizations and individuals use Aliases for simplification.

The Account Alias can also be deleted

My scenario

In my scenario Account Name is different from Account Alias for at least a subset of accounts.
Therefore I could not rely on data source below without additional modifications:

data "aws_iam_account_alias" "current" {}

Luckily I could resolve this, as in my case there is a pattern, so I could manipulate the data.aws_iam_account_alias.current.account_alias value at later step to achieve my goal.

For my scenario it was creating buckets following a certain pattern for logs of one of the landing zone components and I needed to have a unification across the whole Organization.
The optimal solution would be to use aws_s3_bucket data source to read the bucket from each account, when configuring logs that should be pointed to there - but aws_s3_bucket data source doesn’t support wildcards at the moment, full bucket name is required - so this is not a solution for me, it doesn’t simplify anything.

There is an open issue for aws_s3_bucket data source to enable retrieving a list of buckets matching a particular filter, which would work in my scenario really well, but the issue is open since October 2020 and doesn’t seem to be prioritized.
And this is especially frustrating that there was a PR with this feature submitted already!

Luckily, the S3 ARN doesn’t contain any unique IDs, it is in form of arn:aws:s3:::${BUCKET_NAME} and my buckets didn’t require any random ID to be unique, so it was easy to just manipulate the Account Alias accordingly to achieve the final goal.

And I am writing all that because as it was a case for me, it may be as well a case for you!
I work in IT field long enough to see that people follow similar patterns although they have never met each other - some details may differ, but there still can be similar patterns used - and with these patterns, you might fail at some point into this issue, just as I did.

When I wanted to start deploying the app proposed by AWS in their article about this feature, I was hit by the following error when running any kubectl command:

E1230 11:20:11.743615    4991 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E1230 11:20:12.438833    4991 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E1230 11:20:13.117818    4991 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E1230 11:20:13.804336    4991 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E1230 11:20:14.477604    4991 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)

The error message is not really meaningful and helpful, it can be caused by various factors, what I found out when I was googling for a solution, I found this SO question, but the accepted answer was not really covering my case (and it links to AWS re:Post article which is hidden by a premium support paywall).

As my setup was supposed to be really quick, I took the example from Terraform eks_cluster resource docs.

I was diving deeper into the problem and then I finally found out what was going on - my user was missing permissions to the cluster!

So I added the following configuration:

data "aws_iam_user" "jan_tyminski" {
  user_name = "diving.devops"
}

resource "aws_eks_access_entry" "jan_tyminski" {
  cluster_name  = aws_eks_cluster.my_cluster.name
  principal_arn = data.aws_iam_user.jan_tyminski.arn
  type          = "STANDARD"
}

resource "aws_eks_access_policy_association" "jan_tyminski_AmazonEKSAdminPolicy" {
  cluster_name  = aws_eks_cluster.my_cluster.name
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSAdminPolicy"
  principal_arn = aws_eks_access_entry.jan_tyminski.principal_arn

  access_scope {
    type = "cluster"
  }
}

resource "aws_eks_access_policy_association" "jan_tyminski_AmazonEKSClusterAdminPolicy" {
  cluster_name  = aws_eks_cluster.my_cluster.name
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
  principal_arn = aws_eks_access_entry.jan_tyminski.principal_arn

  access_scope {
    type = "cluster"
  }
}

diving.devops is not a real user, this is just my Instagram profile and my YouTube channel.
And this resolved my issue, I could finally start using kubectl, yay!

I have also answered the SO question with my own solution, based on this answer - I showed the Terraform way and added the AmazonEKSAdminPolicy and AmazonEKSClusterAdminPolicy there to be used with note regarding working on PoC in this scenario - so that readers are aware of this issue.

Of course this is just one of the possible scenarios for this error, that was my scenario - it may or may not work for you depending on the underlying issue you have - if that is not your case, I am linking again to the SO question - check it for more solutions.

There was no quick answer available, so I started digging and here's what I found.

CloudFront's docs on CNAMEs and HTTPS requirements say:

CloudFront supports all types of certificates issued by a trusted certificate authority.

CloudFront to Custom Origin setup can only work with trusted third-party CAs:

For origins other than Elastic Load Balancing load balancers, you must use a certificate that is signed by a trusted third-party certificate authority (CA), for example, Comodo, DigiCert, or Symantec.

And in more details CloudFront docs explain that they support the same CAs that Mozilla supports:

When CloudFront uses HTTPS to communicate with your origin, CloudFront verifies that the certificate was issued by a trusted certificate authority. CloudFront supports the same certificate authorities that Mozilla does. For the current list, see Mozilla Included CA Certificate List. You can’t use a self-signed certificate for HTTPS communication between CloudFront and your origin.

AWS also emphasizes in the docs that:

Important
If the origin server returns an expired certificate, an invalid certificate, or a self-signed certificate, or if the origin server returns the certificate chain in the wrong order, CloudFront drops the TCP connection, returns HTTP status code 502 (Bad Gateway) to the viewer, and sets the X-Cache header to Error from cloudfront. Also, if the full chain of certificates, including the intermediate certificate, is not present, CloudFront drops the TCP connection.

Summarising the above - No, it is not possible to set up CloudFront with Custom Origin using a certificate issued by a private Certificate Authority or Self Signed Certificate.

But why do I need to know that?

Because large enterprise companies may have internal CAs for internal purposes and they can be trusted on the company devices and you might be instructed to use a certificate issued by such CA by someone that is not aware of the above.

This short article can help you easily provide required documentation for others in case you are asked if that's possible or requested to implement own certificate in such scenario.

And if you want to understand the reasons behind that, keep reading further.

You are creating an RDS instance with Terraform.

You have coded the instance, executed terraform plan and all looks great!

That's great!
Now you can apply the changes!

So you run terraform apply, Terraform starts calling AWS API and suddenly you get an error:

InvalidParameterValue: DBName must begin with a letter and contain only alphanumeric characters

But why?
You wonder!

And I also wonder!

I have seen this a couple of times already...
But I always forget why this happens...
Mostly I solve this issue once at the beginning of setting up naming convention in the organization and then I forget about this issue...

Now recently I have also forgotten about that, but I was asked by my teammate to help them with the issue.

After a couple of minutes I have finally got it!

We were deploying Postgres!

The problem here is more complex than just AWS or Terraform constraints.
Terraform is supposed to manage AWS resources and AWS provides many different services, some of them are well known open source technologies that can have their own limitations.

Postgres doesn't allow to use hyphens (-) in the database name!
Only underscores (_)!

By the Postgres docs:

SQL identifiers and key words must begin with a letter (a-z, but also letters with diacritical marks and non-Latin letters) or an underscore (_). Subsequent characters in an identifier or key word can be letters, underscores, digits (0-9), or dollar signs ($). Note that dollar signs are not allowed in identifiers according to the letter of the SQL standard, so their use might render applications less portable.

But you didn't set up Postgres?

Apparently MySQL doesn't seem to support hyphens too:

• Permitted characters in unquoted identifiers:

  • ASCII: [0-9,a-z,A-Z$_] (basic Latin letters, digits 0-9, dollar, underscore)

  • Extended: U+0080 .. U+FFFF

And MS SQL Server too:

The first character must be one of the following:

• A letter as defined by the Unicode Standard 3.2. The Unicode definition of letters includes Latin characters from a through z, from A through Z, and also letter characters from other languages.

• The underscore (_), at sign (@), or number sign (#).

I'm neither databases nor SQL expert, but looks like hyphen is not allowed as database name in the most popular SQL engines.

I cannot confirm that for sure, as I cannot find any sort of official SQL Language reference manual in the Internet, but it seems like it is a limitation of SQL for identifiers and database name is an identifier.
Vendors of SQL engines can follow the SQL standard more or less strictly, so there may be engines that could work with hyphens (-) in database name, but this would be an exception to what I have seen in past and during research for this post.

If, by any chance, you know more on that matter, feel free to share your thoughts in a comment!
I'm always happy to hear from you and learn from you!