Shortcuts for infrastructure are inexpensive while creating them; the real price is paid at a later stage.
The process of deployment that saves two days of configuration today may become a month-long migration effort in two years. The staging environment that is “good enough” today might one day lead to an increased reluctance to trust every single release. A small number of cloud assets that no one cares to remove can silently grow into a permanent expense no one feels comfortable cutting back on.
All of these issues don't normally arise because of a poor choice. Rather, they result from a rational choice, which is that a team has to ship fast, so one developer will handle deployment manually, and CI/CD, documentation, and staging can be done later.
There is nothing apparently wrong, and that is precisely what makes it dangerous. The first shortcut sets up the stage for the second one. Manual deployment practices result in small scripts, which in turn have many assumptions built into them. These assumptions are never documented but rather remain in the heads of the developers. Production moves faster than staging, and the trust in the environments is eroded.
By the time the company decides to “fix the infrastructure,” it is no longer fixing one shortcut. It is untangling a system built around years of them.
It Starts With One Person Deploying to Production
At the beginning, manual deployment can be completely reasonable. The team is small, releases are infrequent, and one engineer knows the application well enough to build it, connect to the server, update the configuration, restart the right services, and verify that everything is running. Maybe the whole process takes 20 minutes.
Then the product starts moving faster. Releases happen twice a week instead of twice a month, another service appears, database migrations become more common, and a new developer joins the team. The deployment process, however, stays mostly the same.
The engineer now needs to remember what service needs to be restarted first, what migration must occur prior to deploying, and what environment variable needs to be verified manually. A process that used to take 20 minutes now takes 45 minutes to an hour. This may not seem too bad, but when two releases are done a day, it starts to add up to many engineering hours annually.
Ultimately, deployment starts taking place when the individual who knows about the process is around. Nobody intended to create a bottleneck; the bottleneck merely formed itself out of a process that was never reviewed.
The First Fix Is Usually Another Shortcut
At some point, someone writes a script. It is not meant to become infrastructure; it is just meant to save time.
A Bash script copies files to the server, another restarts the application, and someone adds a command for database migrations. Deployments get faster again, so the problem appears solved.
However, now that the process exists between scripts, local configuration, cloud configuration, and undocumented conventions. The scripts assume that the right environment is there, the credentials exist, everything is running properly, and nothing has been done manually in production last week.
When one of those assumptions stops being true, somebody patches the script. Then patches it again. The temporary automation slowly becomes a deployment system, except nobody designed it as one.
Every Exception Creates More Knowledge That Exists Only in People’s Heads
With sufficient growth in special cases within the infrastructure, documentation begins to lag behind. Not due to a lack of caring about documentation, but due to the system evolving at a faster rate than documentation can be updated.
An engineer realizes that there is a need for an additional restart for a service after deployment. Another engineer discovers that a certain configuration in the production environment cannot be modified without modifying some other part of the system. There is a cron job that looks like it is no longer needed, but is actually used somewhere in some obsolete process.
These details live in Slack conversations, old tickets, or memory. The infrastructure still works because the people who built it are still around, which creates a false sense of stability.
The system looks reliable when, in reality, the team is manually supplying the context that the system itself is missing. And the more this happens, the harder automation becomes, because you cannot easily automate a process if nobody can fully describe it.
Meanwhile, Production Keeps Changing
Production does not stand still while infrastructure maintenance is delayed. A bottleneck arises, and someone adds more memory to one service. There is an emergency that necessitates a configuration update to the database. There is an update to a load-balancing policy or a direct upgrade of a dependency within production to fix a bug.
Staging does not receive every one of those changes. At first, the differences are small, but eventually a feature works in staging and fails in production.
The issue gets fixed, yet engineers trust staging a little less. They add more production checks, test things manually, avoid releasing late in the day, and become more cautious about changes that touch infrastructure.
Environment drift has now created more manual work, and that manual work creates more undocumented exceptions. The loop reinforces itself.
The Infrastructure Starts Shaping Product Decisions
This is when the repercussions begin to emerge beyond the scope of infrastructure efforts. A particular feature is delayed due to the dangerous migration process involved in its release. A number of changes have been rolled out in one shot because releases are a hassle. A particular service isn’t changed due to the uncertainty of how it works in production, or a release is scheduled for Monday since the engineer responsible for production is not available on Friday.
The team has gone beyond running the system; they are now planning around its limitations. This is when infrastructure becomes more than just an internal engineering problem.
Then the Cloud Bill Starts Looking Strange
This is the case in cloud infrastructure as well: the test environment stays after the testing is completed; a bigger server instance is allocated during a performance issue and left big; obsolete databases stay in place because no one knows what relies on them; extra storage space, backups, and redundant services silently stack up.
Now, imagine an infrastructure cost that more than doubled in a year’s time. It could very well be completely justified, or only partly justified, and the rest is due to forgotten environments, oversized servers, unused storage, and other stuff that no one dares to delete.
If the infrastructure is poorly documented, the risk of making mistakes in trying to optimize costs becomes too high. Something can look unused when in reality it is used at least once a month by some old job. A database can be redundant until there is one production script that uses it.
Deleting infrastructure requires confidence, and the team does not have that confidence. So it keeps paying. The same missing visibility that created operational risk now creates financial waste.
Manual Work Quietly Becomes Part of the Engineering Budget
Certain infrastructure costs show up in the cloud bill. Others hide in payroll.
The first developer takes 20 minutes to prepare a deployment, the second takes 30 minutes to check the logs afterward, one developer manually extends the certificate expiration date, another developer makes changes to three configurations, and one incident lasts two hours longer since the developer looking into it cannot figure out where the service is running.
None of these tasks looks expensive on its own, which is exactly why they survive. Repeated across dozens of releases and several engineers, they can consume hundreds of hours a year.
The company is paying for infrastructure debt, whether it has a line item for it or not. The payment simply appears as lost engineering capacity.
Eventually, the System Becomes Dependent on Specific People
At this point, the team is probably already familiar with who to turn to when issues arise. There is the person who understands deployments, the person who knows about the database setup, the person who knows about networking, and the person who understands what alerts mean.
This can look like expertise, and it is expertise. But if critical operations depend on a small number of people, expertise has also become a single point of failure.
The risk becomes obvious when one of them leaves. Suddenly, nobody is completely sure how a production deployment works. A script fails, and the error message means nothing to the rest of the team. A cloud resource looks obsolete, but nobody wants to remove it. An incident takes longer because the person who normally recognizes the failure pattern is gone.
Not only has the company lost an engineer, but also lost part of its operational documentation. This is when the bus factor ceases to be an engineering concept and becomes a business continuity issue.
By the Time You Fix It, It’s No Longer a Cleanup
Eventually, the company decides it is time to fix the infrastructure. At this point, the original problems sound deceptively simple:
- automate deployments;
- document the infrastructure;
- make staging match production;
- move configuration into version control;
- reduce cloud waste;
- set up proper monitoring.
Had the team done those things earlier, some might have taken days.
Now, every one of them is connected to years of accumulated decisions.
Before automating deployments, the team has to understand all the manual exceptions that have become part of the release process. Before rebuilding staging, it has to determine exactly how production differs. Before deleting cloud resources, it has to discover which undocumented scripts and services still depend on them. Before moving infrastructure into code, it has to reconstruct the infrastructure that was created manually over several years.
All of which needs to be accomplished while the production systems continue being utilized by actual users. The team can’t simply replace everything overnight. They need to do it incrementally, with testing, migration, maintenance of uptime, and product development happening at the same time.
What once took a couple of days of configuration now involves an engineering effort in itself, involving all planning, execution, contingencies, and risks that go along with that. It is not a matter of cleaning up their infrastructure but moving out of their legacy system.
The Real Cost Comes From the Chain Reaction
The dangerous part of infrastructure debt is not any individual shortcut. A manual deployment, a temporary script, missing documentation, a slightly different staging environment, or even some wasted cloud spend can all be manageable on their own.
The trouble with this is that problems don’t tend to remain contained. Manual deployments lead to ad-hoc scripts, which contain unspoken assumptions. Over time, those assumptions become unrecorded knowledge, making automation even harder and production-only changes more common.
Production-only changes lead to environment drift. The less like production the staging environment becomes, the less trust that can be put in it, the more that must be verified manually, and the more the existing knowledge in the team becomes relied on.
That dependence leaves fewer people capable of operating the infrastructure safely. Once the system reaches that point, fixing one part usually means touching several others.
The cost compounds because every shortcut makes the next shortcut easier to justify and the eventual cleanup harder to perform.
“We’ll Fix It Later” Only Works If “Later” Is Defined
None of this means teams should build perfect infrastructure from day one. A startup that spends three months building an elaborate internal platform before it has users can waste just as much time as a company that never invests in infrastructure at all.
Shortcuts are often necessary. The mistake is treating them as decisions without expiration dates.
A manual deployment may be reasonable when releases happen once every two weeks, but it becomes a problem when deployments happen several times a day. One engineer owning infrastructure may be reasonable in a three-person company, but it becomes dangerous when twenty developers depend on that person. A simplified staging environment may be acceptable during an MVP, but it becomes risky once customers expect every release to be stable.
So the useful question is not “Can we postpone this?” It is “What will tell us that this shortcut has expired?”The trigger could be deployment frequency, team size, incident rate, cloud spend, onboarding time, or the number of people capable of running production. The important part is that the trigger exists.
Without one, “later” doesn’t mean “when it is efficient to fix.” Instead, it means “when it has gotten so bad that we can’t afford to ignore it anymore.”By this point, the price is not just in engineering time. It’s also in slower deployments, operations risk, cloud costs, recovery from incidents, difficulty hiring, and the ability to scale the business without growing more scared of the infrastructure you have.
Elizaveta Sokolova
Hleb Skuratau
Juliya Moshkareva