7 challenges in resilient System Design—and how to overcome them

As one of the engineers behind the predecessor to Microsoft Azure, I can attest that there are few skills more valuable than learning to build systems for the future.

Systems are never built to last — but to adapt. Every product launch, whether a small app or a large-scale platform, starts with limited resources. The goal is to ship quickly, meet immediate needs, and outpace competitors.

When success comes faster than anticipated, today’s solutions can become tomorrow’s bottlenecks. A system that started for 1000 users might work brilliantly for 10,000 — but will crumble when the number reaches millions.

X (formerly known as Twitter), is one such example of how systems designed for simplicity can falter under the weight of their success.

Failure is an opportunity to learn, improve, and evolve if handled properly. Just as Twitter overcame their failure, so can other large-scale systems — the key is to make your system resilient to failures.

Resilient systems must handle failure gracefully, scale effortlessly, and evolve seamlessly for the future — and they are the foundation for businesses to grow and maintain customer trust.

Today we'll discuss how to make systems adapt, survive, and thrive despite failures, growth, and technological evolution — and the top challenges you’ll face along the way.

Let’s start!

Why we must plan for the future#

Why do we need to plan for the future?

Let's look at another real-world example: Netflix.

Netflix began as a simple DVD rental service with a monolithic system that efficiently handled user orders, inventory, and shipping within predictable bounds.

However, when Netflix shifted to streaming videos in 2007, the system faced significant challenges:

• The monolithic architecture and reliance on in-house data centers proved inadequate for the sudden surge in demand.

• Outages became common during the early days.

How Netflix upgraded its infrastructure is a story for another day, but the happy ending was that it did — and if Netflix hadn't recovered quickly enough, it wouldn't exist today.

Long-term resilience is essential for any system's prolonged success.

Through long-term resilience, we ensure:

• Business continuity: Uninterrupted operations as systems evolve.

• Minimizing downtime: Upholding system availability by anticipating potential failures and implementing measures accordingly.

• Protection against evolving cyber threats: Mitigating security risks by implementing robust security plans and measures.

• Customer satisfaction: Providing the best possible user experience by delivering consistent performance and reliability.

• Data integrity: Resilient systems are built to regularly back up data and employ recovery mechanisms to protect data loss and preserve data integrity.

Let’s explore the challenges we must tackle to design a decade-long resilient system.

7 challenges in making resilient systems#

Designing resilient systems is a challenging feat. Let’s understand what it takes to build one:

1. Planning for uncertainty#

Building for the future isn’t just about speed — it’s about planning for what’s next.

The rise of generative artificial intelligence (GenAI) fully evolved users’ expectations and, hence, businesses’ needs. Foreseeing this and designing accordingly was almost impossible.

Users’ expectations and business needs evolve rapidly, making it difficult to anticipate what a system might need to sustain itself.

Some hard truths are this:

• Systems built for today often struggle for tomorrow. Netflix learned this the hard way—its monolithic setup worked fine for DVDs but crumbled under the streaming demands.

• Businesses often rush to ship products quickly to grab market share. While this is great for growth, it can leave systems unprepared for the long haul.

Long-term resilience is deceptively simple — at the end of the day, it requires planning for an unpredictable future.

2. Adapting to emerging technology#

Technology evolves rapidly. What feels cutting-edge today might be outdated in just a few years.

Hardware and software eventually become outdated. Companies that built their systems decades ago, without adapting to new technology, now struggle to find developers to maintain them. Similarly, companies using monolithic architectures often face challenges in achieving resilience because they lack the flexibility to implement a layered, easily updatable approach — something that microservices architectures naturally support.

Protocols and tools evolve. If your system can’t adapt, you’ll be left behind (think of the companies shifting to HTTP/3). Technologies like blockchain and AI were never quite expected 20 years ago, yet they've redefined so many of our systems today (and continue to do so).

When its old data centers couldn’t keep up with the streaming demands, Netflix faced this head-on. Instead of patching things up, they moved to the cloud. That decision wasn’t just about fixing a problem but embracing change and setting themselves up for the future.

As designers, we have to remain aware of emerging technologies and ready to adapt our systems to integrate them.

3. Scalability challenges#

Scalability isn’t about scaling for high demand on day one. It’s about enabling it to scale and handle demands whenever needed.

But anticipating demand is anything but easy. In some cases, user demand can explode overnight, and downtime is inevitable if your system isn’t ready.

Twitter’s “Fail Whale” became a symbol of how rapid growth can crash unprepared systems.

Here’s the tricky thing:

• Being unprepared for demand can leave your system scrambling when traffic spikes.

• Meanwhile, overengineering for growth is expensive (and you may never see that growth).

Scalability isn’t about anticipating growth perfectly; it’s about being prepared to scale at any time, and at any scale.

4. Security challenges#

As systems get more complex, integrating with third-party services becomes the norm. However, these integrations open new surfaces to attack.

In 2017, a data breach at Equifax resulted in the loss of the personal data of over 140 million customers—with an estimated loss of $1.4 billion. The cause? An unpatched vulnerability in a third-party integrated service, which cybersecurity professionals had neglected.

Threats continuously evolve, hackers get smarter, and today’s defenses never suffice for tomorrow’s threats. With AI, cybersecurity threats are becoming more sophisticated and exploiting new vulnerabilities faster than ever. Meanwhile, compliance such as GDPRGeneral Data Protection Regulation (GDPR) is a European Union regulation that governs data privacy and protection of individuals within the EU and the European Economic Area (EEA)., HIPAAHealth Insurance Portability and Accountability Act (HIPAA) is a US law that regulates the handling of sensitive health information (PHI) to protect patient privacy., etc., are continuously evolving with the threats to tackle users’ privacy concerns.

In building for the future, security isn’t something you can set and forget.

5. Operation resilience#

Failures are inevitable, whether they're from hardware or software failures, human errors, or misconfigurations.

Making a comprehensive plan for handling these failures gracefully is challenging because it requires:

• Anticipating a wide range of failure scenarios

• Implementing layered redundancies

• Coordinating automated and human responses to minimize disruption

As systems grow and evolve, so do the undetectable spots. Monitoring needs to adapt constantly to detect anomalies before they escalate.

Meanwhile, recovering from natural or technical disasters requires robust mechanisms to avoid prolonged downtime or data loss.

In 2018, during the Amazon Prime Day outage, Amazon lost over $100 million in sales during the downtime—an estimated $1.2 million per minute.

Resilience isn’t about avoiding failure—it’s about being ready to handle it without breaking stride.

6. Cost optimization#

Cost optimization is a balancing act, and knowing when to make the right trade-offs is essential to making a resilient system.

Resilience isn’t cheap, and finding the sweet spot is tough:

• Spending too much upfront is risky, especially if you're wasting resources on hypotheticals.

• Meanwhile, spending too little at the start can leave you scrambling with sudden expenses when things break or demand spikes.

Startups often see spending as a luxury because they’re focused on short-term survival, while larger companies have more resources to justify ongoing costs for upgrades and maintenance.

7. Organizational challenges#

Remember COBOL? Companies still relying on COBOL are unable to find enough developers today.

You can't have a resilient system without people to build and maintain them.

With current trends, people are less likely to stay in the same role or company for the next decade. Team members are changing or leaving roles more frequently. As a result, we get gaps in knowledge transfer, making it harder for system to evolve across generations.

Solving these challenges isn’t just about processes; it’s about fostering a culture that values continuity and collaboration.

Solutions to overcome challenges#

Resilience isn’t just a one-time thing; it’s built, iterated, and refined over time. The challenges we explored can feel overwhelming, but the good news is that these hurdles are manageable with the right strategies, allowing us to set the foundation for systems that thrive long after deployment.

Let’s look at some System Design principles that will help us to this end.

System Design for resilient systems#

Let’s explore these principles that will guide the creation of resilient systems:

Design for failure#

Assume things will break and systems will fail.

Hardware, software, networks, or human processes can all break down.

To ensure smooth failure recovery, look to:

• Redundancy

• Failover mechanisms

• Building self-healing into architecture

Layer-wise resilience approach#

Build resilience into every layer of the system individually.

By making each layer capable of withstanding failures on its own, you create a multi-layered defense that strengthens the system as a whole.

Scalability and microservices architecture#

Avoid premature optimization — but design with scalability in mind.

To prepare for the future, you can:

• Use modular architecture such as microservices with loosely coupled components (easier to build layers-wise resilience

• Auto-scaling capabilities

• Sharding

Monitoring#

Monitoring is key to resilient systems, as it requires real-time insights into their health to detect and address issues before they escalate.

Whether you are a start-up or an established giant, build robust monitoring and logging from day one.

Task automation#

Automate everything to avoid human errors.

This practice reduces errors and speeds up recovery.

However, oversight and a pre-defined recovery plan are still needed, as automation can also fail the system, which is rare but has happened with Facebook.

Security#

Security is the cornerstone of resilience. A zero-trust policy within the system and integrating third-party applications are musts.

You can protect the system from ever-evolving threats by implementing:

• Authorized access

• Encryption for data in transit and at rest

• Secure APIs

• Regular audits

Disaster recovery#

We must implement well-tested disaster recovery plans by regularly simulating failures, e.g., chaos engineering, to ensure the system can handle real-world disasters effectively.

Cost optimization#

We should plan cost optimization strategies upfront by identifying and prioritizing the system’s features based on their importance and resource cost.

Focus on sustaining critical features even under heavy load, while less essential features can be throttled or turned off during high-demand periods.

To balance the system’s cost and efficiency, we can use:

• Cloud resources

• Serverless architectures

• Auto-scaling

Adaptable System Design#

Build the system with the expectation that after every X years, the system will need to adapt to new technologies, tools, business needs, etc. Make it ready to evolve when needed.

Knowledge sharing#

Resilient teams are needed to build resilient systems.

Document everything, including designs, decisions, and operational playbooks, with enough details that a new person could easily understand, update, and maintain the system if needed.

By embedding these principles into your System Design, you can create a robust, scalable, and adaptable foundation to the unpredictable challenges of the future.

How to evaluate the resilience of systems#

How can we assess whether the system is resilient?

We can evaluate a system under certain parameters — a clear framework of key performance indicators (KPIs). By continuously assessing these metrics, we ensure the system is resilient and evolves to meet the challenges.

Let’s explore the KPIs for evaluation:

• Mean time between failures (MTBF): The average time between failures. A higher MTBF indicates greater stability and fewer disruptions, ensuring long-term reliability.

• Mean time to recover (MTTR): The average time between system failures. A lower MTTR reflects a system’s ability to quickly restore functionalities after an incident.

• Mean time to acknowledge (MTTA): How quickly teams detect and acknowledge issues. A shorter MTTA ensures faster incident response.

The above metrics — and others, like mean time to respond, mean time to resolve, etc. — ensure the system’s reliability.

But we can also look to KPIs that relate to resilience, like:

• Availability: Measuring the percentage of time a service or product is accessible and performing intended operations under normal conditions. Availability is measured with the number of nines as shown below:

• Scalability: By simulating a real-world high-spike traffic scenario and testing how the system performs under heavy load.

These KPIs provide a strong foundation for evaluating resilience while keeping the focus on the most impactful metrics for long-term success.

We can regularly learn from small failures to prevent large ones and evolve the system to withstand such failures. Moreover, constant improvement and adaptation can help us make the system more resilient.

Getting hands-on experience#

Now that you know how to plan for the challenges of long-term resiliency, it's time to get hands-on with designing modular, scalable, resilient real-world applications.

Here are 3 courses I recommend to get your hands dirty with resiliency in System Design:

1. Grokking the Modern System Design Interview, which will help you brush up on the core principles and practices for building robust systems.

2. Grokking the Principles and Practices of Advanced System Design for a working knowledge of building large-scale distributed systems and cloud service providers.

3. Grokking the Generative AI System Design for a forward-looking course on integrating Generative AI into existing architectures in a resilient manner.

Happy learning!