As hundreds of thousands of households worldwide tapped into Netflix’s Squid Game last month, viewers may have taken something fairly extraordinary for granted. Netflix didn’t buckle under the unprecedented demand for the dystopian drama that would become its most successful title to date — even as other services have struggled to keep their products sturdy under less demanding circumstances.

When many of us fire up our favorite streaming services, we often bump into various fury-making problems: stuff freezes, controls don’t work, or the service crashes entirely. None of these are ideal, but all seem to have become a widely understood cost of cord-cutting. For example, Disney Plus crashed its very first day because its software couldn’t handle the demand (and then it buckled again under demand for WandaVision). HBO Max is so fundamentally broken that its own leadership has admitted that the app is a mess. Even Instagram, whose Stories feature makes it a kind of streaming service in its own right, crashes so frequently it’s started alerting its users when it’s borked. Streaming can be maddening!

A service’s guts, the engineering behind the app itself, are the foundation of any streamer’s success, and Netflix has spent the last 10 years building out an expansive server network called Open Connect in order to avoid many modern streaming headaches. It’s the thing that’s allowed Netflix to serve up a far more reliable experience than its competitors and not falter when some 111 million users tuned in to Squid Game during its earliest weeks on the service.“ANYONE WHO WANTS TO IMPROVE PERFORMANCE IS GOING TO TRY TO PUT A SERVER AS CLOSE TO THE END USER AS POSSIBLE.”

“One of the reasons why Netflix is the leader in this market and has the number of subs they do […] is something that pretty much everybody outside of the technical part of this industry underestimates, and that is Open Connect,” Dan Rayburn, a media streaming expert and principal analyst with Frost & Sullivan, tells The Verge. “How many times has Netflix had a problem with their streaming service over the last 10 years?”

Certainly not as many as HBO Max, that’s for sure.

Open Connect was created because Netflix “knew that we needed to build some level of infrastructure technology that would sustain the anticipated traffic that we knew success would look like,” Gina Haspilaire, Netflix’s vice president of Open Connect, tells me. “We felt we were going to be successful, and we knew that the internet at the time was not built to sustain the level of traffic that would be required globally.”

Nobody wants to sit down to watch a movie only to have their app crash or buffer for an eternity. What Netflix had the foresight to understand was that if it was going to maintain a certain level of quality, it would have to build a distribution system itself.

POPPING THE HOOD ON OPEN CONNECT

Open Connect is Netflix’s in-house content distribution network specifically built to deliver its TV shows and movies. Started in 2012, the program involves Netflix giving internet service providers physical appliances that allow them to localize traffic. These appliances store copies of Netflix content to create less strain on networks by eliminating the number of channels that content has to pass through to reach the user trying to play it.

Most major streaming services rely on third-party content delivery networks (CDNs) to pass along their videos, which is why Netflix’s server network is so unique. Without a system like Open Connect or a third-party CDN in place, a request for content by an ISP has to “go through a peering point and maybe transit four or five other networks until it gets to the origin, or the place that holds the content,” Will Law, chief architect of media engineering at Akamai, a major content delivery network, tells The Verge. Not only does that slow down delivery, but it’s expensive since ISPs may have to pay to access that content.

To avoid the traffic and fees, Netflix ships copies of its content to its own servers ahead of time. That also helps to prevent Netflix traffic from choking network demand during peak hours of streaming.

“We, Open Connect, bring a copy of Bridgerton at the closest point to your internet service provider — in some cases, right inside your internet service provider’s network — and that basically avoids the burden of the internet service provider having to go get it and transfer it through all these servers on the internet over to you,” Haspilaire tells The Verge.

And they’re everywhere. At present, Netflix says it has 17,000 servers spread across 158 countries, and the company tells me it plans to continue expanding its content delivery network. Netflix prioritizes where it places these servers based on where it has the most members and relationships with ISPs, the company says.

“Anyone who wants to improve performance is going to try to put a server as close to the end user as possible,” Law explains. “And in putting it there and by serving the content from that last mile network, it stops the traffic having to transit all the rest of the internet and go back to an origin. So it’s taking a load off the internet, and it’s taking a load off the peering points.”

When Open Connect originally launched a decade ago, the service started working collaboratively with ISPs on deployment. Netflix provides ISPs with the servers for free, and Netflix has an internal reliability team that works with ISP resources to maintain the servers. The benefit to ISPs, according to both Netflix and Akamai, is fewer costs to ISPs by alleviating the need for them to have to fetch copies of content themselves.

“It’s not a huge burden, but it’s certainly a relief,” Law tells me. “It’s the same principle that Akamai is founded on and the same principle that every CDN works on. Netflix’s CDN is no different to other CDNs — except their CDN is dedicated to Netflix content.”

While most major third-party CDNs do multiple jobs and manage multiple requests from many companies — Akamai, for example, says it has thousands of customers — Netflix’s internal CDN does exactly one job: it distributes Netflix content. If a content distributor doesn’t have this kind of CDN partnership or server network in place, Law says, there’s a whole lot of stuff that needs to happen along the way for you to stream a movie or TV show.“THE REASON THAT NETFLIX HAD TO BUILD A CDN IS BECAUSE AMERICA’S ISPS ARE GARBAGE.”

While Netflix doesn’t disclose how much it costs the company to build and maintain these servers, Netflix says it’s invested roughly $1 billion in Open Connect since its creation a decade ago. It’s dumping mountains of money into the CDN because a premium and user-engaged streaming experience is core to Netflix’s entire business strategy. Its whole subscription model, for example, is based in part around what quality of video streaming a user wants for their content.

Netflix also has to account for America’s internet infrastructure being fundamentally fractured and broken.

“The reason that Netflix had to build a CDN is because America’s ISPs are garbage,” Electronic Frontier Foundation’s Katharine Trendacosta tells The Verge. “And what they knew was that their customers don’t want an endlessly buffering screen or degraded quality.”

Now, not every ISP allows Netflix’s hardware in. The Verge spoke with an AT&T executive who confirmed that it still sells Netflix optimal network connections to the streamer rather than having Netflix install physical devices in its data centers. When asked about how this arrangement and others work alongside its Open Connect program, Netflix said that it views “our relationships with ISPs globally as adaptable.” A company spokesperson said the arrangement may differ based on what an ISP’s network supports, and Netflix will find other connection points to bring its titles closer to the viewer instead.

WHY IT WORKS

The thing that Netflix is most concerned with is delivering a good viewing experience regardless of how bad your ISP is.

To do that, Netflix effectively ships three copies of each of its titles to its servers, each with a different level of quality. If your ISP is overwhelmed or your internet momentarily kicks out, the system can swap in a lower-bitrate version of the title, helping you maintain the stream without interruption.

“We will adapt the content to the quality of the network and not vice versa,” Haspilaire says. “That’s the reason why you don’t see when your network has a blip — your streaming stays constant. Because over time, we’re able to adjust the version … so when your internet goes in and out, you won’t get buffering from us.”

So, why three copies? As Trendacosta noted, the internet as we experience it is entirely unreliable. Outages, poor wifi connections, and any other network interruptions might impact your ability to access the internet the way you need to. Netflix can work around many of these network problems through its collaboration with ISPs.“NOBODY IS GOING TO DISAGREE OR ARGUE WITH NETFLIX THAT THEY’VE BUILT A SYSTEM THAT WORKS AT SCALE.”

Haspilaire says Netflix pre-places this content during off-peak hours so it’s not competing with other internet traffic that would occur during high-use streaming times. As far as what content it places and where, Netflix says it anticipates what’s going to be popular and sends its content to servers accordingly.

“Not only are we placing content on all of these servers around the world, but we are pre-placing them based on what is popular. Because we predict what is popular, we’re able to put it as close as possible under the correct server,” Haspilaire says. “This pre-placement of our films and shows allows us, based on prime time viewing hours, to store 100 percent of our catalog locally. And that basically eliminates the whole risk associated with service disruption.”

Netflix then shuffles the videos around its servers based on what it expects to get the most attention. Open Connect has two types of servers: flash, which handles faster delivery, and storage, which holds up to 350 TB of data. If something sitting in storage becomes popular, Netflix will move that title onto the flash server.

“The flash server is designed to serve a larger portion of the traffic, so as demand increases for a show or film, our OCA is built to be dynamic by transitioning it from storage to flash to meet that demand,” the company tells me.

Hundreds of millions of people huddling in their homes and looking for entertaining distractions was a perfect litmus test for this 10-year-old project. “The pandemic really kind of tested our infrastructure, or our technology, in such a way that it was not built for,” Haspilaire says. Open Connect helped future-proof Netflix against that increased demand.

KING OF STREAMING

Open Connect is one of the biggest behind-the-scenes drivers behind Netflix’s ability to perform as well as it did during the pandemic. But there’s a lot of other moving parts that put Netflix far ahead of its competitors. Rayburn points to Netflix’s video and audio encoding initiatives as one such example, though its user experience is another big one. Even if Netflix had a “huge advantage,” as Rayburn says, over competitors with its decade-long lead, you’ve got to have a rock-solid product to grow and retain a subscriber base.

“Nobody is going to disagree or argue with Netflix that they’ve built a system that works at scale, and that is the biggest thing that is missed in our industry,” Rayburn says. “You only get that number of subs if you can deliver good, quality consumer experience at scale. Nobody’s ever had the scale to the degree Netflix has. Nobody has that expertise.”

In other words, users have to actually enjoy the stuff that’s built on top of Netflix’s infrastructure, as well. Beyond just ensuring that streams don’t crash and burn, the consideration that goes into the development of Netflix’s various features are often overlooked simply because we don’t have to think about them — they just work. But Netflix is constantly working on improving its UI.

“We don’t make an assumption that one size fits all. When you think about design, we could just design the product for the western audience, but that doesn’t necessarily mean that it works in Korea or Japan,” says Steve Johnson, Netflix’s vice president of product and studio design. “So we have to think about the nuances and the particulars that happen in those different countries.”

As other services continue to grow, it’s possible and even likely that major streamers will be looking to Netflix’s infrastructure and top-to-bottom business strategy for guidance. With any luck, maybe that’ll even fix the universal nightmare of streaming content virtually anywhere else.