This is Episode 187 of the Community Broadband Bits Podcast. Andrew Blum, author of Tubes: A Journey to the Center of the Internet published in 2012, joins the show to describe his tour of the infrastructure of the Internet. Listen to this episode here.
Andrew Blum: When my Internet at home broke and the guy came to fix it and said, “I think a squirrel is chewing on your Internet” and I realized that if squirrels could chew on the piece of the Internet behind my building there had to be other pieces of the Internet that squirrels can chew on.
Lisa Gonzalez: Hello and welcome to Episode 187 of the Community Broadband Bits Podcast from the Institute for Local Self-Reliance I’m Lisa Gonzalez. The Internet has become so ingrained in our lives we rarely think about its physical structure. In 2012 Andrew Blum published Tubes: A Journey to the Center of the Internet. His book brings us into that unseen world that we often forget about until something goes wrong. In this interview Andrew and Chris discuss his motivation for writing the book, some of the surprising things he learned a long the way and the unrestrained nature of the Internet itself. Here at the Institute for Local Self-Reliance we strive to bring you quality resources like the Community Broadband Bits Podcast and we do it with no advertising.
If you’re one of our listeners you appreciate the unique information we offer and you understand that there’s very few places to get this information please take a moment to donate at ilsr.org or muninetworks.org to help us continue to bring you accurate, interesting, quality information. Any amount helps. Now here’s Chris talking with Andrew Blum, author of Tubes: A Journey to the Center of the Internet.
Chris Mitchell: Welcome to another edition of the Community Broadband Bits podcast. I’m Chris Mitchell and today I’m speaking with Andrew Blum the author of Tubes" A Journey to The Center of the Internet. Welcome to the show.
Andrew Blum: Thanks for having me.
Chris Mitchell: It’s great to have you on. You and I talked around the time the book came out and I’ve really enjoyed it and I should have had you on four years ago.
Andrew Blum: That’s okay. Thank you.
Chris Mitchell: It’s great to catch up because the Internet hasn’t changed much physically I think.
Andrew Blum: No, that’s true its structure is pretty much the same I think.
Chris Mitchell: How would you describe the book just briefly for people who hadn’t heard of it? What are they going to get out of reading your book?
Andrew Blum: The book is the chronicle of my visit to the Internet, in particular I focused not just the data centers which I think are the pieces of the Internet’s infrastructure that are most familiar but the Internet exchange points. The places where networks physically connect to each other and the lines in between. The terrestrial fiber networks that are probably more obvious but also the under sea cables across the oceans that recently have come to more prominence but were certainly forgotten for a lot of years. Then I try to communicate some of why it’s important to understand what these places are and how they fit together and more importantly I think who puts them together and what companies own them and what the consequences are for those questions.
Chris Mitchell: I think for people who may not be as familiar with how the Internet works this is a great introduction. I think you have a background in architecture and an interest in how things are made. I think of it as maybe if 99% Invisible that famous podcast, if they had a book maybe this is what it would look like.
Andrew Blum: Good. Thanks, good.
Chris Mitchell: Anyway, I think you tell the story at the beginning of the book but you’re a bit of a tech savvy person but I don’t think you knew all that much about how the Internet worked. What got you interested in writing the book?
Andrew Blum: There were two main things, one macro and one micro. The macro scene it was 2008/2009 it was the beginning of the great recession. I was writing mostly for Wired of Wired Magazine mostly about big infrastructure about airports and sky scrappers and things like that. Started poking around looking at the broadband stimulus funding with this idea that if there’s this whole conversation about putting shovels in the ground. That was the big refrain of the government stimulus across all things. Shovels in the ground that was going to get the country going. The idea of shovels in the ground and broadband was not something that I had connected in my head in a long time.
As I understood it with the fiber boom in the early 2000s the Internet was built and that was that. I’d read a couple of articles about Google’s mysterious data centers but otherwise there just was no conversation about it even at Wired. Even at Wired we’d forgotten about the wires.
Chris Mitchell: I’ve noticed that hasn’t changed by the way.
Andrew Blum: Yes, although of course now it’s been subsumed by the conversation about the cloud, it’s replaced by different things and the NSA which is another topic but both also relevant. The micro thing that happened that focused me was one, it’s actually it’s a January day like today. It’s exactly seven years ago I guess when my Internet home broke. I’m a Cablevision customer in Brooklyn. The guy came to fix it and went behind my building to see what was going on and saw a squirrel running a long a wire and said … and this question changed my life, was, “I think a squirrel is chewing on your Internet.” I realized that if squirrels could chew on the piece of the Internet behind my building there had to be other pieces of the Internet that squirrels can chew on.
There had to be something physical there and just developed this framework of yanking the cable from the wall and just trying to figure out where it went with the idea that I had to go some place. I remember very early on talking to my editor again we were tech savvy, we thought we knew how the Internet worked. We thought it was your messages were broken up into parts and then reassembled. I’d realized that in the idea of information traveling over many paths across the Internet we had began to believe it didn’t travel over any paths at all in saying that there were many. We assumed that meant really there were none when in fact that wasn’t true at all. There had to be a physical path and probably fewer physical paths and certainly I expected when I started the project.
Chris Mitchell: That’s one of the things that I found interesting was that you found the Internet was more centralized than you had expected from what you’d learned about over the years. I wanted to say squirrels and sharks, right?
Andrew Blum: Yeah.
Chris Mitchell: It’s like the two pieces.
Andrew Blum: The sharks are a bit apocryphal, the squirrels are real. The squirrels have their own Twitter handle now, you know Cyber Squirrel?
Chris Mitchell: No, I had not seen that. I have to check that out.
Andrew Blum: Yeah, it’s the squirrel liberation army, it’s actually quite good.
Chris Mitchell: I guess what I’m curious about then is how you describe it to people in terms of how does the Internet work?
Andrew Blum: The last mile is the piece most people are familiar with because it’s where they pay for it. They pay their cable company, they interact with the cable repair person, they’re aware of their speed or their cellular or cable or DSL or maybe fiber. The part that I think is more surprising to people are the spaces in between networks and the connections between networks of the Internet, the interim Internet. There’s the basic idea that the Internet is a network of networks and those networks have to meet somewhere. They have to be connected by someone. Once that connection is made yes things do run automatically but it has to physically made and configured by hand.
It turns out that the places where those connections are made, where networks connect one to another are relatively few and far between. That they’re pretty specialized. They have a certain scale of geography where you just don’t want them in every city. You may not even want one of these places in every state. They’re really regional places where networks connect to each other that we call Internet exchange points. Carrier hotel is another word for it but that doesn’t quite capture the more vivid and deliberate and chaotic activity that goes on in the big Internet exchange points where lots of networks are connecting to each other. I think it’s worth also acknowledging that the word network is a bit fuzzy.
A network could be something like Google or Facebook of YouTube or I should Amazon or any of our big favorites but a network could also be a small regional ISP or a network could even be a very large law firm or basically anyone that runs their own AS meaning autonomous system. Anyone who declares themselves and independent network on the Internet. For the most part we live inside the Internet, the networks of our ISPs but more and more you do see large businesses and certainly for a long time universities and government agencies all acting as autonomous networks on the Internet and managing their connections between each other really by hand.
Chris Mitchell: I think one of the interesting tugs over the history of this is that whether we’re becoming more centralized or more decentralized because I worry as Comcast has grown from being a small cable company before anyone knew about the Internet to what it is today. It’s taken over many … what formally had been autonomous systems and made it into effectively one, the Comcast System. We’ve seen the same thing with … we had once what? Almost 10,000 Clax, these local ISPs and almost all of them were run out of business. At the same time we do see these other corporate networks coming out and developing the autonomous system, getting their own numbers effectively. The thing for me that I find so amazing is how parts of these are very formal and parts of it are really informal.
You talk about how these are very important networks and how they trade information in these Internet exchange points which are often just abbreviated the IXPs if people have seen that term around. One of the things I find amazing is you talk about some of those big routers, they’ve got 160 ports and then you juxtapose it I think with just the handshake agreements that basically make them come alive.
Andrew Blum: I saw that in the time that I spent inside the NANOG community, NANOG being North American Network Operators Group. I think it’s actually in the last couple of years has become more formalize, the stakes have gotten a little bit bigger. Until very recently it was about the peering coordinator of one network having a beer and trusting and trusting from a technical standpoint basically from a geek standpoint. Trusting the technical capability of their counterpart at another network and then recognizing that they each had a business case for connecting to each other. Sometimes that business case would be sweetened by one becoming the customer of another but otherwise when we say we have lots of eyeballs.
We’ve got lots of one IXP, we’ve lots of customers who need your content. You’re content provider, you need to get your content to customers. Hey we both have a presence at one of a handful of Internet exchange points why don’t we basically string a cable from my cage and my router to your cage and your router? For which we’ll probably have to pay a fee to our landlord and pay for the cost of configuring it. Once that happens the Internet is ours. My network is connected to your sand we don’t have to pay anybody anything to move our traffic around if we’re in those places.
Chris Mitchell: One of the things that I find really interesting is how that gives you a certain kind of control that you wouldn’t otherwise have. If we imagine a three network setup where you and I want to connect and we’re currently maybe connected through the third network that of the listener but we don’t have any control. If we send something through the listener’s network maybe it takes a while to get you. Maybe I have to re-transmit half of the packets because there’s a quality issue or just packets are being dropped from congestion. If you and I interconnect which we can basically do by just having two machines in the same building and then we can work everything out between up we have total control. The thing that also kills me on top of that is if we have a great idea for a business we’re on the Internet, right?
Andrew Blum: Right.
Chris Mitchell: We didn’t have to get permission from Comcast or others. At a certain point if we’re Netflix then Comcast could try and shake us down but the freedom there is remarkable.
Andrew Blum: It is and I was amazed that among the network engineers they were constrained by their business overlords in different ways. They were all private companies, there’s no such thing as a public company on the Internet. It’s a good rule of thumb with very few exceptions. You all have private companies all trying to make their money in different ways. If they recognize that there’s a need, that it’s worthwhile for them to exchange their traffic then they’re going to do it. If it’s worth spending some cost of buying the hardware and other things like that, that can happen in all sorts of surprising ways. Again, as long as the physical infrastructure is there, as long as there places available that are neutral where you don’t have other ISPs.
Other big networks breathing down people’s throats and trying to extract their cut of things then there is the freedom for things to go a little crazy and for the networks of the Internet to connect in ways you might not expect.
Chris Mitchell: Do you think the lack of knowledge that I think most people have and that you probably had before you wrote the book, does that impact how we deal with policy, how we try to solve this problem of getting more people, higher quality Internet connectivity?
Andrew Blum: Undoubtedly. It comes up most clearly in the different net neutrality conversations. It always amazed me that until very recently interconnection was not a part of that net neutrality conversation. The assumption was it’s all about the last miles, it’s all about what your cable company is and isn’t doing. With no sense ... particularly as bandwidth has grown mostly from video and demand has grown so much, with no sense of the other bottlenecks that might exist in the system. Not between you ISP and your cable box but maybe between your ISP and Netflix or Amazon, whoever it is. Recently we’ve become more aware of some of those bottlenecks but it just wasn’t … Again, in Tim Wu’s book about net neutrality the word peering and interconnection does not appear.
Chris Mitchell: In the Master switch?
Andrew Blum: In the Master Switch, yes. Then earlier in his book with Jack Goldsmith about net neutrality it just doesn’t come up. It wasn’t part of the conversation it was about the last mile. I think that the dominance of that and the quietness of the peering community had set us back a certain number of years in the network policy conversation. I will say that I think that’s changed dramatically in the last three/four years. I think it’s really started to enter the conversation. Then you see things get more public and like with Comcast and Netflix or now that Tim Wu is working for the Attorney General now you see he’s demanded the paperwork behind the peering agreements, that should be exciting to see. That’s going to make a difference I think.
Chris Mitchell: There’s these broad trends in both the left and the right, there’s subsections of the left and the right that deeply care about decentralizing. I think the people who built the Internet had some of those on both the left and the right and they really cared about there’s decentralized technology. In recent years we’ve had this more centralizing influence both from government and from these large companies like Comcast. I think to some extent people maybe didn’t see interconnection being a problem because they didn’t really think that when you squeeze the balloon Comcast is going to find a way of popping out elsewhere and creating those issues. In part I’m just wondering, what have you seen in the IXP space because when you say there might only be one in the city and that might be enough I’m curious if that is enough.
Andrew Blum: One other piece with the consolidation and the concerns with that is not just the consolidation, the big companies, it’s the lack of transparency. I think for me that’s become really important. I’m on the Internet history list-serv that’s run out of … I guess maybe it’s out of Berkeley. I don’t know how big it is but it’s a weird thing because whenever somebody asks a question about Internet history like all of the self-declared founding fathers of the Internet write back right away. There was recently a bit of a stir about the series of articles that Ingrid Burrington did in The Atlantic about Internet infrastructure. Her refrain being, “I was shut out, I was shut out. I had to search out these places, they’re secret, they’re secret, they’re secret.”
The response in the list was, “She doesn’t know what she’s talking about and that’s false” it completely missed the forests for the trees with that the point is that we don’t know where these places are. The point is that these places are obscured. The point is that if you’re a founding father of the Internet and you work for Google you’re not helping the case of Internet history if there’s a culture of secrecy around all Internet infrastructure. I just want to really point out that I do think the lack of transparency is as if not more important than the consolidation among the networks.
Chris Mitchell: To some of the extent I always wonder if some of that is an issue of just natural, “Well I know where it is and the people that associate with know where it is so it’s not a secret.” Then there’s also a sense of, “I don’t really want other people to know where it is because my source of power is this unique knowledge.” It’s like being a knight and I know things that other people don’t know or being a part of a secret order. Do you think there’s some of that as well?
Andrew Blum: We can talk all day about the psychology of the network. I think that these businesses have gotten so big now and they’re lucrative for their employees. There’s the sense that it has important strategic advantage to keep all these number of secret. I’m talking about at the biggest level so the Amazons and the Googles and things like that. It’s such a small pool of people who deal with these interconnection agreements and they all go from company to company to company and I’ve seen that over the last five/six years. They rotate through the same lists of companies. There’s a lot at stake for them personally.
I don’t blame them but I do think that we start with a long time where nobody was asking the questions and now we have a lot of people are asking questions about how this happens. How these networks of together and where data centers are and why they’re there and what’s in them and all of that. We still are facing uphill struggle to get the companies to open up about what’s inside of them and what happens.
Chris Mitchell: By the companies you mean basically any of these companies that are moving large bits of traffic that are involved in the interconnection agreement.
Andrew Blum: Yes.
Chris Mitchell: Those are the companies you’re talking about?
Andrew Blum: Absolutely. Again, if Tim Wu’s demanded, this assistant attorney general we hear more about this. This is the need to open up these issues in order to frankly increase the freedom of the Internet I think is really important. This is a little bit controversial but for a long time the net neutrality and the Internet freedom conversations have been dominated by some of these big companies. I think that’s been a bit of a drag on the conversation. It’s the consolidation not only of the network themselves but the consolidation of the intellectual space around these things. It’s one reason the community network piece has always been such a thrill for me because it’s exactly the glorious idea that anyone could be a network on the Internet if we can just build it.
For me the key counter to all of the pressures of lack of transparency, the consolidation is the friction of many networks. As long as one network doesn’t get too big you have more freedom on the Internet. I think that’s a physical corollary to a lot of the policy conversations that we talk about.
Chris Mitchell: That’s where I want to get back to a comment that you were making in terms of the number of IXPs one might have. I just think here in the Twin cities Minneapolis-Saint Paul we have one and like so many of them it’s named by its address which makes it easier to find somewhat but at the same time it’s one. It strikes me that there would be a danger of someone who could take ownership of it or in some ways spoil it so that it would be hostile to new comers perhaps. I would like to see many IXPs in one of my thoughts. I’m curious if there’s a trade-off to that or how I might think about it.
Andrew Blum: Well, there are examples of a change in ownership causing a constriction of opportunities interconnection. The most notable would be the Google building in New York, 111 8th Avenue. Whey bought it in 2010 I guess it is it was the largest real estate transaction of the year in America, $2 something billion. It wasn’t acknowledged publicly that the interconnection piece was a very important part of that deal. They said they wanted it for the office space. They have indeed filled it up with office space but they have by all accounts had a chilling effect on that as an exchange point. Although they’re such a good customers so many of the big networks kind of shuffles out. That’s an example of a major exchange point losing its neutrality, a very prominent example.
There’s been a lot of consolidation among the ISP operators, Equinix has been growing remarkably, Telx was just bought by Digital Realty Trust. Before that Equinix had bought Switch and Data, there’s a shrinking of the number of players in that space. I don’t have a sense of what kind of chilling effect if any that has. I know for example at 350 East Cermak in Chicago the biggest Internet exchange point in the middle of the country and now in the coast, the building itself was owned by Digital Realty and Telx operated the meet me room. The places where actually the networks in the building connected to each other, the patch panel. Now that Digital Realty owns Telx wholesale you lose that tension between the meet me room and the building itself.
Equinix is also a customer in that building although that situation replicate at other places. For me as there’s vertical integration between the Internet exchange operators and the data center space that begins to call into question the whole model. I don’t know how that’s playing out yet, it’s all pretty new I’d say. I’m not following it as well as I used to and that stuff it’s more backroom stuff that I’m not as preview to if I ever was.
Chris Mitchell: Well, let’s get back into some of the work you did and the conversations you had as a result of it. I’m always curious, do you have any memorable reactions when you were describing the actual geography of the Internet to people? I just imagine a number of people thinking, well that just seems way more immature-ish than I imagined, right?
Andrew Blum: Yeah.
Chris Mitchell: It’s Wild West and we discuss ways in which it’s less so today but it’s still remarkably for the thing that’s sweeping and changing all commerce on the face of the planet. It’s remarkable.
Andrew Blum: The thing that comes up in every single conversation, every single event is how do we destroy the Internet? Or if we talk about these places won’t somebody blow them up? I used to joke when I was talking about the book, the topic the law, basically daily you guys think now is the part of my day where we talk about destroying the Internet like this happens every single day. For me it was that people are surprised that it’s apparent vulnerability, it was actually a strength. You can’t protect it by hiding it, you have to protect it by acknowledging it. There are plenty of places that are targets for destruction but are very public. In fact most major targets for destruction, God forbid, are very public that why they’re targets of destruction.
Chris Mitchell: Yeah, and because of the number of people.
Andrew Blum: Yeah, the number of people and the symbolic prominence. The Internet has always lacked symbols of infrastructure so it hasn’t had that piece. I’d like to see that change, it’d be fun if these major interconnection points were celebrated as such and certainly they’re becoming more public. I’ve always dreamt about Internet brewery tours where you get to go see a data center, plug in and I think we’re getting a little bit closer to that. The most interesting exchanges I’ve had from people over the last couple of years have been with the reporters who have been following these Snowden story and working with the Snowden documents and have this amazing cash of documents and that they’re trying to line up with and understanding of the Internet’s infrastructure.
It was surprising to me because it indicated less the capability or the reach of the NSA and more the lack of understanding on anyone’s part about how this all fits together. What’s really going on from a technical standpoint and how extensive and overreaching or not different kinds of government tapping was. I think that’s the anarchicness of Internet. I think is reassuring for anyone critical of government tapping because it gives the Internet a bit of a Wild West. It doesn’t make it seem like it’s just an AT&T operated network and I think opens things up again. Again, the free Internet is the diverse Internet.
Chris Mitchell: I want to go back to an area that I really loved about your book which is talking about municipal network and my bread and butter. You go back, it’s really quite amazing the role of government at all levels of government in this in terms of the great dams of the North West that were built by the federal government providing low cost electricity for data centers. Then in this case tell me about The Dalles in Oregon.
Andrew Blum: The Dalles was one of my favorite trips and visits in my journey to the Internet. The Dalles is the town on the Columbia River and the border between Oregon and Washington just on the other side of the Columbia River Gorge from Portland. It’s this infrastructural pinch point because it’s where the railroad goes through, it’s where the interstate goes through. Then not coincidentally at all it’s where Google built its first ground up data center in secret, not secret for very long. I think a real watershed moment in the history of the Internet’s infrastructure was the front page story in The New York Times speculating about why Google was building a ‘factory’ in this place. What did this mean? Was the introduction to these giant data centers. I think that was 2006 that article.
One other major questions I had was why there? What about this place? Is it cheap power? Is it cooling? It turned out a big piece of the equation was the municipal fiber network there known as QLife that had been built by the town in 2002 and on a bit of a whim because they could and increasingly depressed industrial community into a famous landmark on the Internet. It was many years before anyone acknowledged that Google was in town. There was no sign on the building I think until about 2012 or 2013. The building was smudged out of Google’s online maps in an astonishing example of those who make the maps having the power. All that has changed and there’s a lot more acknowledgement of The Dalles as an important Internet infrastructure hub for the region really.
Chris Mitchell: I think some of the stuff we’re seeing now is that we’re seeing I think more data centers moving toward cooling areas, right?
Andrew Blum: Yeah.
Chris Mitchell: Like you get the potential for Iceland and some of the Scandinavian countries maybe if I understand correctly for the cooling cost.
Andrew Blum: Yeah, the cooling cost and then the balance of latency, that delay in transferring data and the kind of now we all have … now Facebook it doesn’t just need to be close. Now we all have our Facebook back pages, our deep storage. I think there’s the recognition in a lot of these companies that you can go further away and with a little bit fitter connections from a data standpoint for some of your cloud need rather than an impulse a few years ago that everything had to be in the region in order to be whatever. X number of milliseconds away without any kind of delay. That transition seems to be happening.
Chris Mitchell: Let me ask you one final question as we wrap up and that’s if you had to direct people to go to one geographic place where the Internet is to check it out and do a tour what might it be?
Andrew Blum: Actually as a tourist? Like where if you wanted to go look at things, that you go see it or where spiritually would be the …
Chris Mitchell: I think it’s an open question because you and went an actually saw the landing sites of the Transcontinental Cables, I think you visited in Amsterdam, didn’t you? The IXP there which is one of the largest in the world.
Andrew Blum: Yeah.
Chris Mitchell: Where would you direct people if they could go and check it out? Let’s just say hypothetically if they could have a higher level of access than just a tourist.
Andrew Blum: Hypothetically if you get a higher level of access than just a tourist I do think one of the most satisfying spots on the Internet tourist route is the 9th Floor of 60 Hudson Street in New York. That’s where the meet me room, the interconnection point that Telx operates is there. [inaudible 00:30:40] one of the major transit points between Europe and North America. It’s been spread out for security reasons a little bit but that still is the harder things. It’s an old Western Union telegraph building so you have the sense of one’s infrastructures in place it doesn’t move, once networks go to where networks are. For me it it’s as much of a hinge in the physical infrastructure of the Internet as anywhere.
The Facebook data center in Prineville, Oregon is also a pretty great spot. It’s a good looking building and it acknowledges that it is a monument rather than trying to just look like nothing and hide, look like a penitentiary. The data centers there are bigger than you would expect and in some ways more dramatic that you would expect. Both buildings you can just drive up to, both buildings actually are not impossible to get into at least if you’re in the business. I think Telx is happy to show off the fact that they are … brag about and show off this is an important Internet meeting place particularly if you’re a network administrator who might be a customer.
Chris Mitchell: Great. Well thank you so much for taking the time to come on and share with us some of your experiences.
Andrew Blum: Thanks for having me. It’s exciting to reach out to the municipal fiber community as well. It’s such a key part of things.
Chris Mitchell: Yeah, I think as we see these models develop I keep hoping this will be the beginning of something major where many more cities will have a municipal network as one of the options of the overlapping networks that we need.
Andrew Blum: We’ll say it has to happen and I really hope it does.
Chris Mitchell: Well, thank you so much.
Andrew Blum: Thanks Chris.
Lisa Gonzalez: That was Chris and Andrew Blum the author of Tubes: A Journey to the Center of the Internet. We highly recommend his book. Go to your independent book seller, not Amazon, and pick it up if you don’t already have a copy. Send us your ideas for future broadcasts, email us at firstname.lastname@example.org. You can follow Chris on Twitter, his handle is @communitynets. You can also follow muninetworks.org stories on Twitter, the handle is @muninetworks.org. Thank you Kathleen Martin for the new music Player vs. Player licensed through Creative Commons. Thank you for listening to Episode 187 of the Community Broadband Bits Podcast.