Thanks to Jeff Hoel for providing the transcript for episode 97 of the Community Broadband Bits podcast with Jack Belcher and Lou Michael on community network in Arlington, Virginia. Listen to this episode here.
Jack Belcher: They just want to know if they can get access to that high speed, and they can get it in a secure manner, and they can get to places not only in the county but elsewhere.
Lisa Gonzalez: Hello. You are listening to the Community Broadband Bits Podcast, from the Institute for Local Self-Reliance. This is Lisa Gonzalez.
Recently, Arlington County, Virginia, announced that it would begin offering dark fiber services to local businesses through its ConnectArlington project. The county expects to make the service available by 2015. In this episode, Chris talks with Jack Belcher, CIO of Arlington County, and also Lou Michael, the project's Chief Architect. The county began to develop its own Inet just a few years ago, rather than renew a franchise agreement with Comcast. Since then, Arlington County has taken advantage of every opportunity to install conduit and fiber. As a result, they now have a network that serves public facilities, public safety, and soon will be serving local businesses with dark fiber. In this conversation, Chris, Jack, and Lou discuss Arlington's careful planning toward their long-term goal. This local community realized that a community network was not only a way to improve quality-of-life but a necessary infrastructure. They achieved their goal, and the network is already paying off. Here Chris, Jack, and Lou.
Chris Mitchell: Welcome to another edition of the Community Broadband Bits Podcast. I'm Chris Mitchell. And today I'm speaking with Jack Belcher, the Chief Information Officer for Arlington County, Virginia. Welcome to the show.
Jack Belcher: Thank you. Thanks for inviting us.
Chris: We also have a couple other people in the room. We've got Lou Michael, the Chief Architect of the system. Welcome.
Lou Michael: Thank you.
Chris: And some other folks who may be chipping in, and we'll get their names if they do. I'm really excited to get a better sense of this project, just outside our capital. But let's start with learning a little bit about Arlington. How would you describe the county to people who are not familiar with it?
Jack: Right across the river from Washington, DC. We're the home of the Pentagon. And we're the home of Reagan National Airport. And the home of Arlington Cemetery. And so many of the folks who lead the federal government live in Arlington, Virginia. It's a very educated community, a very young community. I think the last estimate we saw, probably 35-40 percent of the community is under the age of 35. So it's unusual for local governments that have that breakdown. Very technology-aware. And our economic health is quite good, largely because of some pretty good planning. We had the Met Subway built through Arlington, and we have many stops here -- I think we have eleven stops that are in Arlington itself. And that whole corridor of subways, has allowed us to attract businesses that are in the corridor and also to provide for residential and other uses in the close proximity. So it's a quite educated community, technology-aware community, a community that's highly expectant of doing some difficult things. And we usually do those things.
Chris: Do you have any rural areas, or is it entirely urbanized?
Jack: There's really no rural area, actually. A lot of green space. With 26 square miles. The Cemetery and the Pentagon take up a portion of that. So we're probably the smallest county -- geographically -- in the United States. We have a population of about 220,000 people that live here, and probably a daytime population in the neighborhood of 400,000 -- that come in and work and **.
Chris: And so, we have this interesting project that -- it's a great name: ConnectArlington. What is this project accomplishing?
Jack: Well, it's **. Really, we've benefited from a broadband fiber network ** from a television franchise, Comcast. And we were able to leverage that to provide for data, voice, and video communications between the county schools and government buildings -- about 93 buildings across the county. And that had immense value for a number of years. We did not only voice traffic -- in other words, rather than having to dial a phone number -- 10-digit phone number -- we can dial a 4-digit phone number and connect. So that saved our money there. High-speed data, broadband communications. And video, as we became aware -- came available, we started utilizing that across the network.
What happened is that the cable providers came to us and said, you know what, our franchise is coming to an end, and we do not want to be in the business of providing you an institutional fiber network. And so if you DO want it, we'll charge you for it; but we'll run it, we'll manage it, we'll provide you with a speed that, you know, meets what you're paying.
The county decided that that was not in the interest of us. We wanted the ability to have our own network, to be able to control the speed, the bandwidth, to manage the security, and such. And we decided we would invest in the building of our own fiber network. And so, we've been doing that for about three years. We've got about 48-49 percent of the buildings up and running right now, and we plan to complete in the next year and a half -- complete the entire ring for the public-education-government purposes of that.
So, as we were doing it, we built enough capacity into the network that we could add redundancy and scalability over time. Because we believe that the demand for bandwidth is going to increase rapidly, as you're seeing now, with all the devices that are out there, the needs for communications and such, we felt that it would be necessary.
So we put four conduits into the ground for this capability -- four two-inch conduits, two of which we filled with fiber -- 144 strands in each conduit, for the public-education-government uses. What we decided to do was to take one of those fallow conduits and populate that with additional fiber. And the reason for that -- we thought that -- we believe -- that this could inspire more economic and community development in our community. And so, we are in the process now of building that. We hope to have that ready to go -- a core network for that -- by January 2015.
Chris: When you were deciding to build this network -- and, actually, to put it in the ground -- one of the things I found interesting was that you didn't really just dig up the ground only to put in this fiber, I think. You coupled it with some other projects. Some of them was fiber base, like the intelligent transportation system. But I think others was working with some other entities that wanted to get fiber in the ground, right? Tell us just a little bit about how you went, basically, keeping the cost down by getting more bang for the buck when you were tearing up the streets.
Jack: Right. A perfect example is the intelligent transportation system, where we got federal grants to be able to refresh the copper wiring that managed our traffic signals in the county. And they were putting fiber in. And we said, while you're putting conduit in for that fiber, I mean, why not allow us to put additional conduit in the ground to run fiber for other purposes? So we were able to leverage that investment. Because the big investment -- the cost -- is the digging up the ground.
And then we looked for every other opportunity where there would be ground open that we could utilize that and provide other service. We needed to be able to ensure better communications for our first responders. Right now, they use something called the 800-MHz radio system. But that's subject to the ** of wind, weather, leaves, and such. And so what we wanted to do there was backhaul those radio towers that are across the county, broadcasting over the air signal, and have a resilient capability. And so we were going to lay fiber to backhaul all those broadcast towers. And so we're doing that as well.
Our electric company in Virginia, Dominion Electric, wanted -- had to upgrade the power grid. And so they began digging in the ground, and laying fiber. And we asked if we could lay fiber as well. And so we rode their opportunity to provide the dropped fiber in the ground.
And then we looked at the building that was taking place. And if you come to Arlington County, you'll see a huge amount of commercial building taking place. Well, every time the commercial building takes place, they open up the ground. And we made a requirement that we could lay fiber in the ground to support that as well.
So, we looked for every and any opportunity, where the ground was opened up, where we could lay fiber. And so, now, the result is that we have a ring that stretches almost 22 miles, through the core of the county, that we're making available for ** and economic development purposes. We have fiber now that stretches across the entire county. And I think that's going to be a significant value going forward.
Chris: Was there any sort of challenge, in terms of trying to design a network that would both accomplish your needs and also fit with, you know, where Dominion happened to be laying fiber, or where the traffic lights were? I mean, it's not always true that where you have the opportunity to get fiber is necessarily where you most want to have it.
Jack: That's true. And so, there were times, I think, where we have had to seek opportunities, where we leveraged other funds we had to connect areas that were not on the routes that, say, the traffic system was going -- or the power capability was. And the reason for that is, we wanted to create a ring -- a physical ring and a logical ring -- so that if there's a cut taking place anywhere in the architecture, it homes in a different direction.
Lou: Back to your first point, about not always needing it. We have a master plan that we look at, where we think something is going to come up. And so you prioritize any work -- you try to buttress that master plan. You know, something might move up on the calendar because of the open ground. We think there's a very unlikely chance that there's a need for a path can be done. We still might put conduit in, because putting conduit in is very inexpensive. And it facilitates future fiber. So, a lot of times, along a property line or right-of-way, or something like that, where construction is going on, even if we didn't put fiber in on that path, we would go ahead and put in conduit. But the idea is that, as Jack mentioned, it's the resiliency -- it's the redundancy. Does it build another route that would make the whole system more reliable?
So, in one case, Dominion was running down a road where we absolutely -- we had a school at the end of the road. And -- boy, oh, boy -- it helped us speed up that section of work. They had another section -- it was a little bit longer section -- and it's an area we never planned to pull fiber, because we didn't have any buildings or traffic signals that needed to be connected along that route. But when we looked at the map, we saw that it was a great expressway -- a bypass -- that would give the whole system better resiliency. And so, we went ahead and did it there also. So, it's about planning and prioritizing.
Jack: The nature of the county is small. Very few areas where we can't reach. And if we had to go to it, it wouldn't be far. And new development takes place. And redevelopment takes place. We have a large area -- a commercial area -- that was developed 30 years ago, next to the airport. Well, that's going through a transition. The government agencies that are there are moving out because of different restrictions and requirements. And we're going to redevelop that area. Well, the fiber we laid, actually goes through a lot of that area. And where it doesn't, we're going to extend it. And it's going to create, again, a base you can build upon. So, we've been quite fortunate, in terms of geography.
Lou: If you think of an urban area, and all the traffic signals. And -- Jack was talking about that redundancy -- every traffic cabinet that is on this system, the connection is equivalent to a building. So the traffic cabinet dual-home. It homes to two hub sites. Those two hub sites are dual-home to two data centers. And then the data centers have an independent separate backbone interconnecting them.
So there's a very few number of feet. If you're in an intersection or something, it might be a few feet -- usually less than a quarter of a mile -- where the path is shared. But quickly, you go on these two divergent paths. Well, what we've ended up with, now, is that it's pretty hard, anywhere in our **, to be more than a quarter to a half a mile from one of these backbones.
Chris: As a result of the multiple rings that you've built, have you seen greater reliability that you previously had?
Lou: I am an old IT guy. And so I do not like to talk about reliability. Because as soon as you say something is reliable, it will fail. But I will say that we are pleased with the outcome.
Chris: Excellent. We'll take that. Aside from redundancy, let's say, what are some of the benefits the schools have seen by having these connections available to them now -- the fiber paths that the county owns, versus the solution they previously had?
Lou: Well, the previous solution, the Inet provided by the cable franchise, was predominantly an aerial network. And so it was highly susceptible to a wide variety of unfortunate circumstances. And, we think, from a truck that was too tall, to a hanging wire, to a ** that came in and took out a bunch of tress. And the new system, ninety-eight point something underground, which then makes it more reliable.
Jack: Well, it's really going to have value -- in that if we didn't do this, we would have to rely on procuring bandwidth from the ISP. And the ISPs are more than happy -- their business model is, they're more than happy to give you bandwidth, but, as you know, at more cost. We are of the opinion that the business model the ISPs have is quantity rather than quality of service. So if you look around our county, and look at the businesses in the county, and see what they're getting for bandwidth, they're happy with 150 megabits of bandwidth if they can get that. And at high expense. And that's not true bandwidth. That'a bandwidth you're sharing with to the folks. And we looked around and we said, that's not what we should be providing. We should be providing far more than that, and it should be based on quality. The idea of putting this in the ground is -- we are of the belief that the abundance of fiber will do a couple of things. One is, it will drive down price -- to access to communications capabilities. And also, increase the speed -- by which it's available. Now, that's an assumption -- yet to be proved. But, frankly, I think that's what's going to happen. I mean, you hear about Google Fiber. You say, that's great -- that's a gigabit in speed. But that's managed services that Google is providing. What the schools would have faced -- what we would have faced, if we hadn't done this, is that they would have had a network, and they would have had speeds which they now totally occupied, at a very attractive rate. Right? A very attractive rate that Comcast, who's our provider, would have provided for them. But we are of the belief that that would only be good for a few -- for a short period of time. And that the schools would come back and say, you know what, we're introducing a new approach where we're putting mobile devices in kids' hands, we're putting content on the devices, we want to deal with -- to communicate with their teachers and such. And then the bandwidth requirements are going to start going up. As soon as they start going up, the provider's going to say, great, we'd be glad to help you, here's the cost we’re going to -- here's what it will cost you to get it.
And then you add -- the other thing from a public safety standpoint -- is -- we really don't want to have sensitive information managed by someone from the -- other than the government. We want to be sure we know what information are being transferred around, and we don't want other people telling us, don't worry, we're securing it for you. We want to make sure that WE'RE securing that information.
Chris: Right. You're part of the NCR net, right? Is it the National Capital Region Network?
Chris: Is this a -- So this is a part of that, where you take it very seriously, and it's very much locked down, and you know exactly who has the keys to be able to get into the network, right?
Jack: Yes, that's exactly it.
Chris: With September 11 hitting right in your home there, what kind of lessons did you take away from that?
Jack: 9-11 took place. The plane, actually, went into the server farm at the Pentagon. It really took out a lot of the communications at the Pentagon. In order for us to be able to provide connectivity back to the -- our people who were responding to the event, our Fire Chief, Jim Schwartz, was actually the commander of the response team at the Pentagon. In other words, he directed the response of not only the local government but state and federal -- how do we address this issue. We had no communications there. So when ** for us to get communications, we actually sent a team up to the tallest building we had, which is our courthouse. And we had that team take a router antenna, and point it at the Pentagon. At the same time, we sent down a Parks & Recreation team to take one of the cherry-pickers they use to change lights on the -- on football stadiums -- and basically what they did, they sent a balloon up. And then we triangulated the position. And what we did is -- we sent the signal from the top of the building down to the parking lot of the Pentagon. And when we got there, we then we ran cable -- coaxial cable -- across the Pentagon floor to where the command center was for our response to that event.
The only reason that took place is because, a week earlier, I had someone from AT&T come in and say, you know, we're getting this YAGI antenna, and it can do these things -- point-to-point communications. And if you ever need it, let us know. So, when we were down there at the Pentagon, and we can't get communications, I picked up the phone, because I still had the card of the guy who called, and I said -- I got a hold of him at eleven o'clock at night -- in his office, I can remember that vividly -- sitting there saying, do you still have that equipment? And he said yeah. And so we sent a team out, they picked it up. They drove by to the local Home Depot, walked into the Home Depot, and said, we need this, this, and this. And to show you what happened that day -- and it's an amazing -- Americans at their best -- Home Depot said, what are you doing? What are you trying to do? What do you need? They went around, they got what they needed, they came back, they came to the cash register, and the guy at the Home Depot, the manager, said, take it, go, it's for you guys. So our guys went out of there, no payment at all, put up the stuff, and then ran a communications down there.
Well, that was quite fortunate. And then, what we learned from that, we got to have more communications in different places. And so what we did is -- we started going to the concept of command vehicles. So the police got a command vehicle. Fire got a command vehicle. And then, we took an old police bus and made it into a technology support vehicle. So that when the fire vehicle went out, and the police vehicle went out, we could then communicate. The police named their vehicle, on the radio, Blue Thunder. That was their thing. Fire named theirs Red October. And we named ours Gray Wolf. And we actually got a CIO 100 award a few years ago for putting these vehicles in the field.
But that's not a good way to do communications. Vehicles going to a place and setting up a command village worked. But the problem is, now you're relying on vehicles you've got to support. So what we've done with the network we've put in is -- we're thankful to John Bayliss, who is the head of ConnectArlington -- we put in something called public safety ports, at every traffic signal. John, do you want to tell them what that's all about?
John Bayliss: Yes. So each traffic cabinet -- if you look at a traffic cabinet -- it has a smaller bay on one side of it that the police can actually get into, to put it on flasher, in case there's an accident. So, to give everybody access to that, inside the cabinet, where there's delicate electronics, that actually control the lights, we put an Ethernet board in that's tied back to our fiber and switch in the cabinet. So, that Ethernet board has the network in it that they need to do exercises with the other command vehicles -- the police, and the fire, and then they also have bomb squad vehicles and other vehicles that they use, for the purpose of making -- they're on the network. We have cameras extended off of them. And they can control them from our command post, or our ECC, which is beside our facility here in Courthouse Plaza. So they get that high-speed, dedicated, secure network, at the traffic cabinet level.
Jack: We just pull up, plug in a USB port, you're on the network and you can broadcast out across the network.
Something that came out of this, we found -- I mentioned, we found this 800-MHz radio system, which backhauled. And it's really the common way of interfacing across the country for all public safety communications. Everybody's gone to this 800-MHz system. That was developed 20 years ago. And it's dated. And so -- all it was was voice. And so what they're doing now, the federal government, they're moving towards the broadband network. They call it FirstNet. And that's going to be based on that 700-MHz sphere, where they basically will have high-quality broadband communications. You don't only do video, you do voice.... I mean, you not only do voice, you do video and data as well.
So, in other words, the first responder goes in the building, they're able to be equipped with a video capability that can be broadcast back and say, where is the first responder? Where are they going? Where do they need to go? And also data capability. So, here's where they are in the building, here's what the health is of the first responder. So that offers great capability.
So what we're doing in a parallel way is, we're asking any new building being built in Arlington to rethink how we do in-building communications -- Not so much to support better resiliency of the 800-MHz, which is the primary reason, but also to lay a foundation for the next network that's going to come in, so that we're more proactively monitoring that health of these communications capabilities in the building.
Put that in real life. There was a horrific event that took place in Washington not too long ago. The Navy Annex. Basically, what happened is, we had a shooter come in and do all kinds of things. Well, the first responders weren't able to communicate in that building. And the reason for that was -- and we're finding across the board -- is, buildings that are being built today, they're energy-efficient. They're green. They have low-emission windows on them.
Chris: They're unfriendly to radio waves.
Jack: You got it. And so -- And, frankly, we don't know they're unfriendly until people go into the building. What if Lou's having an event? And a first responder comes in, and we've got to figure out what the problem is. Well, he's in radio communications with our 9-1-1 center. Well, if he loses communications, he goes and looks at Lou, then walks back out to get communications to figure out what to do, then walks back in to apply something, then walks back out to see if that's working. All that time, Lou is in great risk. And so, we want the capability to use this fiber optic backbone we're putting it, to be able to connect to the building, so that we can see, OK, there's a problem here.
And the problem, by the way, may not be just green buildings. It might be that they've decided that they're putting in all kinds of routers, to be able to provide for wireless communication within that building -- for iPads, wireless devices, cell phones, and such. And those are causing conflict. And so we're doing this now, across the county, to make sure that we have that capability in place.
We had a situation in one of our own buildings -- a gym, a community center -- Arlington Mill, it's called -- and we put in this system. And we found that the day we put it in, that night, we began getting communications issues. And it turns out that what was happening is, the cleaning crew that was coming in wanted to be able to communicate with each other, so they'd gone to Walmart and gotten those little radios, you know, that you push to talk. And those were causing communications issues. However, they system we had in place was 1) unable to alert us that that was happening, and 2) to increase the frequency so we could get above that. We knew there was a problem. But that all emerged from that 9-11 experience: that we have to have more ubiquitous, pervasive communications across the county. And they've got to be simple enough that when an event takes place, you don't have to take out a manual to know how to do it. You've already got the communications in place, and it works.
Chris: Well, you know, I have to say that we appreciate that sort of thing, because when you make it all work transparently, you'll never get the credit for it, because nobody notices. So, I do appreciate both you doing it and offering the explanation.
But I'd like to -- I think I want to finish up by just going back, briefly, to the business services that you're looking to deploy with the dark fiber. And I'm curious if you have a sense -- I mean, being in such a high-tech region, do you have a sense that a lot of businesses are going to be using the dark fiber for themselves, or do you anticipate other providers leasing the dark fiber from you and then using that to connect customers?
Jack: I think it's both. In the state of Virginia, we can't be in a business -- we can't be in a business of marketing and selling those services -- information services. So what we're going to do -- the model we're working out -- and it's still in an evolution -- is to hire a contractor, a third party, who's acting like a broker. And what they're going to do is -- they're going to enter into licenses and sub-licenses for people to utilize that dark fiber. We believe there are going to be many businesses that are going to say, we just want that access. There are going to be other businesses that are going to say, I don't know nothing about this. Can you tell me how to do this, and provide that service to us?
But we've gone around, now, and we've talked to every major builder in the community -- and talked about this. We've talked to the government institutions that are here. We've talked to higher education. There's a huge market for this. Our assumptions are being proven right, that there's a need. The government is coming to us, and we're surprised by this. They're saying that we need more -- we need MORE capacity. We need divergent paths -- we need redundant paths. We're finding businesses that are -- small business -- that are saying, it doesn't work for me -- getting 100 megabits a second doesn't work. I need high-speed broadband access. And there's an absence of knowledge about how to make that happen. You know, we in the technology world know how to get that -- you know, how to bring an ISP in, how to get that dark fiber, how to make the **. But somebody that is running a business is not really ** who needs technology to make it happen. They just want to know whether they can get access to that high-speed, and they can get it in a secure manner, and they can get to places, not only in the county but elsewhere. We believe there's a market out there. But what would it mean? The county can't be in the business of doing retail. So what we're going to try to do is, we're going to own the middle mile -- the core -- and we'll have a third party who will be negotiating a way and facilitating access -- the last mile -- from, say, a building to the core, or from a floor in a building to the core. We want it to be that simple and easy, so people say, I need to get access to this high-speed access -- I need to be able to store this in a secure manner, somewhere else. And I need it to be scalable. How do I do it all? I need to communicate with my branch office somewhere else across the country. How do I do that? And we have that.
Why we're doing this, too, is -- it changes the value equation in Arlington. By having that here, Arlington is a great place to live -- and work. And we're finding that businesses realize that. So, there are businesses who may be somewhere else in the region who have much more space -- footprint -- but -- and pay at a cheaper rate. But when offered the opportunity to come into a place that has access to that broadband capability -- fiber capability -- we're hearing that they're willing to say, you know what, we're going to sacrifice how much space we have, and pay a higher rent, so we can attract a different type of worker into our environment. And, also, we have access to those people who are driving the business and driving the government.
Chris: Excellent. It somewhat reminds me of some of the stories that we saw in Santa Monica, as well. I just did a case study about a network that they built -- often by coupling investments with intelligent transportation and that sort of thing. And they saw the same thing, where they see, you know, people paying more to be in Santa Monica than being a few blocks away, because they have made this fiber access available. So, it's a similar story that we see in many places.
Lou: The fiber -- the resource we're offering in a non-discriminatory fashion. So, your traditional carriers -- your existing ISP -- they also have limited resources. And they'll be welcome to make use of this too.
Jack: The ISPs, right now, we haven't found if they utilize that. And we think they will. The era of putting a cell tower up in an urban community is over. That's not the business model. The business model is to put the in-building base station in, that washes the entire building with the communications ability they want, but then backhaul that someplace else -- where the antenna ought to be able to collect that signal. And we think that reaction to that broadband, in an affordable manner -- it's going to make what we're trying to do here attractive to them as well.
Chris: Thank you for coming on the show.
Jack: Hey, good talking with you. Thank you guys.
Lisa: If you visit arlingtonva.us, and follow the technology services link in the list of departments, you'll find your way to the ConnectArlington page.
We want your ideas for the show. Please e-mail us: email@example.com. You can follow us on Twitter. Our handle is @communitynets. This show was released on May 6th, 2014. We want to thank Valley Lodge for their song, "Sweet Elizabeth," licensed using Creative Commons. And thank you for listening.