Transcript: Community Broadband Bits Episode 26

Thank to Jeff Hoel for providing the transcript for the episode 26 of the Community Broadband Bits podcast with Josh Wallace on the municipal utility dark fiber business in the city of Palo Alto, California. Listen to this episode here.

00:15:

Lisa Gonzales:  Hello, and welcome to the Community Broadband Bits podcast, brought to you by the Institute for Local Self-Reliance.  This is Lisa Gonzales.

In our 26th episode, we talk to Josh Wallace.  He works for Fiber Optics Business Development for the city of Palo Alto, California.  Josh and Christopher discuss the community's extensive fiber network, and how the city uses dark fiber to offer reliable, high-capacity infrastructure to local businesses.

Josh and Chris review how the city became involved in the business of dark fiber, and Josh describes the resulting commercial relationships, and the process.

Here are Chris and Josh.

00:50:

Chris Mitchell:  Josh Wallace, thank you for coming on Community Broadband Bits.

00:53:

Josh Wallace:  Thank you very much.  I appreciate your inviting me, Chris.

00:56:

Chris:  So I've known a little bit about Palo Alto.  I know that you have a dark fiber network.  I've worked with -- or, I've talked with some people from Palo Alto previously, as you've considered making other investments.  But let's start at the very beginning and get a sense of why -- how you came to have a dark fiber network.

01:13:

Josh:  Well, in the late '90s, there was a question about whether or not we would be able to renew some of our electric contracts in an optimal way we had originally had in the late '50s.  So the big search for revenue.  And it was thought that maybe we would have to shore up some revenue from some renewed contracts that might not be quite as good as we hoped.

So, you know, we looked around, and we realized that at the same time that we could connect all of our SCADA system -- our electric network control system -- with fiber optics, we could actually jump into the fiber optics business.

So essentially our business involves licensing to customers in the Palo Alto area a surplus fiber optic telecommunications system.  It takes a couple of fiber strands to connect the SCADA system.  So we said why not put in hundreds of strands and license them to local businesses?

02:14:

Chris:  Was it very expensive to put in all those extra strands?

02:17:

Josh:  Two million dollars, to build the entire network.  And to put in the extra strands -- to answer your question directly -- no.  The big expense is in digging the trenches to create the network and stringing the poles.  The actual materials, it's just a marginal upcharge to go from a few strands to 288.

02:36:

Chris:  OK.  So you had this opportunity, or this requirement, that you had to build this network.  And you built a much larger network than you needed.  Why would you do such a thing?

02:46:

Josh:  Because we saw that telecommunications is going the way of fiber optics, and that the copper systems that were in the ground, run by the private sector, were really not sufficient for a technologically innovative, Internet-based town and business culture.

03:05:

Chris:  And so how exactly does this work?  You're a municipal utility, and you have this dark fiber.  Maybe you can walk me through the process of -- I'm a business coming to town.  What do I do?

03:16:

Josh:  What you would do is get in touch with, well, me, or one of my staff.  And let me know where you are.  And then I can click on the Autocad and -- it's a basemap that has all of the properties in Palo Alto -- all of the utility infrastructure.

And I should say right now that Palo Alto is unique in California in that we own all of our own utilities -- water, gas, wastewater, electric, and telecommunications -- which is the fiber optics business.   So we have lots of choices for infrastructure that we can use.

You call me up.  I see that you're at 123 Main Street.  And I say hmmm.  Well, we do have fiber there; let's just fire it up.  Or, gee, we need to build out to that, and we're going to ask you to pay for that installation.  And, you know, we'll go through the steps to get you installed.

Let's be clear at this point that we are a dark fiber network.  So what that means is that we essentially provide a fiber optic pipe from your building to the Palo Alto Internet Exchange -- or PAIX.  And Palo Alto Internet Exchange is a hosting company that hosts over 80 Internet service providers.  So you get to choose, as a customer, the best deal for your business from, well, any one of those 80 providers.  Basically anybody you know in the Internet carrier -- or ISP -- business has a presence as PAIX.  So we build you a pipe from your company over to our backbone, and then over to the Internet Exchange -- PAIX.  And then we cross-connect you from our cage at PAIX over to your carrier.

And we charge for the piece that we provide.  And you make a separate deal with the Internet service provider.  And you provide your own electronics.

So it's real hands-off for us, which is really a good model for a city.  Because, you know, we don't want to be in the critical customer support business -- you know, with the electronics, and guaranteeing up-time, and having all that liability.  Our system is rock-solid.  It is a fiber optic pipe, in a town where we own the streets.  So it's a great model.

05:25:

Chris:  Oftentimes these dark fiber networks seem to be 20-year agreements.  I take it you have a shorter or different take on this.

05:33:

Josh:  Yes, well, what we do -- we try and model our fiber optic service after the electric service.  So it's not going to be a big surprise to the customers.  We do ask our customers to sign a limited-liability agreement, saying that if the fiber should go down -- and, by the way, it really doesn't.  I can say in my years of experience here, it's rock-solid -- way over the five-nines reliability.  We say to the customer, if -- should the fiber go down, we will pay you for the time that you went down, but we will not pay you for any resulting damages.  So if you're E-Trade, we're not going to owe you ten million dollars for the hour you were down.  Which is just logical, and everybody seems to agree to that.  So it's not a problem.  That's never been a stumbling block.

06:22:

Chris:  Are these typically monthly contracts?  Or yearly?  Is there like a minimum term?

06:26:

Josh:  Yeah, we go one year -- we put one year up there.  But it's somewhat decorative.  Because what we do is -- we charge for the installation on our side up-front.  So we're very conservative.  We charge for the installation up-front.  So, you know, we tell you that, as a small business, it's going to cost you, let's say, two or three thousand dollars to get installed -- just picking a number.  You know, you pay that up-front.  So, well, we cannot lose any money.  If you understand -- that we're not amortizing our installation over your service period.  We get it all up-front, so we're safe.

07:01:

Chris:  Is there a common yearly cost on an ongoing basis?

07:06

Josh:  Yes.  'Cause we charge by the month.  And our minimum is $635 -- for our part.  $635.  Then, there are going to be additional charges, then, for whatever your electronics are, and whatever the Internet Exchange charges, basically, for the cross-connect fee, going from our equipment over to your carrier -- your ISP -- that's about $300 -- I'm just throwing out some rough numbers.  And then there's Internet service ...

07:33:

Chris:  Um hum.

07:33:

Josh:  ... which is a wide, wide range of costs.  I mean, if you're buying a gigabit -- if your buying ten megabits -- Now, if you're buying it for five years -- if you're buying it for 6 months --  You know, it's just all over the place.  Basically, Internet bandwidth is a commodity at this point.  So all kinds of ways of getting a good deal on that.

07:54

Chris:  The way this arrangement works, then, if I'm using your dark fiber and I'm doing 10 megabits a second, I'm probably not being very smart, because it would cost me the same from Palo Alto to do a gigabit as it would 10 megabits.

08:10:

Josh:  Absolutely.

08:10:

Chris:  There would be a difference in costs for the ISP that I connect with.

08:15:

Josh:  Yes, absolutely.  But you make a good point -- in that we are a flat fee.  And fiber optics, just by the nature of it -- it's light -- there is a zillion bandwidth there.  We are never the bottleneck.  Some R&D shops have run 13 terabytes over a single strand of fiber optics.  And that's a hard number to get your arms around.  That's a huge amount of capacity.

08:42:

Chris:  So, then, one of the questions I had was -- If I'm a small business, what expertise do I -- Do I have to have the expertise of knowing what electronics to terminate with?  And something along those lines?  Or is there someone in Palo Alto, or nearby, that would help me to solve that problem?

09:01:

Josh:  Typically, there's an IT guy, or a computer guy, at every company.  Because if somebody's going to come to us looking for that -- for it -- they have that type of service, and realizing that it's far-and-above superior -- if I may say so -- to, well, the commercial offerings.  There's somebody who kinda knows what's going on.  So I really haven't run into anybody who says I can't do this because I don't understand it.  Or I can't do that because we don't have that kind of staff.  It's actually VERY simple.  These are components -- the electronics at either end -- to be found at, basically, electronics shops.  I don't want to say Best Buy.  But if you get to go to -- we have Fry's Electronics out here -- that, you know, they'll have some routers, they'll have -- you know, anybody that has a consumer-level -- consumer and pro-sumer level telecom gear -- pretty easy to put together.  We have people who really are not experts in the IT field running our service that never had a problem.

10:02:

Chris:  Palo Alto has an incredible reputation for very intelligent people.  I'm curious -- do you have many residents that have tried to connect to your network?

10:12

Josh:  Yes.  The answer is -- There are lots and lots of very savvy residents who would like to have this going to their home.  Unfortunately, we have determined that the business model for fiber-to-the-home is suboptimal.  Especially in Palo Alto, where we have two or three of the national providers here, who have the dominant market share and are extremely hostile to upstart places like Palo Alto trying to start their own network.  You know, we run the risk of, you know, legal action -- or being tied up in court.  Predatory pricing.  And what not.  It's a very volatile business.  You know, it's also a  volume business -- fiber-to-the-home stuff.  Or just residential telecommunications.  Insomuch as we would have to engage a television provider, an Internet service provider, a telephone, video, and Internet -- triple play -- very costly, hard to amortize over a reasonably small town.  Whereas the national incumbents are amortizing over -- basically the lower 48 in the United States.

11:24:

Chris:  Right.  I fully -- we fully understand all those difficulties and challenges.  What I had meant to ask -- and I'm glad you answered that question, because that was the question I was going to ask next -- but I'm curious if you have extended the dark fiber network to any residents who are very ambitious or, you know, have very high needs?

11:43:

Josh:  Yes.  The answer is yes.  But they are treated as commercial customers.  We do not have a residential pricing tier.

11:49:

Chris:  Right.  That's about what I would have expected.  So, how has the presence of this network impacted your local job scene?  Have you had more businesses coming into town because of this?

12:01:

Josh:  Well, you ask the tough question.  And it's hard to quantify economic development based on an indirect influence such as a robust telecommunication system.  We do consider having this network to be an added value to doing business in Palo Alto.  We have two or three local ISPs and several very large companies that say we cannot do without this.  We have one that likes to be talked about.  We have -- anyway -- a veil of secrecy for others, to respect their privacy.  But with VMWare, for example, virtualization company based in Palo Alto, before they consolidated into a huge contiguous campus, had eleven offices in Palo Alto, spread over the town.  And they connected all of their buildings with our network.  And this is a major R&D shop.  So they are running gigabits -- tens of gigabits -- over our fiber, to connect their buildings -- and send all of their R&D information to the additional buildings, as well as just supporting their business.  So they love us.  I'd like to think that's one of dozens and dozens and dozens of companies that feel the same way.

13:30:

Chris:  Can you share the number of customers that you have?

13:33:

Josh:  We have about 60 at this point.  Several of those customers are resellers -- take our fiber and add the electronics and add the bandwidth, and create a turnkey for the end user.  So we have hundreds and hundreds of connections.

13:51:

Chris:  That seems, actually, fairly similar to what Santa Monica has.  I think they just crossed the 70 threshold.  And, you know, they do dark fiber and some lit services now.

14:02:

Josh:  Right, but to be sure, that's 70 customers, with hundreds and hundreds of connections...

14:06:

Chris:  Right.

14:07:

Josh:  ... because a particular reseller may have a hundred of connections.

14:11:

Chris:  Right.  Now -- And I guess the other question is: Is there a type of business that is in particular?  I mean, is it mostly IT companies that have been interested in this?  Or is it a broad range of companies?

14:22:

Josh:  It's a broad range -- and anywhere from R&D companies -- you know, software, biotech, space industry, legal.  I characterize our customers as those needing Internet connectivity as a tent pole of their business.  And the lawyers like it for the security.  Everybody likes it for the reliability.  The R&D folks like it for the speed.  We have two- or three-man -- two- or three-employee -- multimedia shops that use our fiber.  And we have 10,000-employee software companies that use our fiber.   So it's not really based on size, but the big companies all have it.  So.  Yeah.  We're not exclusive -- it's not like, oh, you're too small for us -- it's do you need the telecom service that we provide?  If so, let's do it.

15:21:

Chris:  OK.  Is there anything else we should know about this network?

15:25:

Josh:  Yeah.  We are essentially in the economic development business at this point.  We are at a point where we have the model set up so that we recover all of our costs -- because it's a municipality and we have a cost-recovery model.  We recover all of our capital costs.  So we sit back and the checks roll in.

 Now we are looking at lots of different projects that we can do for the city with our surplus revenue.  And we're in the process of partnering with the school system to build a fiber optic network for them.

16:07:

Chris:  [laughs]  I'm sure that's necessary.  I actually read about the franchise negotiations with Comcast, where they REALLY -- they REALLY threatened to raise the prices dramatically high.  And I don't know where it ended up.  But I'm very glad to hear that you're going to be building -- or that you are in the process of building a network for the schools, because they need it a lot.

16:26:

Josh:  Yeah, they need it, and they like us a lot.  And, you know, we're going down that road.  So we're going to break ground on that in another couple of months.

Another question that often comes up is, how do our prices compare with the private sector, in terms of telecommunications.  It's difficult to look at other services in an apples-to-apples fashion.  Because we're a dark fiber service, basically.  The AT&Ts and Verizons and what not are all bundled services, with turnkey bandwidth and what not.  But some of our sort of back-of-the-envelopes have shown us to be anywhere from 20 to 80 percent less than those carriers.

17:12:

Chris:  Well, I would -- I mean, I think that's a pretty modest claim.  And I want to make sure listeners are aware of that.  Because, you know, as you described the connection with VMWare, with the kind of traffic that they're moving, and knowing the companies that serve you flat-out refuse to provide dark fiber, typically, because they want to offer the services where they can make more based on the volume, I have to think that VMWare's savings are even higher than what you just quoted.

17:40:

Josh:  Well, right.  And that would be perhaps the upper end of the spectrum, with the amount of data that they're pushing.  Because we don't meter data.  It's a -- you do whatever you want -- and they do.

And, I mean, the rocket companies here.  We have two rocket companies.  And they use a lot of bandwidth.  They have what I call a virtual campus.  I mean, their connectivity between buildings creates complete transparency.  There's just no bottleneck.  So their data is like the guy sitting next to you.

18:14:

Chris:  Right.  I think we often get hung up on this idea of connecting to the Internet.  And, so much for businesses, it's just about getting across town, or even across the street, without having a bottleneck.

18:24:

Josh:  Oh, absolutely.  And, I mean, if you have a physical building that's not contiguous, or if they're even just next door, as you say, you know, you've got to have a pipe across the street.

18:35:

Chris:  Thank you for coming on, and giving us a better sense of how the dark fiber network works in Palo Alto.

18:42:

Josh:  Well, I appreciate that.  And thank you for inviting me on, Chris.

18:45:

Lisa Gonzales:  That was Christopher, talking with Josh Wallace from the City of Palo Alto about their extensive dark fiber network.  You can learn more about several other communities that have capitalized on this great tool for economic development.  Visit muninetworks.org and click on the "dark fiber" tag.

If you have any questions or comments, please send us a note.  Email us at podcast@muninetworks.org.  Our handle on Twitter is @communitynets.

This show was released on December 18th, 2012.  Thanks for the Mojo Monkeys for the music, licensed using Creative Commons.  The song is called "Bodacious."