This is Episode 17 of the Community Broadband Bits Podcast. Joe Knapp, IT director for Sandy, Oregon, describes a unique idea for fiber deployment for SandyNet. Listen to this episode here.
Lisa: Hello. This is episode 17 of the Community Broadband Bits podcast. I'm Lisa Gonzalez with the Institute for Local Self-Reliance and muninetworks.org. Today, Christopher Mitchell talks with Joe Knapp, the IT director for the city of Sandy in Oregon. Joe describes how SandyNet began using fiber as a way to keep up with the ever-increasing demands of the community. Joe also discusses SandyNet's new project that uses sewer and stormwater infrastructure as a pathway for new fiber installation. Here are Christopher and Joe.
Christopher: I am here with Joe Knapp, the IT director for the city of Sandy, Oregon. We were going to talk a little bit about what his community has done and is planning on doing to make sure that everyone has access to good broadband. So welcome to the show, Joe.
Joe: Thank you.
Christopher: Joe, you've been in Sandy, running a network for quite some time. Why don't you start by telling us a little bit about Sandy. Where are you located? What's it like?
Joe: We're in Oregon, just about 30 miles east of Portland. Sandy's known as the gateway to Mt. Hood. We're situated basically right between Portland and Mt. Hood. It's a great location. We're 30 to 40 minutes from all the wonderful amenities of the bit city of Portland, and then we're also 30 or 40 minutes from three different ski resorts on Mt. Hood and all sorts of outdoor recreation activities up there.
Christopher: The last time I was in Sandy ... I've been through it a couple of times ... Hood to Coast was going on. Fantastic race.
Joe: It's an interesting time of year. I think we get something on the order of 50,000 come through this town on that weekend. It's pretty interesting.
Christopher: Sandy has been running a wireless network for a very long time. We're going to talk toward the end of this show about the fiber network that you're moving toward, but I want to make sure people are aware of just how long Sandy's been working in this space. Can you give us a little bit of history in terms of the city became involved in making sure everyone had access to good broadband?
Joe: Yeah, absolutely. We basically had started this venture by necessity. About 10 years ago now, this city was pretty bad as far as broadband connectivity options. In fact, at our own City Hall, we were unable to get a DSL connection from the phone company, which was GTE at the time. After quite a while of frustration, the city decided that we just needed to take care of it ourselves and started a municipal ISP. Initially, we actually started with DSL. We became a CLEC, which we still are, and began doing lineshare DSL services across the phone company's infrastructure. That has just continued to evolve over years as demand has increase and increased. The next step after DSL was wireless networks. We've tried several different wireless technologies over the years. Currently we're using a combination of 802.11 in Wi-Fi, which is what you have in your house. But we've got some high-powered outdoor equipment now all over the city on the order of about a little over 100 access points around the city. We're currently serving about 1,300 customers with that system. We've also got several wireless towers that we've constructed around that perimeter of the city that are serving our rural area as well, to help give those folks some options as well, because they are still pretty despaired about what is available in the rural area.
Christopher: GTE was actually later purchased by Verizon. It's a very large company. They were just not getting the job done then.
Joe: Yeah. They've also now subsequently been purchased by Frontier, which to Frontier's credit, they've been starting to roll out more rural DSL, which is great because there are areas that we still can't reach due to trees and terrain. Frontier's come along and is rolling out some rural DLANs that are allowing that to be served. Our cable company also ... That's another point worth mentioning. 10 years ago, the cable company was Falcon, and they had no broadband. Then they were purchased by Charter, who also had no apparent interest in upgrading their system to have broadband. Up until about three years ago now, our cable system was all one way. Then finally, Charter sold their franchise off to Wave Broadband. Wave has since come back and started upgrading the system, and is actually giving cable broadband as well now. We do have three service options now. There's SandyNet, Frontier DSL, and Wave Broadband Cable. But we found that having established ourselves early in the market, we've continued to be very competitive. People love having the local service option.
Christopher: I'm actually fairly surprised to hear that. A lot of cities find themselves having cable access that's available everywhere, and then DSL is available in a lot of places. Without SandyNet and without having a cable provider offering broadband, then I would imagine there would have been a number of people who were totally unable to access the internet at high speeds just because DSL just doesn't go very far in terms of being able to service everyone in the community.
Joe: Yeah, absolutely.
Christopher: SandyNet came along, and it served that vital need. I know that when you and I met, I want to say two years ago now, you already had plans to try and get in front of the next wave of innovation, the fiber optic networks out to everyone. What was your step? What was your next step?
Joe: We've had some experience with fiber networks. The way that's evolved is we saw a need. I've been with the city for a little over five years now. When I started on our internal network, we had no connectivity between our facilities. We're a small community of about 10,000 people. We have five municipal buildings. Essentially they were all running their own separate networks. We had no phone system integration between those locations, no network integration between those locations. So people are literally taking files back and forth on thumb drives. It was crazy. We started looking at options for getting those facilities connected. One of the big driving factors behind was so we could put in a voice over IP system and start saving some money on our phone bills. We were actually able to work the numbers on that and show that we could get a return on investment on building some fiber networks out relatively quickly that would enable us then to have a more integrated network as a city. Since we had the SandyNet utility, we also looked at that as an opportunity to start extending fiber service to our business customers, because the path that we needed to take to get our municipal facilities connected just happened to go right through our whole downtown core and past several hundred businesses. Our first foray into fiber was to look at connecting those businesses as an economic development strategy of how can we provide cheap, huge broadband speed to these companies to meet their needs, while also accomplishing that goal of getting the city network built out. That's worked out really well. We haven't had a lot of people take that service because it's relatively expensive to do construction to those individual businesses. But a lot of our larger businesses, I think probably all but one of our real estate offices has taken the fiber service. Then the majority of our large employers in the city saw the tremendous benefit of that fiber connectivity, and have taken that service. We've got one company, for example, that does aeronautic manuals. They build and maintain these manuals for planes, for private jets and stuff like that. They actually have a offsite facility in Costa Rica that they use our fiber connection to do hot replication of their server farm from Sandy to Costa Rica. It's amazing. We love getting testimonies from that business owner because he says, without SandyNet, we wouldn't be able to do this. It's fabulous. The price that we're able to do that at for our business is typically cheaper than a T1 connection for a full 100 megs of internet. That was our first step. Then what we've seen with our wireless network is over the last four or five years, the demand that has been placed on the internet due to technologies like Netflix and other video services has just been changing. It's a constant battle for us to keep up with it. Our city manager accurately refers to our SandyNet crew as like the circus act where the guy's spinning all of those plates and keeping them up in the air. He says, that's what you guys do. You have all these plates spinning, and you're trying to keep them from falling with this wireless network. We've seen this trend over the last several years of not being able to keep up with the demand. As we started evaluating options to stop that trend, fiber seemed to be the logical choice. That's how we got to that next step.
Christopher: Was it a situation where that was about the time when Google announced they were going to be building fiber somewhere? It was about two years ago now. Three years ago, maybe.
Joe: Yeah. At that point, we had already been doing some pretty significant build-out of fiber in the business district. Probably the biggest compelling thing was, one, obviously the speed is phenomenal. Two, I'm going to knock on wood, but to this day, we have not had a system outage on our fiber network. We've had customers in operation on the business side of things for three years ago, and we did never have outages. It is so solid.
Christopher: Do you have redundant rings?
Joe: No, no. Building a ring network, buy design, it almost doubles your costs, so we haven't done that. But all of our fiber's underground, so we don't have any risk of aerial hits. Since it's in the business district, we have the benefit of the municipality because everybody has to pull permits from us to open the street. We know any time any street work is going to be done, and we make sure that we're there to locate our fiber for them to ... don't hit this, along with our water and sewer infrastructure and stuff like that as well.
Christopher: As a municipality, and people pull permits, have you taken advantage of open streets to expand your fiber, or have you had to open the streets, put in the fiber, and close them back up again?
Joe: We have. We've talked about the open streets, of doing an ordinance similar to ... I think city of Portland has something like that. Several cities around the country have done that. We haven't enacted anything yet, but we have talked about it several times. That's, I think, is definitely a good direction to move. Both of our major projects, what we've done ... We've done three major fiber projects so far. All of those were just existing build-outs. But the interesting thing about Sandy is our first major fiber project was done in conjunction with an undergrounding project that we did in our downtown core to beautify the city, if you will. Before the project was done, we had over 200 street crossings in about a quarter of a mile distance through our downtown core of overhead cables. The city council basically decided we wanted to eliminate that. We partnered with all the utilities that were in the area to relocate their facilities underground. The city paid for a portion of that, and then utilities were on the hook to pay for a portion of that as well. At that time, since we had this huge open trench, everyone engineered together. This is where the facilities are going to be located. At that time, the first main SandyNet fiber conduit was put in the ground. Then we've extended that on a couple different projects, one of which was in conjunction with the waterline infrastructure install that we did. We've got another project like that coming up with a pretty major waterline tie-in that we're going to do that's going to have fiber conduit laid in there as well.
Christopher: Basically, when I became aware of Sandy, it was not because of all of these great things that you just described. It was because of a contest that had a terrific name. Can you tell us where that came from and what it was called?
Joe: Yeah. We ran a contest called the Why Wait for Google contest. Essentially what happened there was, as everyone in the industry knows, Google announced their project. We read that announcement and we were very excited, but it was very apparent to us that we were way too small for Google to consider us an option.
Christopher: I think Portland actually brewed a specific IPA and was pretty hard to compete with.
Joe: Yeah, I think it was BridgePort Brewery made a Gigbit IPA, which is actually a pretty good beer. There was lots of crazy things the cities did. We looked at the contest and said, boy, how nice would that be? But there's no way that Google would come to us. We're just too small. We started throwing ideas around, and we decided to have a contest called the Why Wait for Google. What we did was sent out letters and media to our residents and said, here's what we're going to do. We're going to open up a poll. We built a website and had a survey that people could go in and fill out. We essentially mapped those responses by neighborhood, and said the neighborhood with the highest level of interest will be the subject of a pilot project that we're going to do to see if we can make this fiber to the home thing work. Really, the push for that came from our council. Our city council has been very cognizant and supportive of the fact that broadband infrastructure is critical for the economic success of the city. They have been very, very behind this project, and enthusiastically encouraged my staff to move forward with exploring those options. The problem with the whole Why Wait for Google contest was we launched this thing, and we pretty much got equal response from almost every neighborhood in the city. Then we were like, great. How do we pick? There was three neighborhoods really that were just a little bit above everybody else. Of those three, we picked which one. It was literally down to one or two-response difference in which one was going to go first. Of those three neighborhoods, we picked the one that just had a slight edge, and decided to move forward. We issued a formal RFP at that point. This was all, again, for traditional build-out. We did a formal RFP and received several responses. Honestly, I don't remember where we came up with this number, but we had this number in mind of we want to provide 100-meg service, synchronous ... 100-meg synchronous service for $39.95 a month. That seemed like a pretty lofty goal, but we felt it was achievable. When the RFP responses came back in, we were so close on the dollar figures that needed to be there, but just not quite. We couldn't get the price down to where we needed it to be with the take rate that we were getting. We ran ... it's ridiculous amounts of survey to gauge interest in these neighborhoods and predict what our take rate would be, and just didn't quite feel comfortable with where the numbers would be.
Christopher: The RFP, what was the model envisioned? Was this going to be a partnership of some sort, or was this an RFP of expecting that you would own and operate it, and presumably bond for it or find some way of paying for it?
Joe: The initial RFP, it was completely owned and operated by the city. The financials would have been probably ... A revenue bond was the model that we were looking at, although there was just some traditional loan options as well. I'm not a finance guy necessarily, but our finance director had all sorts of projections, and schemes, and ways of figuring out how this was going to be funded. I'm very thankful that we had him on staff, because he also explained things to me like coverage ratio, and how to get a revenue bond, you need to have these dollar amounts, and this has to work this way. He was able to give me very real numbers of what you need for a take rate to make this project work with what you want to charge to customers. That was very helpful, to have solid financial data going into it, to basically be able to say at the end of the day, this won't work. We can't safely do this. That's something that, at SandyNet, we've always tried to do things that were very minimal risk with our wireless network. We've grown out neighborhood by neighborhood, because we do not want to come in and invest millions of dollars into a network to have it fail, so that we would become another black eye for the municipal broadband initiative that's happened in this country now. That we all know the stories of communities that have done this and didn't have the expertise or experience, or were misguided by a consultant maybe, and unfortunately things fail. Then the industry looks at that and says, see? We told you municipal broadband doesn't work.
Christopher: Right. There are, of course, many accusations of failures that are not true, but there are some that have had exactly those problems that you described. I think it's by taking such an intelligent view and making sure you're doing your research, as Sandy's done, then you avoid that.
Joe: With the pilot project, we unfortunately, like I said, at the end of the day, had to say we just can't get this to pencil out. At that point, we decided, as a city, we said, let's have a brainstorming meeting here and see what options are available, if we can think of any ways to continue to push forward with this and keep the cost where we want it to be. We had a meeting between my staff and our public works department. Our engineering department at the city was there. A couple other departments have come in and said, what can we do? It was at that meeting that we said ... Somebody asked the question, what infrastructure do we own? I think this was spurred by we looked at some of these successful fiber networks, like Chattanooga and other areas that have done this. One common factor that exists amongst a lot of these networks is they own electrical utilities. They've got that network of poles, and it makes it so much easier for them to install this infrastructure because they've already got the underlying stuff there. We asked ourselves the questions in light of that, of what do we have available that's unique to us, and how can we utilize that? One of the things that came out of that was our stormwater and sewer infrastructure. That's an asset that's unique to cities. It's almost entirely municipally owned in the United States. We started to look at ways to use that. We've actually gotten in contact with i3. They submitted an intent to respond on our initial RFP that we had issued for our pilot project, but we never got a response from them. I called them up and said, what exactly? I know you mentioned some sewer stuff, but it sounded crazy. What exactly does your guys' product do? That has since obviously morphed into this project that we're going to undertake with them that's a very interesting model that I think has never been done before.
Christopher: I do think it's a new approach in the United States, but I think that the general technology has been around a little bit. I want to say in Asia and in Europe.
Joe: Absolutely. In fact, i3 gave us a lot of examples. They've been deploying the same technology. They've been deploying it for about eight years. A lot of their work they've done in South Africa. They told us they've got a lot of network deployments in the UAE, a few in the UK. I think New Zealand and Australia, they have some small deployments as well. From that perspective, we're pretty comfortable with the technology that they've proposed. I think it's a very compelling solution. The model that's different than anything that's been done before is where they have a capital investment on this thing is going to work.
Christopher: Is it the technology in which you run the fiber cable through existing sewer pipes?
Christopher: Can you give us a quick sense of where it goes in, where it comes out? Obviously there's a joke that Google played back on April Fool's Day, where it actually had a fiber cable snaking out of the toilet, but it doesn't do that, right? Where does it enter and leave the sewer pipes?
Joe: It's going to use sewer and stormwater infrastructure both. The minimum pipe size diameter for their technology is 6 inches. That almost, for our city, at least, certainly limits it to just the main distribution cables of the network. What their technology allows us to do is use our existing sewer and stormwater facilities to get our feeder cables, or our trunk cables, if you will, out to the neighborhoods. In our specific build, our central office, if you will, is going to be at our City Hall. From there, they'll run large feeder cables, like 432-strand cables, into the sewer lines, that will then go out into the neighborhoods. Then essentially, they come out of the manholes and up into the right-of-way behind the sidewalk. From that point, it's a traditional build to the houses. Where the tremendous savings on both deployment costs and the deployment time come into play is what's typically done with lateral drilling to get those feeder cables out to the neighborhoods. That's all just pulled through existing infrastructure. They've been able to dramatically decrease the deployment time. The speed with which they can deploy is tremendously fast, which inherently dramatically decreases the labor cost because you're not paying the crews to be out there as long. They were able to bring the network costs way down with this technology.
Christopher: It also reduces the financing costs because you're not spending a lot of time with other people's money and not getting revenues back on it, right?
Joe: At this point, they're deep in the design phases of the network. We're going back and forth with plans, but they're very confident that they can have about a 90% build-out in less than nine months. For Sandy, that's about 4,000 premises. They figure they can get about 4,000 premises deployed in nine months or less.
Christopher: Let's talk about the capital and the financing model of that. How is this going to work?
Joe: When we initially started talking with i3, we explained to them what we wanted to do with the pilot project, and so the direction we have been taking. They said, that's all fine and good, and we'd be happy to help you with that. But would you be willing to entertain, maybe, a more aggressive approach? We said, sure, yeah. We'll listen to anything you want to look at. They said, what if we take the capital investment and not just build to this neighborhood of 500 homes, but what if we build out your entire city? Essentially, the model that we've agreed to with i3 is they are doing all the capital investment in this network, so they're building the network and they will own the network. The city has a 20-year agreement with them that we are committed to pay them on a per-home-passed basis with a minimum commitment rate of 50% homes passed. We essentially are paying them for half of the homes that they pass on the network. Our assumption, obviously, is that we're going to be able to get 50% or greater take rate to make that happen. The benefit of that to us, obviously, is the major capital investment that we would have had to come up with, a revenue bond or some other means of financing that, that burden's no longer on us. We're just committed to make the monthly payment to them. The other great thing is that monthly payment ramps up. As the houses are passed, they release those homes to us as being ready for service. Then we start making the payments once the fiber's lit up.
Christopher: What is the payment?
Joe: We've agreed to a rate of $30 per home. It gives us that margin of basically $10 per customer to cover our operating costs, which is a very thin margin to cover operating costs, but we're used to working on a pretty thin margin. It's nothing new for us.
Christopher: At that price, obviously you're not going to have a lower tier. Are you going to offer one tier, then?
Joe: No, we're going to do two tiers. The 100 megs is the low tier for residential. The 100 megs is our low residential tier at $39.95.
Christopher: That's a little bit of a bragging point.
Joe: Yeah. Then we're also going to offer a gig, so a full gig synchronous, for $99.95 a month.
Christopher: How is it that you have access to the kind of backhaul that ... If I saw this from a private carrier, I would just assume that they were oversubscribing the heck out of it. How do you make it work in rural Sandy?
Joe: A customer that takes the gig service, for example. If we had 40 customers taking a gig, we're not going to have a 40-gig pipe to the internet, because that's just unnecessary.
Christopher: Right. No, absolutely. Everyone oversubscribes to some amount, but I think some of the private carriers push it to ridiculous proportions. Usually municipalities are much friendlier to the consumer in terms of being honest about what they're offering.
Joe: Right, definitely. One of the reasons I think that we can keep the price so low is due to a partnership we have with Clackamas County. They were awarded a BTOP grant to build a middle mile ring. This is a success story of why that middle mile infrastructure is so good. Sandy is essentially a corner on that ring in Clackamas County. We have partnered with them to allow them use of our right-of-way. We gave them some conduit that we had to get them through various parts of our city where they couldn't be aerial. Then we also gave them space in our data center at City Hall to place one of their fiber optic distribution racks, basically a patch rack. We have a 288-strand cable coming in from the west into Sandy. 144-strand extends east out of Sandy, up Mt. Hood. Then I always get these fiber count ... I think it's a 212-strand goes south out of Sandy. We're like a corner on that ring with the spur extending out from us. Part of the deal that we worked with Clackamas County was we'll give you all these resources within Sandy to make this thing possible for you guys, because we think it is a very important project. Then in exchange for that, that gave us a couple strands on the ring, so we essentially have dark fiber back into Portland at no direct cost to us. It was a trade that we did with them for the infrastructure piece. They're getting effectively free rent in our data center, and we're getting free rent on their fiber network.
Christopher: We've seen those trades in a number of cases. Of course, they're very commonplace in the private sector as well. You must have just an excellent contract in Portland, or else there must be an assumption that you're going to be using so much that you'll be able to keep the rates low as you're negotiating with providers back to peer with others and to connect to the wider internet.
Joe: Our internet connection is pretty easy to get cheap access. If you're buying stuff that's at capacities of a gig, it's not difficult these days to get that at a pretty decent cost. The problem for us historically has been transport. To get the actual fiber patch from Sandy back into Portland has always been very expensive. To have that piece effectively eliminated by the county building this middle mile project has opened up huge doors for us to be able to get all sorts of bandwidth and provide redundancy because we're on a ring. We can take that ring in both directions to either get to different points in Portland or to pass to the same point, however we want to structure that. It has allowed us to really up our capacity to the internet without adding much cost.
Christopher: We're almost starting to run long, I'm noticing here. But I do want to get to the point that you said that i3 will own it, and you will guarantee that revenue. My understanding is that city some options down the road to purchase it back. Can you tell us about that?
Joe: Sure, yeah. Like I mentioned, the 20-year agreement. Halfway through the agreement, at the 10-year mark, there's a buyout option. I would say with almost complete certainty we will exercise that buyout option after the 10-year mark. We will always be the operator of the network. i3's going to build it, and they're working very closely with us now on the network design of how we want it and what we need to meet our needs. Then once it's built, their contract holds them to repair and maintenance of the network, as well any network extensions. If a customer wants to hook up after the network's been built, i3 sends a crew out and does that. That's all spelled out in the contract. They'll operate the network from a physical infrastructure perspective because it's their asset. But from the management and provisioning of the customer equipment, that falls to Sandy and its staff. Then after that 10 year point, when we buy that network from them, then it would basically completely remove them from the picture. The network will be totally owned and operated by SandyNet. At that point, we'll obviously have to ramp up crews for the physical maintenance of the network and things like that, but we also will no longer have that monthly financial obligation to them, so it'll open up a lot of opportunities there.
Christopher: What will the costs be in 10 years? Do you know what that is?
Joe: Yeah. The contract spells that out as basically, as you look at the course of the agreement, we're going to be paying them on a monthly basis. The contract states that whatever we pay i3 for the 10th year of the contract, then we would take amount and multiply it by two and a half times, and that would be the purchase price of the network. We've run with our projections of what our take rate is, what we're assuming we're going to get. We think the total cost to the city over the course of the 10 years, including the buyout, will be somewhere on the order of about $10 million, which is interesting in light of the fact that as we were running our pilot project, we also looked at data for building a fiber network for the entire city. Our cost that we came up with to do a traditional build for the entire city was somewhere on the order of about $15 million. We're effectively getting exactly what we wanted for two-thirds of the cost.
Christopher: I certainly hope it works out well. It's always great to see these new approaches. We're excited to have learned what Sandy's done. We're excited to see where Sandy's going. I'm sure we're going to check in with you in a year or two to see where you're at and get a update. Thank you so much for coming on this show and telling us about the network.
Joe: Yeah. Thank you.
Lisa: That was Christopher and Joe Knapp from Sandy, Oregon. You can learn more about the community and SandyNet at cityofsandy.com, or visit muninetworks.org and search for SandyNet or sewer to find out more about the specific project. If you have any questions or comments, email us at firstname.lastname@example.org. Our handle on Twitter is @communitynets. This show was released on October 16, 2012. Thank you again to Fit and the Conniptions for the music, licensed using Creative Commons. This song is called Got My Modem Working.