This is the transcript for episode 391 of the Community Broadband Bits podcast. In this episode, Christopher interviews Director of Technology Services Dave Williams about Ponca City, Oklahoma's new fiber network. Read the transcript below, or listen to the episode.
Dave Williams: So as we become more competitive when it comes to attracting businesses, we stand a chance of surviving into the future that way.
Lisa Gonzalez: Welcome to episode 391 of the Community Broadband Bits Podcast, from the Institute for Local Self-Reliance. I'm Lisa Gonzalez.
Lisa Gonzalez: For a long time now, Christopher has wanted to bring a representative from Ponca City, Oklahoma on the podcast. Dave Williams, Director of Technology Services joins him today. Known as a pioneer in city Wi-Fi, the community has recently launched a fiber network utility for residents. Dave shares some historical perspective, discusses the utility and describes how it fits in and collaborates with the city's other utilities.
Lisa Gonzalez: He talks about Ponca City's free city-wide Wi-Fi, including successes and challenges that have arisen. Christopher and Dave discuss the community's decision to expand Fiber-to-the-Home, marketing the new service and going the extra mile to make the service subscriber friendly. Now here's Christopher talking with Dave Williams from Ponca City, Oklahoma.
Christopher Mitchell: Welcome to another episode of the Community Broadband Bits Podcast. I'm Chris Mitchell at the Institute for Local Self-Reliance up in Minneapolis. Today I'm speaking with Dave Williams, the Director of Technology Services in Ponca City, Oklahoma. Welcome to the show.
Dave Williams: No, thank you. Thanks for having me.
Christopher Mitchell: Oh, I was just telling you before we started the interview. I'm quite excited. I think what you've done in Ponca City is fascinating. We're going to talk about wireless. We're going to talk about fiber optics. We're going to talk about things more than 20 years ago, and it's wonderful. But let's start off by getting a sense of what Ponca City's like.
Dave Williams: Ponca City's in North Central Oklahoma. We're about 20 miles south of the Kansas line.
Christopher Mitchell: The first thing I'm learning is that it's not Ponca, it's Ponca. Okay, I can remember that.
Dave Williams: Ponca. That's all right. We understand you northerners. Ponca City's actually what's called a micropoliton area. We're a city with a population of over 25,000 in a county with a population under 50,000. So we're nowhere near as big as Oklahoma City, Tulsa, some of the larger cities in Oklahoma. But for the geographic area we're in, we're the city that other cities look to in our general area. When it comes to certain technology projects, we're the city that stands out in Oklahoma as being the first to do a lot of things. City-wide free Wi-Fi. We're not the first to do municipal broadband offering, but we'll be the first to actually complete our project.
Christopher Mitchell: Well, let's just jump right into that then. Ignoring the all the history, what is it that you're working on right now?
Dave Williams: City commissioners here in Ponca City believe that our residents deserve the same type of internet service available in larger cities. Unfortunately, none of the incumbent providers were willing to step up to the plate and either build or rebuild their outside plant to offer those services in a consistent, meaningful fashion. So we decided to fix the problem ourselves.
Dave Williams: On July 1st of 2019, Ponca City Broadband was created. It's a new utility service in Ponca. It'll be the fifth utility that Ponca City actually owns through our utility authority. But Ponca City Broadband will offer ultra high speed internet access to every home and business inside the city limits as its initial process, and eventually we'll cross the city boundary lines and offer it to neighboring communities as well.
Christopher Mitchell: One of the things you mentioned is one of the things I'm quite curious about in that it is a utility, the city has an electric department, but this project it seems to be run from your office and not from that. I'm curious about that because that seems a little bit irregular from what we normally see.
Dave Williams: There was discussion back and forth on whether it belonged in the energy office or the technology services office. Historically, the fiber throughout town... Ponca City's had fiber optic networks since 1996, primarily for the city's private network use. But then we rapidly expanded that to cover the public schools, the university center and our hospital here in town. And then expanded that even more starting in about 2005 to offer business broadband services.
Dave Williams: It was a good partnership between technology services and energy throughout the years. Energy had the equipment and the people to string the fiber up. I mean, it's somewhat similar to stringing up electric cable, but they didn't have the internal knowledge of how to operate a network. So we've always worked on a partnership with that. But with the formation of Ponca City Broadband, where we're now offering this to residential customers, we moved the entire operation over to technology services. With the overriding network philosophies that go into this, it was a better fit in my department than it was in energy.
Christopher Mitchell: Sure. Do you still share trucks and things like that then, or are those things totally separate now?
Dave Williams: It's a hybrid model right now. We actually have two physical plants in town. We've got our legacy fiber system that the energy department will maintain and continue to work on the outside plant. But the new fiber plant that's being installed for residential services is a different set of employees. We're actually hiring employees, buying our own equipment and beginning to expand that service. We started this project with a fairly small phase one area of town, and we picked a fairly small area intentionally. We're not sure what it actually takes to operate a new utility. So we're going to learn as we go and grow as we go.
Christopher Mitchell: Well, let's go back to what you were talking about, and you're going to give us the thumbnail sketch of the history of it. But why did you expand to the schools and the businesses? What was the motivation for that at the time?
Dave Williams: The schools, we have a quid pro quo agreement with the schools. We ran fiber through the school to take advantage of network switch room locations that were already present there so we didn't have to build new. And in exchange for that, made sure the schools had their... It's dark fiber for us. We just provide the fiber. They actually run their own network. So that was a good agreement at the time that we didn't have to build new NOCs or network locations throughout town. We used existing buildings that were there, and the schools offered us a a pretty good geographic dispersement of those buildings.
Christopher Mitchell: If I had to guess... a pattern that we've seen elsewhere is that you did that, and the schools were getting good access and you had high quality access around town, and the businesses came to you and said, "Hey, why can't we get on this?"
Dave Williams: The business broadband portion of this precedes my time of employment with the city, so I'm dealing with some history and just word-of-mouth knowledge now. We did have a couple of businesses that approached us and expressed an interest, and at the time that happened, the city wasn't prepared to take that on. But it was kept in the back of everybody's minds that as we grew and our fiber footprint got larger, my predecessors saw an opportunity to pick up a revenue stream to actually help fund some of the other initiatives that were happening in town, increasing the fiber footprint, the city wide free Wi-Fi, several other things in town that happened within a few years after that.
Christopher Mitchell: Let's talk about that city-wide Wi-Fi. That seemed like it came after the Wi-Fi bubble elsewhere, in that it was... maybe had learned some lessons of what worked and didn't work in other places.
Dave Williams: Well, in some cases, yes, and in some cases, we probably set the example of what to do and what not to do. The city-wide free Wi-Fi was an expansion of another project that was going on in town here. It was called the Smart City Initiative. Rather than have meter readers go out and read the electric meters and the water meters, we now have that all reported back up through a radio or Wi-Fi frequency.
Dave Williams: The bandwidth and the equipment that was necessary to do that for the utility departments, we saw an opportunity to expand that and be able to offer it as a free service. And this isn't hotspot service. You don't have to go to the library to get this. Most people in town, from their homes, can see the Ponca City free Wi-Fi radio signal and just attach to it.
Christopher Mitchell: I had read somewhere that 25% of the nodes were actually on the fiber backbone, which is a recipe for keeping the quality pretty high if you don't have too many hops.
Dave Williams: Right. We have approximately... it's 85 gateways in town, and a total of 452. Of the 452, 82 of them are the gateways, the rest of them are just repeater nodes. And the goal has always been never to make more than two hops. From a repeater to a repeater you need to talk to a gateway.
Christopher Mitchell: Mm-hmm (affirmative). So one of the things that I remember reading, although it could have been incorrect, or I could be misremembering, was that you also had a significant public safety component of that too. Was that right?
Dave Williams: Well, initially, the fire and police departments were going to use a private channel on the same Wi-Fi radios. But that's something I was talking about earlier, whether it was a good decision or a bad decision. If we had just left this as a muni-wide free Wi-Fi or muni-wide Wi-Fi, the police and fire components would have worked really well on that. The muni-wide would have been the roughly 85 radios throughout town.
Dave Williams: So that gives people a chance to, as you're traveling through town, you're not disconnecting from one connecting to the next, connecting to the next. By expanding that to offer city-wide free Wi-Fi, what we saw was an unexpected amount of additional connections and traffic on the network that made it a little less than reliable to the police and fire departments. So we struggled for a couple of years trying to make that work and finally abandoned the public safety portion of that and looped them over to private cellular carriers.
Christopher Mitchell: Okay. And one of the things that we've seen in Wi-Fi networks that have stood the test of time is that as the streaming video has picked up and picked up and picked up, that people have ultimately just wanted more and more and more. And so was that a part of the reason that you decided to go to the Fiber-to-the-Home ultimately?
Dave Williams: That was a good push to start with, but the major emphasis on Fiber-to-the-Home is that it's now the future-proof network. The speeds and bandwidth that we're able to do in fiber optics right now, the fiber itself can handle hundreds or thousands of times more bandwidth than that. You are right though, that our residents were screaming for more, that free Wi-Fi wasn't cutting it when it comes to streaming a movie. There were delays getting the movie started. It buffers several times. And even though it's free, it's not deemed to work as well as it should. Ultimate evolution of that is let's give them something that now is future-proof.
Christopher Mitchell: One of the things that we haven't mentioned is I'm presuming that there was some cable and DSL available at that time, but many people were choosing your service instead.
Dave Williams: Well, free is hard to compete with.
Christopher Mitchell: Yes. Although I'll tell you that as someone who uses this quite a bit, in my example, I'm generally willing to pay for quality. And I'm guessing that as you're going to market, you're going to be having to compete against others. So you're going to have to have different expectations now perhaps.
Dave Williams: Right, and we're addressing that through our marketing strategies right now. Most all of the incumbent providers here in town have an asterisk in their plan. And if you read it carefully, it'll tell you, you get up to the bandwidth you subscribe to. Our service agreements, we don't require contracts, we have a service agreement, just like a typical utility services agreement. And our agreement says you'll never get less than what you pay for. We guarantee you'll get the bandwidth you paid for.
Christopher Mitchell: Do you have a mechanism for testing that in the gateway at the home then, basically?
Dave Williams: The hardware provider we went with also has analytics software, so we can actively monitor home connections to see, is anything falling through the crack? Is there a bad Wi-Fi signal? Is there potentially a bad experience in a home? Better than 70% of the time we contact the subscriber before they have a chance to contact us.
Christopher Mitchell: And say, "Hey, that 10-year-old Wi-Fi router is not cutting it anymore."
Dave Williams: Fortunately, none of our equipment... We just started operation July 1, 2019, so all of our stuff is still very new, but what we are seeing and we're able to share with our customers is, "Your 10-year-old laptop is connecting and using a 2.4 gig signal, and it's locked into one or two channels. It can't surf the channels like it needs to." Unfortunately, there's nothing we can do to make that 10-year-old laptop work any better, but we can recommend you might consider replacing that if it's causing you problems.
Christopher Mitchell: Right. Now I want to talk a little bit about how you've accomplished this because one of the things that I've seen in communities that have done what you've done, in terms of this incremental approach making remarkable investments at reasonable costs, is that it seems like you do a lot of work in-house as opposed to going with contractors. And I'm curious, can you just tell us how you've managed that?
Dave Williams: Well, actually we've adopted the opposite approach to that. We hired two employees to start with, and we outsourced most of the operations so far. In fact, all of the outside plant construction is contracted out. A majority of the home installations, I have in-house staff that can do up to 10 installations a week. We have a local contractor that can do up to 60 more installations a week, and then we have a larger contractor that I could bring into town, and they can potentially do 150-160 a week if needed.
Dave Williams: We're outsourcing our help desk. We're outsourcing our network engineering function, primarily because we really didn't know going into this, what's exactly involved in that. I mean, how many calls are we going to get a week? How long does it take to satisfy the customer on that phone call? And it just made more sense to outsource that. But we did outsource to a fairly local... we went with a Midwest company for our help desk for a couple of reasons. We want somebody that understands how to speak Oklahomese, Okie. We need him to understand our accents.
Christopher Mitchell: No, I have thought that maybe it was prior phases, I had read. I just would've been doing some background research to prepare and maybe it was previous phases you'd done more in-house work. I've seen both strategies work, but one of the things that we've seen is a challenge with, especially in a tight labor market, having the contractors take enough care to keep your reputation with your new customers. But that's not been a problem for you then.
Dave Williams: Right. Our service level agreement with the help desk, we're very specific on that, the amount of time they have to answer the phone, how many rings, how long a customer could be on hold, a threshold percentage of problems that are handled with that single phone call compared to a truck roll, even compared to a second phone call. And so far we've been very happy with that, and our customers absolutely love it.
Christopher Mitchell: Did you benefit from planning ahead with dig once policies in terms of, years ago, getting conduit in the ground and things like that to prepare some of these neighborhoods?
Dave Williams: Well, I wish we had, but the truth is no we didn't, and we're going to pay for it now because we have, in the last two years, have adopted a... If any city department puts conduit in the ground, they put spare conduit next to it. But there's just a small percentage of the town where that's already available. Now our energy department is a good partner with me on this.
Dave Williams: They're beginning to go through some of the problematic areas in town where we've had what's called phase problems with the electric. It's where the utility poles are in a back fence easement, and trees grew up through them, and a tree branch can short the two phase power out or the three phase power. So our energy department is actively moving to put their primary energy circuits underground, and while they're doing that, they're also pulling vacant duct for fiber to go that route as well.
Christopher Mitchell: Oh, isn't Oklahoma hard to dig in? Isn't that a significant cost?
Dave Williams: Parts of it are. In fact, Ponca City's roughly 21 square miles. The east part of town is built up on some fairly dense limestone rock, so that's going to be some major expense for us as we go underground because it's not only underground, it's through rock. But the geographic formations, I mean, it is what it is. And we're going to have to address it and deal with it as best we can when we get there. Fortunately, those areas of town are the last two phases of a five-phase project here at Ponca City.
Christopher Mitchell: And how was the first phase treating you? When I read what appears to me now to be a perhaps a faulty source, you had completed the first phase more or less, you're working on phases two and three of laying them out and things like that.
Dave Williams: Right? Phase one construction is complete. We opened the doors to the public on that July 8th, 2019. Our customer numbers, it's actually a little slower than we anticipated, but we're getting there. We're on track to meet our take rate assumption for the overall phase one area, but we have identified some unanticipated challenges of getting customers to switch. Most people, when you think about changing an internet provider, you think about how painful it was the last time you did it.
Christopher Mitchell: That's right.
Dave Williams: There's some reluctance then that unless you absolutely cannot stand your current provider anymore, you're willing to put up with them a little bit more. Word of mouth is going to be our best advertisement on that because it's one thing for me to tell the world how great Ponca City Broadband is, but when your neighbor tells you how great it is, you'll listen to that person.
Christopher Mitchell: And you're starting to see that now, six months later.
Dave Williams: Yes. Yeah. Our customer onboarding numbers now are... it was higher last week than it was the same week of the month before, so we are seeing a definite upward trend in that. Another problem we faced here in phase one, phase one is roughly one square mile, almost in the center of town. And it's very difficult to mass market to one square mile in a 21 square mile town with various direct marketing. Some work, some don't, but again, word of mouth is kicking in, and we're hearing from new customers, "I'm here because my neighbor said you're the best thing since sliced bread."
Christopher Mitchell: Well, I'd imagine it's a political question in terms of if you want to start bragging to the entire city, knowing that some of them are going to be waiting a few years. You don't want to upset them.
Dave Williams: Right, right.
Christopher Mitchell: How do you know that this is a smart investment? Are you hearing anecdotes from people, or do you have a sense that it's attracting people to town now that you have this much higher quality service? What's your external validator?
Dave Williams: We're working off and on closely with our development authority and our chamber of commerce. They're feeding us information from businesses that are sniffing around the edges of Ponca, wanting to know, is there office space, or is there a commercial space available for them? One of the questions that always comes up is what kind of internet connectivity, what kind of broadband access do you have?
Dave Williams: And here in the last year, now that we knew that this was going to happen, there's actually... we've attracted three outside businesses in the last year that weren't even on the radar scope prior to that. Now, I don't know that it's fair to say it's because broadband's here, but broadband certainly gives us the competitive advantage that we wouldn't have otherwise.
Christopher Mitchell: Well, I saw that your impressive access for small businesses that you've been making available previously, combined with your wireless network had led you to get a recognition from American Express, in terms of being a great location for small businesses several years ago.
Dave Williams: Mm-hmm (affirmative). We keep trying to capitalize on that and trying to find new, additional, different ways of getting that information out. Because Ponca City's a population of roughly 25,000 people. It's really not a bedroom community to anything close by, even though it is... We're in a triangle between Oklahoma City and Wichita, Kansas and Tulsa, Oklahoma. It's about an hour, hour-and-a-half either direction to get to one of the major metropolitan areas.
Dave Williams: Small town USA doesn't typically attract a migration of people wanting to move to it. If you want to retire to one of those communities, or if you've got family there and you're trying to get close to them, that's typically the reason that people move to a small town. So as we become more competitive when it comes to attracting businesses, we stand the chance of surviving into the future that way.
Christopher Mitchell: Do you have any advice for a person from another town of 30,000 people, 25,000 people, who's a director of technology services as to how to go about this? I mean, I imagine this person is sitting there in a position where they're frustrated with their access, and they're afraid that the city council's going to laugh at the idea of spending millions of dollars on a broadband system.
Dave Williams: Well, actually, I wasn't in that boat. It was the other way around. Our city commissioners brought this project to us and said, "Find a way to make this happen." If you find yourself in a position where you're trying to sell this, I'd encourage you not to look at it as a money making opportunity. Look at it in terms of quality of life. That's the real selling point to broadband is it is a quality of life issue.
Dave Williams: Anymore, access to the internet is really not an option. Now how you get to it might be an option, but needing the access is just becoming more and more of an everyday... We need it like we do energy and water anymore. Cities that don't have it are likely to wither and die.
Christopher Mitchell: That's what we're hearing from the people we're talking to as well. Well, let me ask you if there's anything else that I should've asked you or any points you wanted to make regarding the network.
Dave Williams: It's been a whirlwind project. We thought about it and kicked the idea around for five or six years, and when we pulled the trigger on it... If you find yourself in a similar position, just be ready. It gets out of control fast, in a good way.
Christopher Mitchell: Is that just the sense of all the possibilities, there's just so many details to keep track of? What makes it get out of hand like that?
Dave Williams: A little of both. A lot of decisions have to be made very quickly, but they need to be made very carefully because you make a wrong decision, it's expensive to change your mind later on. Broadband utility services are not like the traditional utility service. I mean, at a high level they are. It's a huge investment upfront, but you can't depreciate the full cost of that out over the 20 to 30 year lifespan that you can on a water plant or an energy plant. About half the costs of broadband are tied up in electronics, and they're going to have a five to eight-year refresh cycle. The fiber plant itself might be good for 30 or 40 years, but the electronics driving it, you've got to keep up with the times.
Christopher Mitchell: Right. I've actually known those numbers for a while, but I never really thought about how that's significantly different from the other sorts of heavy infrastructure that we're used to. Well, Dave, thank you so much for taking the time today. I really appreciate the opportunity to learn more about Ponca City and look forward to checking in a few years to see where you're at.
Dave Williams: All right. Thank you.
Lisa Gonzalez: That was Christopher with Ponca City Director of Technology Services, Dave Williams, discussing the community's Wi-Fi and fiber-to-the-home networks. We followed Ponca City throughout the years, so check out more stories about the community on muninetworks.org. We have transcripts for this and other podcasts available at muninetworks.org/broadbandbits. Email us at email@example.com with your ideas for the show. Follow Chris on Twitter. His handle is @communitynets. Follow muninetworks.org stories on Twitter. The handle is @muninetworks.
Lisa Gonzalez: Subscribe to this podcast and the other podcasts from ILSR, Building Local Power and the Local Energy Rules Podcast. You can access them anywhere you get your podcasts. You can catch the latest important research from all of our initiatives if you subscribe to our monthly newsletter ILSR.org. While you're there, please take a moment to donate. Your support in any amount helps keep us going. Thank you to Arne Huseby for the song, Warm Duck Shuffle, licensed through Creative Commons, and thank you for listening. This was episode 391 of the Community Broadband Bits Podcast.