Transcript: Community Broadband Bits Episode 433

This is the transcript for Episode 433 of the Community Broadband Bits Podcast. In this episode, Christopher speaks with Stacy Cantrell, Vice President of Engineering at Huntsville Utilities in Alabama. They discuss the network's partnership with Google and how it leverages fiber for other utility service to save resources and residents money. Listen to the episode, or read the transcript.

Stacy Cantrell: We're going to continue to see more and more benefit from this now that the build is substantially complete, we're really starting to be able to use it. So we're just now really seeing the benefits from it.

Ry Marcattilio-McCracken: Welcome to episode 433 of the Community Broadband Bits podcast. This is Ry Marcattilio-McCracken here, at the Institute for Local Self-Reliance. Today, Christopher talks with Stacy Cantrell, Vice President of Engineering at Huntsville Utilities in Alabama. Huntsville is a large metro area, and Huntsville Utilities serves well beyond the city boundaries. Their municipal electric department built a major network that gets close to every house within the city limits. Providers, of which Google will be the first, can lease that network and attach homes to it. But Huntsville Utilities also uses that network for internal services, bringing value to those living in the city. Stacy shares with Christopher that they just finished the project and would do it again given the benefits they're seeing. Now here's Christopher talking to Stacy.

Christopher Mitchell: Welcome to another episode of the Community Broadband Bits podcast. I'm Christopher Mitchell at the Institute for Local Self-Reliance in St. Paul, Minnesota. Speaking today with Stacy Cantrell, the Vice President of Engineering at Huntsville Utilities. Welcome back to the show, Stacy.

Stacy Cantrell: Thanks Chris. Glad to be back.

Christopher Mitchell: I think we talked to you many years ago, I'm sure it seems like a lifetime ago, when you were starting this project. And now I'm very excited to get a sense. In the email, I joked that I know it's not over, these things always have something that wraps on. But if you don't mind, tell us a little bit about Huntsville to start and what you're doing down there.

Stacy Cantrell: Okay. Well, I've been in Huntsville for about 25 years and it's it's a great place. If you haven't been to Huntsville, it's a great place to visit. A lot of growth has continued in Huntsville, even when other areas have slowed down. So if you haven't seen us in the news, in recent news, SmartAsset listed us as a number two in places for career opportunities during the pandemic, and Business Facilities has listed us as number one best climate for small business.

2:21

Stacy Cantrell: So we continue to move along and thrive. Huntsville is actually the county seat of Madison County. So we are lucky to be surrounded by equally fun and productive areas. So within a 50 mile radius, we have about 1.2 million people. Huntsville has access to state parks. We've got a very active land trust. So there's a lot of hiking and mountain biking and outdoor activities to do around here. Also, very thriving arts and entertainment and outdoor venues are becoming very popular, outdoor music and other entertainment. And we've got a lot of micro-breweries popping up and local coffee shops. So it's just a fun place to just be and live and, and walk around. We're certainly known as a thriving technical community.

Christopher Mitchell: Yes, a very advanced technical community.

Stacy Cantrell: Right, right. In fact, I've heard, and I didn't verify this stat, but I've heard that we have one of the highest concentrations of PhDs. So when you see folks walking around here, they are rocket scientists.

Christopher Mitchell: Right.

Stacy Cantrell: A lot of them are. But we've got a great education institutions here, University of Alabama, Huntsville is here, University of Alabama A&M, Oakwood College, just to name a few. So there's a lot of educational opportunities here. Redstone Arsenal is here. Wernher Von Braun's offices were here back in the day. So still a very, very significant space presence here. So it's just a very diverse and fun place to be.

Christopher Mitchell: And Huntsville Utilities serves, I think if I remember from reading recently, about 105,000 households and roughly 200,000 people, I think. So the metro is quite large, but in Huntsville's obviously one of the larger cities that has a municipal network, but there's quite a bit of growth around it, then more than I'd realized.

Stacy Cantrell: Sure, sure. Yeah. Huntsville's not sitting here by itself. There are other dense and thriving areas right around this. So the 105,000 number, that you remember from our Google contract.

Christopher Mitchell: Oh, okay.

Stacy Cantrell: We actually serve more like 200,000 electric customers. So we are the electric, water and gas utility in Huntsville. Those three footprints are a little different. They're not all the same.

5:04

Christopher Mitchell: Of course, that would be too easy.

Stacy Cantrell: That would be too easy, yeah. Electric, we serve all of Madison County. So we have more electric customers than anything. And there are, I believe, over 200,000 people actually in Huntsville projected right now. We've got a little bit smaller gas and water footprints. They're more lined up with the city limits, but not exactly. And then our fiber is basically to serve all of our utility needs. So it can pretty well follow the electric footprint.

Christopher Mitchell: Okay, let's talk about the fiber then. You have a model that, it was quite innovative. There's there's people whose feelings are hurt if it's a claim that you were the first to do this. So let's say that you iterated on a model that a few had used in which are owning the network, and then you've built out to mirror a lot of your meters, but not actually to them. You build in the neighborhood and then that can be extended to the line by Google. But give us a sense, as you were contemplating that investment, what was the network fully envisioned to be used for?

Stacy Cantrell: As you said, it seems like almost a lifetime ago that we were talking about this and certainly it's been some long years. But at that point, before we launched our initiative with Google, we had been building a substantial amount of fiber, really just with our electric system, our electric employees, and a couple of guys that had learned how to splice. So we were trying to get to our substations, to our water plants, to different facilities. We had started working, or had been working with the city of Huntsville, to get to a lot of their facilities. So we had about a hundred miles of fiber that were being used by different people in Huntsville Utilities and the government. And knew that it was time to put some structure around that and that, okay, we are seriously in the fiber business now, we need to act like it.

7:14

Stacy Cantrell: So we were also getting overwhelmed with requests to get dark fiber to different facilities. But by the city, somewhat from the county, certainly internally, as we start to have bigger and better security systems and camera systems and things like that. And we want to connect those in with fiber. So we took a step back and we were trying to figure out what's the best way to build this out strategically. And it's, as most people know, a very expensive asset to build. So we needed to plan carefully how we were going to use our funds. And about that time, Google approached us or actually approached the City of Huntsville with wanting to do a very substantial lease from us. So that really gave us a big influx of cash to build this network. So it really worked well.

Stacy Cantrell: It's been a painful build, and at first it was because it takes a while before you get it to a point where you can really start leveraging it. But we've been able to put more and more of our Huntsville Utilities facilities on this network, we've been able to put more and more of the city's facilities on the network from traffic cabinets, bus stations, just different administrative buildings, a variety of things for the city. So that's really, we've gotten to a point now that we've finished that initial Google build, we can really leverage the asset that's out there. And we're doing that every day, connecting out more and more things. So it's exciting to see what we're going to do with that.

Christopher Mitchell: I'm surprised to hear you say that the fiber is expensive, given all the other things you do. And so let me ask, if you're building out to a new subdivision, for instance, that was being built, wouldn't fiber actually be the cheapest of all the infrastructures that you were extending?

Stacy Cantrell: As far as the actual cost, the fiber cable itself is fairly expensive. But you're right, the fiber build probably is not that much more expensive than building electric or water or gas when you consider all of the effort that goes into the trenching and the cost of the pipe and all that. It's probably in line with electric, maybe more expensive than water or gas since there just aren't as many parts there.

9:45

Stacy Cantrell: But one of the things that happened, and most utilities know this, developers usually contribute in aiding construction to help offset the cost. That's not done in this case. The fiber is really part of our system for our use. So we pay for all of the construction of that. So to us, it's more expensive because it's not immediately offset by a contribution. Now, we do recover our costs over time by leasing those services and in the value that it provides to the utilities in not having to lease services from other providers and in the value that we get out of the different systems that we put on there, our SCADA systems, security systems, backup, connecting our two data centers together and that sort of thing.

Christopher Mitchell: That's really useful to know how that works for your accounting. I was just speaking with someone, Lee Brown, from Erwin Utilities in Northeastern, Tennessee. And he was mentioning all the benefits they're getting from their fiber, for the water system, which I feel like five years ago, we weren't really seeing the fiber as useful for water utilities. And so it just made me want to ask, are you seeing a lot of benefits with having built this fiber network when you attach water system to it?

Stacy Cantrell: So we do. In fact, our water plants and boosters are one of the primary things that we want to connect up to the fiber and mainly that's for SCADA so we can monitor a levels and pumping and that sort of thing. So our water system is definitely a heavy user and benefactor of the fiber system. Before I forget, I'll make one other comment about cost. The other thing that makes this seem more expensive is it's just a matter of scale. We've built our electric systems, over time, a little bit at a time. And then all of a sudden, we launched to build 1100 miles of fiber network, partly overhead, partly underground. So it's more compressed. So it's a larger capital outlay at one time, whereas our other systems are just gradually built. So it makes a difference.

12:06

Christopher Mitchell: Yes. Yeah. The inheritance and the next generation will not think of the fiber as being nearly as expensive as you do, I'm sure. So regarding, where does the fiber go? Is it following your entire electric footprint over time, or where is it today and where's it going?

Stacy Cantrell: So our ultimate goal would be to have it follow all of the electric system and actually even more than the electric system, of course, even our water and gas facilities or other facilities that they also have are connected to the electric systems. So ultimately, it would completely follow the electric system. We have built out, substantially built out, City of Huntsville. And that's because that was our agreement with Google was to build out that part of it. Now, as far as expanding beyond that, we talk about that. I think we're all a little bit tired and want to regroup before we kick off any other large build. But certainly we'd like to get more out into the county to get to our other facilities that we have out into the county. And as we start to have more and more smart devices, certainly we want to have a further reach. It's also played a big role in our AMI system, which does cover our whole county, but there's significantly more build if we want to cover our whole footprint,

Christopher Mitchell: How are different entities using it? You mentioned already the city's using it. We know that Google is leasing the entirety of it or close to it. But what have you heard from others in terms of their interest?

Stacy Cantrell: So others, as far as other utilities or other potential customers?

Christopher Mitchell: Actually, I'm really curious about anyone that wants to give you money to use the fiber in any capacity. So if you want to break that down a little bit, I'm just curious, because I imagine there's probably some. I don't know if Verizon or others want to use it to get to their small cells or things like that.

Stacy Cantrell: Sure, sure. And it's certainly available for that. And we've had a variety of different companies call and inquire. And a lot of that is under NDA, so I won't go into who or to what extent we're having those discussions. But just a reminder, we lease dark fiber. That's all we do. We give you a pipeline from here to there. So we've learned that that's not always attractive. Some folks don't want to pay for that huge pipeline. They would rather just lease some bandwidth. And we're not in a position to do that right now. So it's really offering some smaller businesses than what they really need and more than they need to pay for.

14:54

Stacy Cantrell: But we've had larger customers that they are interested in leasing one, two fibers, point to point. Maybe they want to be on our backbone ring, which somewhat circles the city and connects our six different huts together. So entities can lease co-location space from us and put equipment in those huts. They can lease the whole ring and have equipment in multiple huts and provide some redundancy. So it's really larger type users that we probably can best serve right now, and we do see a lot of interest in that.

Christopher Mitchell: For okay to have an idea. Does that mean Fortune 500 companies, Fortune 100 companies or just ISPs? What kind of entity would we be interested in that?

Stacy Cantrell: So certainly data centers would be interested. Certainly anybody that wanted to do small cell would need fiber. Now they may opt to use fiber they already have, or certainly leasing from us is an option. So they're just options and we're just starting to see small cell really emerge in Huntsville. So it'll be interesting to see how that actually pans out. But with the city, again, it's just a pipeline for them to connect all their different facilities back to their data centers, to have backup data centers, that sort of thing. And I think that's what other folks are using it for too. It's a nice web of a network. If you've got a lot of things to connect, it's very useful for you.

Christopher Mitchell: Yes. And I would expect that we'll see more of that as people figure out how to, it's always a little bit of a learning curve, even for knowledgeable folks. I want to talk a little bit about how things went in terms of your expectations versus things you might give advice to others if they're contemplating something like this.

Christopher Mitchell: One of the things I wanted to note was a quote that I'd found from Mayor Tommy Battle when this was being started, because when a utility makes this kind of investment, often there's a storyline or an expectation that it's a community that is poorly served by existing providers. And nobody said that about Huntsville. Huntsville was already fairly well-served by the standards of United States cities. And he said, "If Huntsville is to remain a technological leader in this hyper-connected global world, we must be able to offer broadband access that can accommodate the growing demands of business, research institution, entrepreneurs, residents, and public safety." And I think more cities are developing that mentality now. But I'm curious, are you where you'd hoped you would be in terms of all of this work these three long years? Have they been worth it?

17:49

Stacy Cantrell: Yeah. They've been worth it. So people often ask, "Would you do this again?" And yes, I would say we would do this again. This has been good for Huntsville Utilities, it's good for Huntsville and the area. We're going to continue to see more and more benefit from this now that the build is substantially complete. We're really starting to be able to use it. So we're just now really seeing the benefits from it. There's been a lot of work these past three or so years, three or four years. And a lot of ways we were too busy doing the build to actually enjoy the benefits. So now we're getting to where we can really use the network and see some benefit from it.

Stacy Cantrell: As far as are we where I thought we would be, we had all hoped to be finished with the build sooner. But realistically, I think we've done about the best we could. 1100 miles is a lot to build in an urban setting. A lot of underground, a lot of concrete, a lot of things already in the ground, a lot of old things in the ground. There's a lot of older Huntsville infrastructure. And sometimes you just don't know what you're getting into until you start digging it up. And so certainly construction is a messy, difficult business. And if you try to rush it, you're going to have problems. So I think we've done well. We've had great folks at Huntsville Utilities managing this construction. Obviously, we use contractors to build this. It's such a large bill in such a short time, but we've got a great fiber operations team here at Huntsville Utilities that was able to keep a watch on that and manage it.

Stacy Cantrell: What I wish we could have done differently when we were having our negotiations with Google, this was so new to us and still, we're not communications experts. But a lot of this was so new, we were kind of learning our way as we went. So I felt like we jumped off without fully knowing everything we needed, but sometimes that's the only way to get it done. You just have to get started and do it, ready or not. You're not going to get ready until you have a reason to get ready. So looking back, I wish we'd been a little more prepared as far as understanding our make-ready issues and maybe going ahead and getting started on make-ready issues. But overall, it's done and it's worked and we're glad to now be the proud owners of a fiber network.

20:41

Christopher Mitchell: Can you share with us any technical processes that started off one way, and then you realized it would work better another way that try and prevent others from making those mistakes?

Stacy Cantrell: So I don't know that we substantially changed anything technically. As far as our architecture, we knew how we were going to build our backbone with the huts and then the passive network out from the huts. And we knew how our split ratios were going to be. So we didn't substantially change any of that. We made a few tweaks to materials.

Stacy Cantrell: One of the things that comes to mind, we used a multi-service terminals, MSTs, pretty common. So there's a tail or a service cable connected to that, that goes back to your ISC. Now those tails come in different links in their connector eyes for easy construction. We were ordering those as a combination. So you'd have all these MSTs with the tails. We started ordering those separately. So you have the MST, and then you just have the different links of tails and get helped a lot with our stocking and with flexibility of installation. So that's really a minor thing. Something that we didn't know much about at all were communications huts. So we went through a lot of learning, and obviously we use consultants to help us with that. But understanding what we would need to build and how we would need to build it, I wish we had known more about that before we got into it, just so that it would have been smoother.

Christopher Mitchell: The term hut seems kind of funny when I'm assuming that this is something that's built to withstand a tornado. It's built to be cool in your summers and things like that. So it's quite a bit of engineering, I'm sure.

Stacy Cantrell: Yeah. I like it because I always picture a little Tiki hut.

Christopher Mitchell: Right.

22:36

Stacy Cantrell: But it's not that at all, you're right. It's built to this sustain a certain force winds and impacts and has redundant air conditioning, has natural gas generators for backup and all sorts of things. So yes, it's certainly not a hut.

Christopher Mitchell: So one of the things that I have to assume is that people are coming to you and trying to figure out if they should do something like this. Is that an experience that you're having?

Stacy Cantrell: It is. We've had really, since we launched this and it became public and newsworthy, we've had a lot of different calls from utilities. And utilities, especially we're in public power, so we share. We like to talk about what we did, what we learned, and we ask each other, "Hey, you did this. What did you learn?" So that's the nice thing about public power is that we all try to learn from each other.

Christopher Mitchell: Well, you in particular seem to be trading folks with Tennessee fairly regularly with TVA and other utilities within Tennessee.

Stacy Cantrell: That's true. So within TVA, we have even a stronger bond I think, and more sharing. So we've had quite a few municipalities and municipal utilities call and ask us, "Tell us more about how you did this. What'd you learn? What would you do differently? How did you fund it?" Different things like that. A lot of cases, they are actually looking at providing broadband services themselves. So that's a main difference in us and what a lot of folks are looking at.

Christopher Mitchell: I'm surprised to hear that. I'm sorry to cut you off, but my impression would be that at this point, in almost 2021, if you're not offering service directly yet, you've probably been looking for a partner. But it sounds like the folks you're talking to are more, they are still interested in doing it themselves.

Stacy Cantrell: Well, and I think a lot of those folks too, and you mentioned Huntsville can't say, "Oh, we were the underserved community. So we had to build this." But there are a lot of underserved communities, a lot of municipal and co-ops utilities have very rural areas. So they are probably better suited to provide a broadband service then already having a pretty significant infrastructure. So maybe that's where that comes from. I can't say for sure that all of the folks we've talked to have been rural or underserved areas because I believe we talked to some more urban areas too.

25:19

Christopher Mitchell: And then actually, I wanted to get this in earlier and I forgot, but the estimate that I saw had thought that the build for your side, your electric department owns the fiber that you've deployed and then you lease that core to Google and others who are interested. And it was estimated that it would be about $70 million. And I'm curious if that was more or less where you ended?

Stacy Cantrell: So overall, no, it ended up significantly higher than that. And part of that was a lot of the make-ready work that we didn't anticipate or had underestimated the cost of that. When you look back at the cost. Other folks in the industry say, "Well, no, what you built is pretty much in line with what we would expect." So I think we simply underestimated all that was involved in the build.

Christopher Mitchell: Yeah. That makes sense because I was thinking 70 million would be quite a bargain.

Stacy Cantrell: Yeah. Yeah. And we might've been able to do it in more of a rural setting, but with all the issues you deal with a more dense in an urban setting, it just inflates the cost.

Christopher Mitchell: Is there anything else that I should ask you about before we wrap up?

Stacy Cantrell: I can't think of anything else. We've already covered, like I said, what most people will ask is, "Would you do it again?"

Christopher Mitchell: Right.

Stacy Cantrell: And that seems to be the tell all question. And yes, we would do it again. I would like to be able to do a slower build. I think cost could be managed better with a slower build and with a little more preparation time. We were kind of under the gun and did the best we could. And I think our folks did a fantastic job.

Christopher Mitchell: That's excellent. The group that I'd interviewed a few weeks ago, Lee Brown from Erwin, they had done an incremental build specifically for that reason. It kept the cost down, but they also then, you have to deal with that political challenge, someone has to wait six years, rather than three years to get their Internet turned on. So it can be a challenge too.

Stacy Cantrell: It's still a political challenge to tell somebody they're at the end of three years.

27:28

Christopher Mitchell: Right. I'm sure. Well, thank you so much for your time today, Stacy. I really appreciate it.

Stacy Cantrell: Thanks Chris. I enjoyed it.

Ry Marcattilio-McCracken: That was Christopher talking with Stacy Cantrell. We have transcripts for this and other podcasts available at muninetworks.org/broadbandbits. Email us at podcast@muninetworks.org with your ideas for the show. Follow Chris on Twitter. His handle is @communitynets. Follow muninetworks.org stories on Twitter. The handle is @MuniNetworks. Subscribe to this and other podcasts from ILSR, including Building Local Power, Local Energy Rules and The Composting for Community Podcast. You can access them anywhere you get your podcasts. You can catch the latest important research from all of our initiatives if you subscribe to our monthly newsletter at ILSR.org. While you're there, please take a moment to donate. Your support, in any amount, keeps us going. Thank you to Arnie Huseby for the song Warm Duck Shuffle, licensed through Creative Commons. This was episode 433 of the Community Broadband Bits Podcast. Thanks for listening.