This is Episode 207 of the Community Broadband Bits Podcast. In this episode, Bruce Patterson returns once again. This time, he explains how virtualization gives residents control on the open access infrastructure. Listen to this episode here.
Watch the video "Ammon's Model: The Virtual End of Cable Monopolies."
Bruce Patterson: The words that the public is going to use to say the same thing is virtualization. We are actually are taking this technology and bringing it to the customer edge and letting them access the same type of functionality from their home.
Lisa Gonzalez: Welcome to Episode 207 of The Community Broadband Bits Podcast from the Institute for Local's Self-Reliance. I am Lisa Gonzalez. We have had Bruce Patterson, Ammon, Idaho's IT director on the show before to talk about the city's fiber network. This time he is joining Chris to discuss recent developments as the city gets down to the business of building out its Fiber-to-the-Home network. As well as implementing a unique funding approach, Ammon is using its open access infrastructure to give users more control than they would have with the typical connection. In this interview, Bruce provides more detail about the plan and how Ammon is moving forward one neighborhood at a time. Be sure to check out our new video "Ammon's Model: The Virtual End of Cable Monopolies," to learn more about their journey to become a network of the future at MuniNetworks.org. We have a number of stories that cover their progress. Now here is Chris and Bruce Patterson, IT director at Ammon, Idaho.
Christopher Mitchell: Welcome to another edition of The Community Broadband Bits Podcast. I am Chris Mitchell and today I'm speaking again with Bruce Patterson, the technology director for the city of Ammon in Idaho. Welcome back to the show.
Bruce Patterson: Thanks for having me, Chris.
Christopher Mitchell: I am really excited to talk to you again. We have just completed this long, less than a year odyssey, but longer than we thought it would be to make this video about your project. Tell me how you describe to people what you are doing in Ammon?
Bruce Patterson: That can be a bit of a challenge at times. The simplest thing I tell them is that our focus is on the infrastructure, meaning we are focused on the fiber and trying to give residents and businesses in Ammon their own fiber and a sense of ownership over the that fiber so that they can do the things they want to do with that piece of infrastructure.
Christopher Mitchell: For people who are not familiar, this is infrastructure that will actually be owned by the city of Ammon, but it's the way you are implementing it that will give people a sense of ownership in terms of how they can use it, right?
Bruce Patterson: That's right. We often hear open access and that is condensed down from this expression "open access networks," which implies people that are users within that system can access multiple networks. We have taken it a step further and we actually allow the user to create their own networks, which means a user can technically become a provider and infrastructure itself is open to them to do innovative things.
Christopher Mitchell: So you end up with multiple virtual networks on top of one physical network?
Bruce Patterson: That's right. To the end-user, it appears to be that they have their own virtual fiber to do with as they would like.
Christopher Mitchell: I am excited to see what turns out as you turn more people on. How many people are signed up to date?
Bruce Patterson: Last count, last solid account we had was 188 out of  homes. Our first project encompasses  homes, so we were at exactly 50%, but we have had a number sign-up since then and we are still in the process of collecting all of that data and trying to find out where our total is. If you were to ask me today where I think we are at, I think we are at 200 out of that  and climbing from there.
Christopher Mitchell: You have already had several pilot people that have been walking through the system for a while, as well as your network is connecting a lot of businesses in the schools and that sort of thing, right?
Bruce Patterson: That's right and the same technologies are being used. What's a little bit different with these homes that are starting to come on the day is we are actually giving them a user portal that allows them to interact directly with the network infrastructure and do the things that we are describing here.
Christopher Mitchell: That's actually worth just mentioning I think. As you move forward service providers could get on that portal, but you also have service providers that have been offering service without using the portal.
Bruce Patterson: That's right. It's something of a business model transition, if you were. I think the city's interest in maintaining the ability for outside businesses and entities to come in and negotiate contracts. What we want for our residents within the city and our businesses within the city is a true utility rate, meaning it's ubiquitous and the cost is very low to maintain and operate the infrastructure because everybody is paying into it and we should all get the benefit from it.
Christopher Mitchell: For people who might be a little bit confused, definitely recommend that you watch the video before you listen to this podcast because I think we are definitely going to assume some basic background knowledge. The rollout is being accomplished with a local improvement district. When we had talked for the video, we were not sure what those costs would look like. Can you tell me if I am a person signing up, what kind of costs I am likely to pay and how that's broken down?
Bruce Patterson: We had to make our best guess going in and we felt $3000 was a reasonable target if we could get 50% sign-up rate, meaning out of every 10 homes, five in the area would hook up. We think that's a doable number based on the backbone infrastructure we already have in place. Because we already have that investment, there is no need for the residents to pay for that. They are really just paying for an installation of a fiber from the edge of their neighborhood into the heart of their neighborhood to their home, some installation of some equipment. We think that $3000 is a good number. As we are working through the process, everything we have done to date supports that number. We expect to be able to validate it completely over the next couple of weeks.
Christopher Mitchell: We have had a few questions from people as we have written about this model. Can you just remind us exactly how the fees work? There is a one-time fee to get the fiber on the side of my house and that's as you mentioned, it's in the order of $3000 if you hit the targets what you are forecasting in terms of the number of people signing up. That can't be broken out over many years of payments. What's the second fee that I would have to pay on an ongoing basis?
Bruce Patterson: The second fee is the maintenance and operation of the fiber system itself and that is done by the city and we will do that to a utility rate, meaning you get a bill from the city for sewer, water, or garbage pickup, included in that if you decide to join the utility, you will receive a bill for $16.50 a month if you want a one gig connection. We do offer a ten gig is a resident desires that.
Christopher Mitchell: If I started paying that and then I stop paying it, what happens?
Bruce Patterson: Basically, if you don't pay the utility service, then that affects our ability to operate and maintain that fiber. We have power that we have to supply. We do locates. We do all these things to maintain the system. What happens is we will not maintain that fiber for you and we basically turn it off. It is still there at your home because you paid for it, but we are not getting paid for maintenance and operations. Sometimes that's actually done voluntarily, Chris. We allow residents to fill out a paper form and say, "I am leaving for a long trip." Maybe they are going to be out of town for six months or a year and they simply want to not pay that utility bill. They can voluntarily have it turned off and we will not bill them for the period of time they are gone. When they come back, we charge a $45 fee to turn that back on for them.
Christopher Mitchell: I guess I am curious. What happens if I decide I don't want to opt-in into his local improvement district, but then two years later I change my mind and I say, "Hey, I do actually want a fiber?"
Bruce Patterson: At that point, we take a look at what it's going to cost to connect that home and because you have not taken advantage of the option of doing it at scale with others that helped to defray the cost, it's likely going to be a little bit higher than what everybody else has paid to come into it and we can't amortize it, meaning the bond process is closed at that point. We can't find a way to take that debt, attach it to the property and spread it out over 20 years at low interest rates. What we would require is that you would end up having to write us a check for probably $3,300 or $3,500.
Christopher Mitchell: If I understand correctly you have actually given an assurance to the people who do sign-in that people who connect afterword are not going to get a better deal, so there is no incentive to wait longer?
Bruce Patterson: That's correct. That's the feeling of city council and I think all of staff and the residents themselves appear to agree that the initial investment is made by those of them that opt-into the system. There is a general feeling that nobody should be allowed to come in any cheaper and we agree with that. Frankly, the economics would indicate that it's not ever going to be cheaper than it will be today.
Christopher Mitchell: One of the things you had mentioned previously, and we actually talked about it in our last podcast, but I think it's so important I wanted to touch on it again is the lifeline idea. I have paid my $3000. I am paying my maintenance fees. I had been paying an ISP to get a gigabyte service and maybe some other ISPs for other kinds of services, but I lost my job and I am trying to cut expenses. Tell me about this lifeline feature that the city is going to be offering other network.
Bruce Patterson: The city will allow a resident that is not taking service from an Internet service provider, a commercial Internet service provider to tap into some Internet bandwidth that this city maintains. We are calling that a lifeline service. Basically the way that will work is the city can login to their user account on the city's portal and rather than signing up with a commercial provider and paying a certain fee per month for that Internet bandwidth, they can go to the city under lifeline service, ask, request for bandwidth and it will be given to them in 45 minutes increments, meaning they are basically allowed to have 25 Mbps Internet service for 45 minutes at which time it will time out. They can, of course, go back in and request another batch of 45 minutes and another and another. We will put a monthly data cap on it. At this point we are not certain exactly what that is, but our intention is as you say, people that for some reason are disconnected from an ISP, maybe it's economic or maybe there is something wrong with the service they are getting through their ISP temporarily or other things happen, they have another outlet they can get instantly to be able to connect in a time of crisis or in a time of need to meet those needs and let their kids still do their homework, let them still apply for jobs and so forth.
Christopher Mitchell: Yes, it's kind of funny. When we started recording this interview our Comcast service here at our office went out and it sure would have been nice where I had a portal where I could have just gone and quickly hooked on a different ISP to get through it. You might have one ISP that has an outage that's unrelated to the local fiber and you would be able to route around that's really quickly, like the Internet is envisioned to do. The last question I want to pursue on this topic is the city is not going to be an ISP on the network other than this lifeline service and perhaps some public safety services, but I just want to verify you are not actually going to be selling Internet service to customers?
Bruce Patterson: No, this is intended to be a lifeline service. For those that are down in the weeds clinically to understand, it's a temporary connection. They would not even receive their own dedicated public IP addresses. It's really a temporary connection, but it will meet all the needs they would have to be able to do the things that they need to do for critical functions day-to-day life because we are all starting to see the Internet as essential to our everyday life, whether it's banking or as you mentioned job applications or our kids being able to get information to do their schoolwork. Those are important things that we want our residents to be able to do even when they might be a little bit strapped or going through a difficult time.
Christopher Mitchell: I want to turn this idea of software defined networking, which is really what powers your active shooter application that we talk about in the video. It's what powers this customer portal. What is software defined networking?
Bruce Patterson: The word that the public is going to use to say the same thing is virtualization. It's the idea that you take a physical asset and you can split it up and use it multiple times virtually. That's something we have seen in the data center world for quite a period of time. It has made its way into networking. It's not widely adopted by carriers, metro carriers, but it is something that we are seeing being implemented. It's something that oftentimes is implemented by carriers to help them manage their own networks. In other words, they make these virtual connections to deliver services and they can create one network for an Internet service and another network for their phone service and they can create another network for their TV service as it were. We are actually taking this technology and bringing it to the customer edge and letting them access the same type of functionality from their home.
Christopher Mitchell: What does that mean in terms of if you were to compare yourselves to another city that was building a fiber network and offering a gig and whether they were doing open access or not, what are you doing differently? What's different inside of your network? Is it just different technologies? You are using the same fiber, right?
Bruce Patterson: It's the same fiber and a lot of the equipment is the same. As you look at the vendors and manufacturers that make fiber optic equipment, many of them support these types of technologies. They are just a little bit off in some of the ways they apply them, but the functionality is largely the same. The idea is in the traditional method of setting up a network when you have multiple pieces of equipment and they span a large geographic area. What happens is you have to go to each piece of equipment, whether you are sitting there physically or you remotely login to it and you have to set up each connection individually and you say, "Take this incoming port and connect it to this outgoing port and create this virtual connection through it, so that it looks like a single fiber from point A to point B," and that's how you get it to pass through all of these pieces of equipment. This idea of software defined networking takes the control and puts it in a central location and allows you to just dynamically say, "Give me a connection from here to here," and the system itself goes in and makes those changes dynamically without having to touch, physically login to each piece of equipment as it were.
Christopher Mitchell: I think that's the view from the network operator. My understanding is that the ISPs find this really important because it allows them to do the things they might not otherwise be able to do. In particular if you look at a future of open access in which you define open access as the ability of multiple ISPs to connect and offer services concurrently, at the same time to home users. I have heard some that are big proponents of software defined networking say that it is not really possible to do that unless you are using software defined networking.
Bruce Patterson: I think it usually simplifies the task because as you can imagine to bring on a new provider if you are doing all these things manually is an enormous task in labor, whereas if you already have software defined networking or virtualization set up in your network, it's a very, very easy to bring a new provider on. I agree with you 100%, Chris. That's really the secret is how you implement the software defined networking or this virtualization and that's a choice each network operator has to make about how they want to do it. Again, I think where we are a little bit different than most anybody else in the country is we are actually using it for our own benefit internally, but we are also taking that same functionality and turning some of it over to the edge user. From your home you have some ability to say, "This is my provider. Oh and I want this service as well and I want these things." You don't have to call us and as us to provision it. You can provision it yourself.
Christopher Mitchell: What are the security implications of that? When you have multiple networks, multiple virtual networks on the same physical network, how can we be sure that one isn't spying on the other?
Bruce Patterson: There is basic security protocols and procedures that every operator needs to implement. I will make this comment that I think the things we are doing are actually more secure than the traditional method. To prove that you could take your cable modem if you are on a cable network and go to your neighbor's house and plug it in and you will notice how it works no matter where you are at, as long as you are plugging into their system because their system is not aware of where you are at graphically. Our system, on the other hand, if you were to take your edge device from your home, go to your neighbors and plug it in, our system would tell us that you had moved it and it would not connect you. We do have some security things that are in place that are dynamic and a little bit different. Again, it's the software defined the technology that allows us to actually do some of the specific security procedures. There is actually some pluses to the methods we are using not negatives.
Christopher Mitchell: When you say "edge device" you mean the ports, the actual where the fiber comes in and that's the physical piece of hardware, not like a computer or something like that?
Bruce Patterson: That's right. That's exactly correct. It's the piece of equipment that we bring our fiber into that converts it from optical signal to an electronic single, like you will typically see so you can cross a normal copper network cable. Device then if you were to take that and try to move it, we would be aware of it and it would not work in another premise or another location. That is not true with your DOCSIS cable modems that you get from your cable providers. The phone company, they are typically aware when you move your DSL modem because they register that with your address. There are some nuances here and I don't want to get into all the technical details. I am simply pointing out that SDN I believe implemented correctly improves the security. It does not reduce it.
Christopher Mitchell: I have to assume that with all these whiz bang features that are coming out, are you paying a premium on your network to implement SDN?
Bruce Patterson: There used to be a significant cost difference to implement these types of technologies, but may have become pervasive and most of the standards and protocols are implemented in the core, in the access equipment. The equipment that's actually housed within the switch points in our network typically have this functionality, so you might pay a little bit more but it's an incremental increase. It's not exponential, so you might pay 10% more for that. There is some cost distinction for the device at the home because as I mentioned to you, that's one of the ways we are different is we give this functionality to the actual homeowner or property owner to be able to do these things. For us to enable that functionality all the way to that edge at the premise, there is a slight cost increase for that piece of equipment at the home. Having said that, it's an interesting dynamic because if I were sitting in a position where I was starting from scratch today as a network designer-operator, I would look at it and I would say, "It costs me virtually nothing to put equipment through my core that enables this, so I am going to do that," and that gives me the flexibility later if I want to go down this path of doing software defined networking of being able to simply do it because then it's just software. It's not hardware, which means I don't have to buy new equipment.
Christopher Mitchell: That's one of the questions I have about virtualization. My understanding of virtualization is it allows you to basically get away from the high-end hardware and start to run it more on commodity hardware?
Bruce Patterson: That's true and we are seeing that point in the market space, at least when we buy our equipment, where we are now able to finally get to white box commodity hardware and we get to choose our software or our operating systems and firmware that are running on these switches from different vendors. As long as the standards match up, we could always go with the best price and so you are exactly correct, Christopher. Over the last year we have seen prices drop per report dramatically.
Christopher Mitchell: When you net it all out, do you think over the next five or 10 years you'll be saving money from going with this approach rather than a traditional approach one might have used five years ago?
Bruce Patterson: I think it's more future-proof. The first comment I will make is our core and access equipment is substantially cheaper than what we would expect to pay from a traditional vendor, so we can get white box solutions as you are describing. The piece at the home because there is more power there and it does more than the traditional model, it costs a little bit more, so that equalizes out. Where I think we are going to save is we think this equipment will last us much longer. When you look at the equipment change-out cycles that so many of the carriers go through, we go from DOCSIS 1 to DOCSIS 2 to DOCSIS 3 in the cable networks. We have gone from DSL to DSL 2 to VDSL within the traditional phone carrier networks. This is all virtual, meaning we think the hardware will last a long period of time and all the upgrades we are going to see are going to happen within the software, which comes at a very small price. That's where we think we are going to get major gains is through time.
Christopher Mitchell: One last question about a city that might think about this. I am curious. In terms of your expanding this to connect more homes and businesses across the city, given that you are focusing on the infrastructure and you have so much on automation and virtualization, what kinds of staffing considerations have you had to consider in terms of having enough personnel and the right personnel to be able to roll this out?
Bruce Patterson: We are in our infancy. I can talk about where we are at today and where we think we will be. We have yet to prove a lot of things, Christopher as you know. We feel confident on our course, but in answer to your question about staffing, I think we have a lot less staff than many, many operators do. We have very, very few outages and the outages that we do have are typically related to something at the premise that the property owner has control over or it could be something with the service. The chain of help ticket support comes from the provider for us. What I mean is the property owner calls their provider first. That provider helps them with that first tier of support. We get called if the provider cannot assist the property owner and typically that means the provider of the Internet service cannot see that premise or edge device at the address, which means there is a link down, which would be our problem. What we have found is we have very, very little time actually spent in support. With fiber optics, it's very stable. Our network is very stable. It just runs and operates. For example, last year I can recall one outage we had and it was due to a lightning strike that took out a piece of equipment on the side of a metal building. We changed that out and have them back up and running within 45 minutes. Other than that, I don't think we had a single outage last year.
Christopher Mitchell: That actually raises a point I was thinking about getting into and I was not sure, but just to touch on in terms of ISPs, one of the challenges historically in open access networks the way they were built without this virtualization and automation, ISPs would complain if they did not have visibility into the network and they were not able to troubleshoot as well. Is that something the way you are implementing the network gives ISPs greater ability to figure out what's wrong with the connection without having to call you?
Bruce Patterson: The great thing about virtualization is we can do whatever they would like. The device we put at the home allows us to create a virtual box, a virtual switch or router, whatever that ISP would like we can create virtually, which means there is no need for the customer or the provider to have to buy another piece of equipment, send it to that address, have them install it. We don't have to do that. We can basically create a virtual box, turn it on and the ISP can do anything they would like with that. Once they see that ability they understand. We can prove that we have a certain level of performance at the address. We can test and measure what the bandwidth is ourselves. What's been very interesting about this, Chris, is as we give them ability that's of course, a layer of complication I will call it, but it's a traditional way. Some of them have decided that this is crucial to their business model and they want that ability. Others have decided that they don't need that layer of complexity. All they want to know is can they see that edge point? They don't care about being able to do specific tests. They just want to know if it's up and if it's on. The way our system works, they can do that without having a virtual presence at the premise. Some of them have decided they want a VM and some have decided they don't. I think that's an indication of the market forces that are going to drive them towards least costs with performance that is acceptable to them and the customer.
Christopher Mitchell: Well, thank you once again, Bruce. I think you have given a tremendous amount of time to helping other people understand what you are doing and sharing your knowledge. I greatly appreciate that.
Bruce Patterson: Well, thank you, Chris, appreciate your time as well.
Lisa Gonzalez: That was Chris and IT director, Bruce Patterson from Ammon, Idaho. Don't miss our video on Ammon. You can see it at MuniNetworks.org and on ILSR's YouTube channel. Don't forget the transcript for this and other Community Broadband Bits Podcasts is that MuniNetworks.org/broadbandbits. Send us your ideas for the show. Email us at firstname.lastname@example.org. Follow Chris on Twitter. His handle is @communitynets. Follow MuniNetworks.org stories on Twitter, too, where the handle is @MuniNetworks. Thank you so much to the group Forget the Whale for their song "I know where you have been," licensed through Creative Commons. Thanks for listening to Episode 207 of the Community Broadband Bits Podcasts.