Sandy Damage Raises New Questions about FirstNet Use of Commercial Networks

FCC Commissioner Robert McDowell told us network hardening is important and must be weighed against the costs and time it takes to build a public safety network. “A public safety cell tower isn’t going to be any more hurricane resistant than a private sector cell tower,” he said. “The private sector has as much of an incentive to harden its networks as the public sector. So I'm not sure you would have seen any difference in durability between public safety specifications versus private sector specifications.”

"The strength of the network is the weakest link,” said Charles Werner, chief of the Charlottesville, Va., Fire Department. “One of the questions is can networks be utilized in a regionalized type system and ‘hardened’ ... as necessary to withstand the challenges that will be faced, and be able to address the capacity needed. This is going to be a very challenging and complex issue. While public private partnerships are desirable and will potentially dramatically reduce costs, the question remains: ‘Will they be able to fulfill the requirements of such a network?’ The fact that towers exist and backhaul is available, these components must be of such reliability that they can accomplish the required outcome."

Another second public safety official said he’s not concerned about FirstNet making use of some commercial facilities. “First off, a lot of this network is going to be data rather than voice communications initially,” the official said. “Secondly, in the truly broadband world, if there is an outage in one location you should be able to work around that so to speak. ... In any environment such as the sort of storm the East Coast was hammered with, no network is going to be 100 percent assured of survivability. That’s just the nature of nature, I guess. But if there is an outage, the ability to move information traffic around among those systems that are still up and running, the ability to move portable towers and COLTs [cells on light trucks], satellite services on trucks, to be able to move them in and power them fairly quickly in terms of recovery, capability is pretty good."

FirstNet may make use of commercial facilities, but will be hardened beyond the capability of carrier networks, said a third official. “It is my view and many others’ that FirstNet should take advantage of existing commercial infrastructure, but the details must be worked out and are very complicated.” Base stations, for example, “will be owned by FirstNet and not commercial, but may often be located at existing commercial sites,” the official said. “Existing commercial power supplies in some cases may need to be beefed up from battery only power to gasoline, propane gas or natural gas powered generators that are capable of longer term power than just batteries. ... Existing commercial infrastructure may need to be beefed up to include more redundancy and alternate paths to assure more reliable service."

Hurricane Sandy is a prime example of how commercial networks can fail, BayRICs Interim General Manager Barry Fraser told us, along with past examples in other disasters like the June derecho storm and 2011’s East Coast earthquake. “It’s a concern when we hear the FirstNet board [is] building this out to commercial standards,” Fraser said.

"Public safety officials are used to having their own dedicated hardened networks,” Fraser said. They often stay open even when commercial networks fail and are “built to public safety grade,” intended to endure bad weather and conditions fraught with peril, he said. BayRICs is a California Broadband Technology Opportunities Program grantee tasked with building out a broadband public safety network, one of seven projects NTIA suspended this summer in anticipation of FirstNet (CD Sept 24 p18). Commercial networks fail in various settings, such as Wednesday’s large San Francisco parade celebrating the Giants’ World Series victory, a setting with many people but few cell sites, he said. “That’s another concern -- that’s exactly the time our public safety folks need to communicate.” BayRICs is addressing its concerns in comments to NTIA, due Thursday, as part of its FirstNet notice of inquiry. FirstNet will need to rely on commercial networks to some extent for financial reasons, Fraser acknowledged. But BayRICs is encouraging FirstNet to rely on existing public safety channels for the core of the national safety network and use commercial networks to enhance that, he said.

But “almost all” of government runs on commercial networks, said Level 3 Government Markets Group Senior Vice President Edward Morche when asked about the many telecom failures the FCC reported, as well as the prospect of a FirstNet operating on similar commercial networks. “The issue is to make sure they spend money on the agency and the agency’s interests,” he said.

Level 3 encounters a spike in government agencies interested in the best ways to create a reliable network after natural disasters like Hurricane Sandy or Katrina, Morche said. Level 3 consults with more than 300 entities about the best ways to create reliable continuity-of-operations/disaster recovery plans, which he’s helped with for the past half decade. “We do tend to see more reactive than proactive activities,” he said of agencies, saying there’s often a “post-mortem” examination of failures after big disasters. But agencies need to demand reliable service of communications providers, he said.

"As an agency, you have that power,” Morche said. “We have to send that message more often.” Agencies should demand accountability and should pursue the questions if, say, a provider argues its route maps are proprietary -- that’s a “complete falsehood,” he said.

Level 3 promotes four questions that agencies should ask their communications providers, which Morche described: Can you demonstrate physical route diversity? Where do these providers’ fiber routes go geographically and are there multiple locations? Do you use carrier-grade communications or the less redundant enterprise-grade equipment? Have you created a service level agreement that encapsulates the first three questions and provides straightforward metrics for the service levels that should be maintained? Agencies should ask providers questions about their physical security, their cooling mechanisms and other strategic concerns, he said, describing “a job to educate people on what they needed” as part of the free consulting Level 3 provides. The questions all scrutinize potential “points of failure” for these networks and look at the “fundamental” issue of how a provider’s servers connect to the outside world, he said. Tight budgets and a shrinking number of data centers make the need for these plans all the more important, he said.

"One argument in favor of a public network would be that it is hardened for disasters,” said public-interest communications lawyer Andrew Schwartzman. “So maybe it would have 96 hours of emergency power, rather than 48, etc. But in a storm like this one, you can do only so much. The counterargument is that the money would be better spent hardening the public network, i.e., installing 96-hour emergency power on the public network. Who is right? I have no idea."

"Communications networks need to be more resilient in general, and especially if they're going to be used for” public safety, said Public Knowledge Staff Attorney John Bergmayer. “Quite a few cell towers were knocked out, and I would like to know how much of that is due to a lack of backup power."

Meanwhile, NTIA posted early comments on a presentation on the conceptual network architecture made at the FirstNet board meeting Sept. 25 by board member Craig Farrill. NTIA previously sought comment on the “framework for designing the public safety network architecture in a manner that leverages existing resources and infrastructure,” as laid out by Farrill (http://xrl.us/bnr7fp). NTIA said Wednesday in recognition of the storm it was delaying the due date for comments until Nov. 9 (http://xrl.us/bnw9it).

The Visiting Committee on Advanced Technology of the National Institute of Standards and Technology addressed network resiliency head-on. “Without question, communications in support of public safety must be reliable, especially under stressed conditions, including, for example, loss of power, loss of infrastructure and lack of operating personnel,” the group said (http://xrl.us/bnw887). “It seems appropriate to observe that this objective may be met not only through redundant provisioning but also through rapid deployment of temporary or even permanent infrastructure. Not only will first responders need rugged equipment but they will also need an ability to deploy auxiliary or replacement gear quickly, at need. The utility of common standards should be obvious in this context -- national, state and local-level caches of common equipment will be far more feasible if standards that permit interoperability can be established, adopted and applied."

RCC Consultants offered a detailed critique of the presentation. “The development of a nationwide wireless broadband system is a task that has huge inertia, and getting the process underway and on the right track requires the collective effort of many talented and dedicated people,” RCC said (http://xrl.us/bnw7wd). “Making sure that the system development is on the right track from the beginning is crucial or that same project inertia will make it immensely difficult to change the project direction down the road.” RCC said the “central merit of the Presentation is its candid recognition that the funds available under the [February law starting FirstNet] for the development of the PSBN may not be adequate to support the development of a stand-alone PSBN. That candid recognition of resource constraints is a key condition to any serious consideration of the development of the PSBN."

Doug Westmoreland, an engineer with the now-defunct New York Statewide Wireless System, said it makes sense for FirstNet to work with carriers. “The major issue with a large police radio network is availability and reliability. It’s very hard to take down individual radio system[s] that are scattered around a very diverse area,” he said (http://xrl.us/bnw7yh). “A national system must have multiple redundancies: multiple trunking lines, multiple redundant servers, switches, and server sites. The big issue with the associated commercial vendors is going to deal with the details of how reliability is implemented during normal and emergency situations. Who gets the bandwidth when something important is happening? How will the base stations implement emergency level priorities on the sites switching? How will the [FirstNet] developers know that sufficient testing of these emergency protocols is implemented correctly in a network that is shared? What is to stop them making a change in the base station and network code after the testing is complete? Who will approve code changes? How will an industry that is known for its speed to deploy new features deal with the need for increased security, availability, and reliability?”