Illinois FCC LNP Field Test
Week 5 Summary
Summary
Week 5 of the Illinois FCC LNP Field Test (Field Test) consent rated on completing the second round of porting numbers and brought a new to LNP service provider into the FCC Field Test in an area that is already open to portability. Just prior to this second round of porting and continuing into week 5 a series of outages were experienced. These outages have slowed the progress of the FCC Field Test. Service provider number 5 has not been able to complete porting in of their telephone numbers due in large to the various outages experienced this week. For this reason the service providers were asked to detail connectivity problems experienced over the last two week’s. Individual service providers responses will be included in the report below.
The FCC Field Test remains on schedule for call through testing for three of the original four participant’s. The fourth original service provider has made progress with call through testing but has slowed to an extent by the outages of last week. The fifth participant has not completed the porting in of telephone numbers due in part to connectivity problems between their LSMS/SOA and the NPAC. Connectivity resolution is now the number 1 priority item for next week’s activities.
NPAC Incident Report
--------------------
Network Interruption 9/4/97 - 9/5/97
------------------------------------
The network connectivity interruption between the Midwest NPAC Field Test System and the participating Service Providers has been investigated by NPAC personnel. The results of the investigation are provided for your information and are as follows:
Cause of the Interruption
--------------------------
The interruption is attributed to human error and violation of NPAC Procedures.
On 9/3/97 after the close of daily operations a Network Engineer moved a piece of network interface equipment from its existing rack location to a new rack location. This move required the uncabling of the equipment. The engineer performed tests which indicated that all was in order and left for the day.
At the start of business 9/4/97 it was discovered that network connectivity to all service providers had been lost. This loss of connectivity is directly related to the movement of cables in and around the equipment which was moved.
Procedure Review
----------------
Well defined NPAC procedures are in place governing the planning and execution of equipment maintenance and movement requiring planning, approval and notification.
These procedures were not followed.
Procedures for responding to such situations also have been reviewed and we find that some additions and clarifications are required with respect to notification and escalation. These modifications are being incorporated.
Disposition
-----------
The Network Engineer has been re-assigned.
Procedures are being modified as required.
Network circuits and components are being correctly configured by the service providers to ensure that secondary circuits can be quickly activated via the BGP-4 Dynamic Routing Protocol.
Individual SP reports.
----------------------
SERVICE PROVIDER - A REPORT
SP-A participant reports having completing the porting in and out of TNs between SP-A and the other service providers with the exception of one number.
Last week SP-A experienced a 2 days outage from the local SOA and LSMS to the NPAC. This outage was on Thursday and Friday, September 4th and 5th. The outage initially was caused by unscheduled off hour work performed by NPAC personnel (see NPAC report). SP-As inability to re-establish our connectivity was due to our internal network firewall security procedures. AP-A is currently re-examining our internal processes and establishing new procedures to correct these problems. In addition SP-As network organization will be meeting with the NPAC to establish a clear understanding of the required network connectivity and disaster recovery requirements of the service providers.
SP-A experienced several outages this past week to the NPAC which interfered with our ability to fully participate in the FCC Field Trial. The majority of these problems were related to additional failover software SP-A was attempting to load this week. Unbeknownst to us this software caused Oracle operating system problems which did not allow us to recover our system each morning. We have discontinued this installation at this time.
SP-A’s connectivity is fully restored and we do not anticipate further problems.
TEST CASES COMPLETED:
TEST CASE NO. DATE PASS/FAIL COMMENTS
4.7.1 8/26 P 4 test calls made all passed
4.7.2 8/26 P 5 test calls made all passed
4.5.1.1.1 9/9 P 6 test calls made all passed
4.5.1.1.2 9/9 P 3 test calls made all passed
4.5.1.1.3 9/9 P 1 test call made
4.5.1.1.4 9/9 P 1 test call made
4.5.1.1.5 9/9 P 1 test call made
4.5.1.1.6 9/9 P 1 test call made
4.5.1.1.7 9/9 P 1 test call made
4.5.1.1.8 9/9 P 1 test call made
4.5.1.1.9 9/9 P 8 test calls made 3 passed
4.5.8.1.1 9/11 P 8 test calls made 7 passed
4.5.8.1.5 9/11 P 4 test calls made 3 passed
4.5.8.1.7 9/11 P 2 test calls made both passed
4.5.8.2.1 9/11 P 6 test calls made all passed
4.5.8.2.2 9/11 P 5 test calls made all passed
4.5.8.2.3 9/11 P 3 test calls made all passed
SERVICE PROVIDER - B REPORT
Note* SP-B has completed their call through testing, what remains for
SP-B is continued testing of Operations Process Flows.
Summary: The second wave of porting continued to be an education process.
Several human errors were made and problems with connectivity between our SMS and NPAC caused the due date on two of the orders to slip. The human errors encountered with both waves of porting have proven to be a positive learning experience, and in no way affect the viability of LRN as the technology solution to Local Number Portability.
TEST CASE DATE PASS/FAIL COMMENTS
4.4.1.1.1 9/08 P While this porting test case completed
successfully, the recipient LSP had an
internal problem with the trunk group
that was assigned. Once corrected, the
test case passed. A typing mistake caused
the problem.
4.3.4 9/12 P This conflict resolution test case
experienced
a variety of problems including mismatched
due
dates, the donor LSP failing to issue a
confirm port request, and finally,
connectivity
problems between both LSPs SMS and NPAC. The final result was the same, however, in the process, we tested a timer expiration with NPAC and learned many other lessons for porting.
4.4.1.1.1 9/12 P This port test case was originally due
9/10, however, due to a mismatch of due
dates and connectivity problems with NPAC
and SMS, the due date was changed to 9/12.
4.4.2.1.1 9/08 P This order was supposed to be a port for a
DID
range. As a result of a misunderstanding,
we failed to issue the LSR for the range of
numbers. This test case was used to test
the cancel process, so no major problem
resulted
in the misunderstanding.
4.4.2.1.1 9/12 P
SERVICE PROVIDER - C REPORT
SP-C participant reports completing the following for the Week Ending 9/12
Note* SP-C has completed their call through testing. What remains
for SP-C are operations process testing.
Test Case Date P/F Comments
4.5.3.1 8-Sep-97 P CLASS W/ SPRINT
4.5.3.2 8-Sep-97 P CLASS W/ SPRINT
4.5.3.3 8-Sep-97 P CLASS W/ SPRINT
4.5.3.4 8-Sep-97 P CLASS W/ SPRINT
4.5.3.5 8-Sep-97 P CLASS W/ SPRINT
4.5.3.6 8-Sep-97 P CLASS W/ SPRINT
4.8.1.1.1 9-Sep-97 P OP SVC RETEST
4.8.2.1.1 9-Sep-97 P OP SVC RETEST
4.8.4.1.1 9-Sep-97 P OP SVC RETEST
4.5.1.1.10 8-Sep-97 P SECOND PORTING TEST
4.5.1.1.11 8-Sep-97 P SECOND PORTING TEST
4.5.1.1.12 8-Sep-97 P SECOND PORTING TEST
4.5.1.1.13 9-Sep-97 P SECOND PORTING TEST
4.5.1.1.14 9-Sep-97 P SECOND PORTING TEST
4.5.1.1.15 9-Sep-97 P SECOND PORTING TEST
SERVICE PROVIDER - D REPORT
Due to the many problems experienced this reporting period with the LNP provisioning and administrative porting processes SP-D was unable to realize significant progress in the testing arena.
Specifically, I am in reference to the test bed Number Portability Service Control Points "NP/SCP" being populated with the ported test numbers in a synchronized manner required to support an effective test effort. The reasons were various, such as: connections down between SOA/LSMS’s and the NPAC, and associations lost to the NPAC. SP-D also had an internal problem with lost connectivity between the LSMS and the NP/SCP’s. The problem turned out to be an incorrect IP address.
SP-D has been tracking our SOA to NPAC and LSMS to NPAC availability since August 18/1997. Data to date shows SOA to NPAC at 86% and LSMS to NPAC 79% availability, this will be unacceptable in a production environment.
IN SUMMARY: If the problems we experienced this week occur while in production we as an industry will be inflicting duration of service outages on our customers. While the technology appears sound at this point in the testing, the processes DO NOT.
SP-D performed 26 regression tests during this reporting period to capture AMA data for billing reconciliation. SP-D has yet to conclude that LNP billing can be performed as required.
During the week of 9/8-12/97 SP-D completed 18 tests in the DMS-100 and 3 tests in the as follows:
DMS-100
4.5.1.1.10 -3 times
.11-3 times
.12-3 times
.15
.17
.18
.19
.20
.26-3 times
5ESS
4.5.1.1.11
.15
.17
SP-D has 52 tests remaining to be executed.
SERVICE PROVIDER - E REPORT
SP-E reports a week of problems in the area of connectivity/association with the NPAC. These problems made for very limited progress this week. During the porting process: SP-Es SOA databases were out of sync with the NPAC and the other service providers SOA’s. This was due in part to our problems and problems encountered by the other service providers, several of which also experienced connectivity/association problems at various times. This did, however, provide some real life testing scenarios.
On Wednesday 9/10, The NPAC experienced an interruption of service which was traced to their router. SP-E then checked their databases to determine if there were any changes to be made, none were necessary. SP-E then worked with the NPAC to resolve the problem. Researching the problem we found that there was a power outage of some kind experienced by the NPAC over the week end. The circuit was recovered using a different router. Later they fell back to the old router not realizing that the old router did not flush the route the router that they tried to use for disaster recovery. This means that their router thought there was a duplicate route and would not pass data across the link. The resolution was for us to have to change IP Addressing. NPAC personnel are logging a call with Ascend in an effort to determine why their router did not flush the route. Problem resolved at 12:oo CDT.
On 9/11 there was a major fiber cut which caused SP-E to lose connectivity with the NPAC. As of 9/12 at 12:30 pm, the ASMS was operational again. The SP-E SOA database was brought in sync with the NPAC at 2:30 CDT. Thereafter all outstanding subscriptions were entered. Number porting processes were complete at 5 pm with the exception of a number not ported in from SP-D.
Testing Status:
Test Case Date P/F Comments
n/a 9-2 P 4 pre-porting 911 test cases
4.4.2.1.1 9-9 P
4.3.4 9-12 P
4.4.1.1.1 9-12 P
Dick Dowd
FCC Field Test Coordination Manager
LLC/ICC
NPAC Phone 312-382-8090
NPAC Fax 312-382-8080
972-498-5069
972-918-1499 fax
972-498-5022 alternate fax
Vnet 757-5069
Pager 1-800-PAGEMCI Pin# 1742797
Email Richard Dowd / MCI ID: 258-4443
Internet 0002584443@mci.com