Illinois FCC LNP Field Test

Week 5 Summary

Summary

Week 5 of the Illinois FCC LNP Field Test (Field Test) consent rated on completing the second round of porting numbers and brought a new to LNP service provider into the FCC Field Test in an area that is already open to portability. Just prior to this second round of porting and continuing into week 5 a series of outages were experienced. These outages have slowed the progress of the FCC Field Test. Service provider number 5 has not been able to complete porting in of their telephone numbers due in large to the various outages experienced this week. For this reason the service providers were asked to detail connectivity problems experienced over the last two week’s. Individual service providers responses will be included in the report below.

The FCC Field Test remains on schedule for call through testing for three of the original four participant’s. The fourth original service provider has made progress with call through testing but has slowed to an extent by the outages of last week. The fifth participant has not completed the porting in of telephone numbers due in part to connectivity problems between their LSMS/SOA and the NPAC. Connectivity resolution is now the number 1 priority item for next week’s activities.

NPAC Incident Report

--------------------

Network Interruption 9/4/97 - 9/5/97

------------------------------------

The network connectivity interruption between the Midwest NPAC Field Test System and the participating Service Providers has been investigated by NPAC personnel. The results of the investigation are provided for your information and are as follows:

Cause of the Interruption

--------------------------

The interruption is attributed to human error and violation of NPAC Procedures.

On 9/3/97 after the close of daily operations a Network Engineer moved a piece of network interface equipment from its existing rack location to a new rack location. This move required the uncabling of the equipment. The engineer performed tests which indicated that all was in order and left for the day.

At the start of business 9/4/97 it was discovered that network connectivity to all service providers had been lost. This loss of connectivity is directly related to the movement of cables in and around the equipment which was moved.

Procedure Review

----------------

Well defined NPAC procedures are in place governing the planning and execution of equipment maintenance and movement requiring planning, approval and notification.

These procedures were not followed.

Procedures for responding to such situations also have been reviewed and we find that some additions and clarifications are required with respect to notification and escalation. These modifications are being incorporated.

Disposition

-----------

The Network Engineer has been re-assigned.

Procedures are being modified as required.

Network circuits and components are being correctly configured by the service providers to ensure that secondary circuits can be quickly activated via the BGP-4 Dynamic Routing Protocol.

Individual SP reports.

----------------------

SERVICE PROVIDER - A REPORT

SP-A participant reports having completing the porting in and out of TNs between SP-A and the other service providers with the exception of one number.

Last week SP-A experienced a 2 days outage from the local SOA and LSMS to the NPAC. This outage was on Thursday and Friday, September 4th and 5th. The outage initially was caused by unscheduled off hour work performed by NPAC personnel (see NPAC report). SP-As inability to re-establish our connectivity was due to our internal network firewall security procedures. AP-A is currently re-examining our internal processes and establishing new procedures to correct these problems. In addition SP-As network organization will be meeting with the NPAC to establish a clear understanding of the required network connectivity and disaster recovery requirements of the service providers.

SP-A experienced several outages this past week to the NPAC which interfered with our ability to fully participate in the FCC Field Trial. The majority of these problems were related to additional failover software SP-A was attempting to load this week. Unbeknownst to us this software caused Oracle operating system problems which did not allow us to recover our system each morning. We have discontinued this installation at this time.

SP-A’s connectivity is fully restored and we do not anticipate further problems.

TEST CASES COMPLETED:

TEST CASE NO. DATE PASS/FAIL COMMENTS

4.7.1 8/26 P 4 test calls made all passed

4.7.2 8/26 P 5 test calls made all passed

4.5.1.1.1 9/9 P 6 test calls made all passed

4.5.1.1.2 9/9 P 3 test calls made all passed

4.5.1.1.3 9/9 P 1 test call made

4.5.1.1.4 9/9 P 1 test call made

4.5.1.1.5 9/9 P 1 test call made

4.5.1.1.6 9/9 P 1 test call made

4.5.1.1.7 9/9 P 1 test call made

4.5.1.1.8 9/9 P 1 test call made

4.5.1.1.9 9/9 P 8 test calls made 3 passed

4.5.8.1.1 9/11 P 8 test calls made 7 passed

4.5.8.1.5 9/11 P 4 test calls made 3 passed

4.5.8.1.7 9/11 P 2 test calls made both passed

4.5.8.2.1 9/11 P 6 test calls made all passed

4.5.8.2.2 9/11 P 5 test calls made all passed

4.5.8.2.3 9/11 P 3 test calls made all passed

 

 

SERVICE PROVIDER - B REPORT

Note* SP-B has completed their call through testing, what remains for

SP-B is continued testing of Operations Process Flows.

Summary: The second wave of porting continued to be an education process.

Several human errors were made and problems with connectivity between our SMS and NPAC caused the due date on two of the orders to slip. The human errors encountered with both waves of porting have proven to be a positive learning experience, and in no way affect the viability of LRN as the technology solution to Local Number Portability.

TEST CASE DATE PASS/FAIL COMMENTS

4.4.1.1.1 9/08 P While this porting test case completed

successfully, the recipient LSP had an

internal problem with the trunk group

that was assigned. Once corrected, the

test case passed. A typing mistake caused

the problem.

4.3.4 9/12 P This conflict resolution test case

experienced

a variety of problems including mismatched

due

dates, the donor LSP failing to issue a

confirm port request, and finally,

connectivity

problems between both LSPs SMS and NPAC. The final result was the same, however, in the process, we tested a timer expiration with NPAC and learned many other lessons for porting.

4.4.1.1.1 9/12 P This port test case was originally due

9/10, however, due to a mismatch of due

dates and connectivity problems with NPAC

and SMS, the due date was changed to 9/12.

4.4.2.1.1 9/08 P This order was supposed to be a port for a

DID

range. As a result of a misunderstanding,

we failed to issue the LSR for the range of

numbers. This test case was used to test

the cancel process, so no major problem

resulted

in the misunderstanding.

4.4.2.1.1 9/12 P

 

 

SERVICE PROVIDER - C REPORT

SP-C participant reports completing the following for the Week Ending 9/12

Note* SP-C has completed their call through testing. What remains

for SP-C are operations process testing.

Test Case Date P/F Comments

4.5.3.1 8-Sep-97 P CLASS W/ SPRINT

4.5.3.2 8-Sep-97 P CLASS W/ SPRINT

4.5.3.3 8-Sep-97 P CLASS W/ SPRINT

4.5.3.4 8-Sep-97 P CLASS W/ SPRINT

4.5.3.5 8-Sep-97 P CLASS W/ SPRINT

4.5.3.6 8-Sep-97 P CLASS W/ SPRINT

4.8.1.1.1 9-Sep-97 P OP SVC RETEST

4.8.2.1.1 9-Sep-97 P OP SVC RETEST

4.8.4.1.1 9-Sep-97 P OP SVC RETEST

4.5.1.1.10 8-Sep-97 P SECOND PORTING TEST

4.5.1.1.11 8-Sep-97 P SECOND PORTING TEST

4.5.1.1.12 8-Sep-97 P SECOND PORTING TEST

4.5.1.1.13 9-Sep-97 P SECOND PORTING TEST

4.5.1.1.14 9-Sep-97 P SECOND PORTING TEST

4.5.1.1.15 9-Sep-97 P SECOND PORTING TEST

SERVICE PROVIDER - D REPORT

Due to the many problems experienced this reporting period with the LNP provisioning and administrative porting processes SP-D was unable to realize significant progress in the testing arena.

Specifically, I am in reference to the test bed Number Portability Service Control Points "NP/SCP" being populated with the ported test numbers in a synchronized manner required to support an effective test effort. The reasons were various, such as: connections down between SOA/LSMS’s and the NPAC, and associations lost to the NPAC. SP-D also had an internal problem with lost connectivity between the LSMS and the NP/SCP’s. The problem turned out to be an incorrect IP address.

SP-D has been tracking our SOA to NPAC and LSMS to NPAC availability since August 18/1997. Data to date shows SOA to NPAC at 86% and LSMS to NPAC 79% availability, this will be unacceptable in a production environment.

IN SUMMARY: If the problems we experienced this week occur while in production we as an industry will be inflicting duration of service outages on our customers. While the technology appears sound at this point in the testing, the processes DO NOT.

SP-D performed 26 regression tests during this reporting period to capture AMA data for billing reconciliation. SP-D has yet to conclude that LNP billing can be performed as required.

During the week of 9/8-12/97 SP-D completed 18 tests in the DMS-100 and 3 tests in the as follows:

DMS-100

4.5.1.1.10 -3 times

.11-3 times

.12-3 times

.15

.17

.18

.19

.20

.26-3 times

5ESS

4.5.1.1.11

.15

.17

SP-D has 52 tests remaining to be executed.

SERVICE PROVIDER - E REPORT

SP-E reports a week of problems in the area of connectivity/association with the NPAC. These problems made for very limited progress this week. During the porting process: SP-Es SOA databases were out of sync with the NPAC and the other service providers SOA’s. This was due in part to our problems and problems encountered by the other service providers, several of which also experienced connectivity/association problems at various times. This did, however, provide some real life testing scenarios.

On Wednesday 9/10, The NPAC experienced an interruption of service which was traced to their router. SP-E then checked their databases to determine if there were any changes to be made, none were necessary. SP-E then worked with the NPAC to resolve the problem. Researching the problem we found that there was a power outage of some kind experienced by the NPAC over the week end. The circuit was recovered using a different router. Later they fell back to the old router not realizing that the old router did not flush the route the router that they tried to use for disaster recovery. This means that their router thought there was a duplicate route and would not pass data across the link. The resolution was for us to have to change IP Addressing. NPAC personnel are logging a call with Ascend in an effort to determine why their router did not flush the route. Problem resolved at 12:oo CDT.

On 9/11 there was a major fiber cut which caused SP-E to lose connectivity with the NPAC. As of 9/12 at 12:30 pm, the ASMS was operational again. The SP-E SOA database was brought in sync with the NPAC at 2:30 CDT. Thereafter all outstanding subscriptions were entered. Number porting processes were complete at 5 pm with the exception of a number not ported in from SP-D.

Testing Status:

Test Case Date P/F Comments

n/a 9-2 P 4 pre-porting 911 test cases

4.4.2.1.1 9-9 P

4.3.4 9-12 P

4.4.1.1.1 9-12 P

Dick Dowd

FCC Field Test Coordination Manager

LLC/ICC

NPAC Phone 312-382-8090

NPAC Fax 312-382-8080

972-498-5069

972-918-1499 fax

972-498-5022 alternate fax

Vnet 757-5069

Pager 1-800-PAGEMCI Pin# 1742797

Email Richard Dowd / MCI ID: 258-4443

Internet 0002584443@mci.com

rdowd@mcimail.com