Troubleshooting LTE RAN NOKIA

Nokia Networks

LTE Radio Access, Rel. RL70, Operating Documentation, Issue 02, Documentation Change Delivery 4 Troubleshooting LTE RAN DN09185928 Issue 02 Approval Date 2015-09-15

Troubleshooting LTE RAN

The information in this document applies solely to the hardware/software product (“Product”) specified herein, and only as specified herein. This document is intended for use by Nokia Solutions and Networks' customers (“You”) only, and it may not be used except for the purposes defined in the agreement between You and Nokia Solutions and Networks (“Agreement”) under which this document is distributed. No part of this document may be used, copied, reproduced, modified or transmitted in any form or means without the prior written permission of Nokia Solutions and Networks. If you have not entered into an Agreement applicable to the Product, or if that Agreement has expired or has been terminated, You may not use this document in any manner and You are obliged to return it to Nokia Solutions and Networks and destroy or delete any copies thereof. The document has been prepared to be used by professional and properly trained personnel, and You assume full responsibility when using it. Nokia Solutions and Networks welcome Your comments as part of the process of continuous development and improvement of the documentation. This document and its contents are provided as a convenience to You. Any information or statements concerning the suitability, capacity, fitness for purpose or performance of the Product are given solely on an “as is” and “as available” basis in this document, and Nokia Solutions and Networks reserves the right to change any such information and statements without notice. Nokia Solutions and Networks has made all reasonable efforts to ensure that the content of this document is adequate and free of material errors and omissions, and Nokia Solutions and Networks will correct errors that You identify in this document. But, Nokia Solutions and Networks' total liability for any errors in the document is strictly limited to the correction of such error(s). Nokia Solutions and Networks does not warrant that the use of the software in the Product will be uninterrupted or error-free. NO WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY OF AVAILABILITY, ACCURACY, RELIABILITY, TITLE, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, IS MADE IN RELATION TO THE CONTENT OF THIS DOCUMENT. IN NO EVENT WILL NOKIA SOLUTIONS AND NETWORKS BE LIABLE FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA THAT MAY ARISE FROM THE USE OF THIS DOCUMENT OR THE INFORMATION IN IT, EVEN IN THE CASE OF ERRORS IN OR OMISSIONS FROM THIS DOCUMENT OR ITS CONTENT. This document is Nokia Solutions and Networks’ proprietary and confidential information, which may not be distributed or disclosed to any third parties without the prior written consent of Nokia Solutions and Networks. Nokia is a registered trademark of Nokia Corporation. Other product names mentioned in this document may be trademarks of their respective owners, and they are mentioned for identification purposes only. Copyright © 2015 Nokia Solutions and Networks. All rights reserved.

f

Important Notice on Product Safety This product may present safety risks due to laser, electricity, heat, and other sources of danger. Only trained and qualified personnel may install, operate, maintain or otherwise handle this product and only after having carefully read the safety information applicable to this product. The safety information is provided in the Safety Information section in the “Legal, Safety and Environmental Information” part of this document or documentation set.

Nokia Solutions and Networks is continually striving to reduce the adverse environmental effects of its products and services. We would like to encourage you as our customers and users to join us in working towards a cleaner, safer environment. Please recycle product packaging and follow the recommendations for power use and proper disposal of our products and their components. If you should have questions regarding our Environmental Policy or any of the environmental services we offer, please contact us at Nokia Solutions and Networks for any additional information.

2

DN09185928

Issue: 02


Table of Contents This document has 94 pages

Summary of changes..................................................................... 9

1

LTE troubleshooting methodology................................................10

2 2.1 2.1.1 2.1.2 2.1.3 2.1.4 2.1.5 2.1.5.1 2.1.5.2 2.1.5.2.1 2.1.5.2.2 2.1.5.2.3 2.1.5.2.4 2.1.5.2.5 2.1.5.3 2.1.5.4 2.1.5.5 2.2 2.2.1 2.2.2 2.2.3 2.2.3.1

Monitoring LTE RAN.....................................................................11 Hardware and software monitoring...............................................11 Handling faults and alarms .......................................................... 11 Troubleshooting faults and alarms............................................... 12 Viewing alarms in BTS Site Manager...........................................12 Alarm monitoring in NetAct.......................................................... 13 LED indications............................................................................ 14 LEDs of the Flexi Multiradio System Module (FSME).................. 14 LEDs of the Flexi Multiradio 10 BTS............................................ 15 LEDs of the Flexi Multiradio 10 System Module (FSMF)............. 15 LEDs of the Flexi Multiradio 10 System Module (FSIH)...............17 LEDs of the capacity extension sub-module (FBBA)................... 18 LEDs of the capacity extension sub-module (FBBC)................... 20 LEDs of the capacity extension sub-module (FBIH).................... 21 LEDs of the transmission sub-module......................................... 21 LEDs of the Flexi Zone Micro/Pico BTS.......................................22 LEDs of the Flexi Zone Controller................................................ 26 Performance monitoring...............................................................27 Viewing counters in BTS Site Manager........................................28 Performance monitoring in NetAct............................................... 29 Network monitoring using Traffica................................................30 Traffica general use case for LTE troubleshooting....................... 30

3 3.1 3.1.1 3.1.2

Collecting and analyzing troubleshooting data.............................32 eNB snapshot...............................................................................32 eNB snapshot content..................................................................32 Saving a Flexi Multiradio BTS LTE snapshot file for troubleshooting purposes.............................................................33 LTE1099: Event-triggered symptom data collection and provisioning ................................................................................. 34 Collecting eNB logs using the LTE1099: Event-triggered symptom data collection and provisioning feature.......................................34 LTE1909: BTS Diagnostics Toolkit............................................... 35 Triggering snapshot collection using LTE1909: BTS Diagnosis Toolkit........................................................................................... 36 Cell trace...................................................................................... 36 Trace features.............................................................................. 37 LTE163: Subscriber and Equipment Trace.................................. 37 LTE433: Cell Trace.......................................................................38

3.1.3 3.1.3.1 3.1.4 3.1.4.1 3.2 3.2.1 3.2.1.1 3.2.1.2

Issue: 02

DN09185928

3


3.2.1.3 3.2.1.4 3.2.1.5 3.2.1.6 3.2.1.7 3.2.1.8

3.3.2 3.4 3.5

LTE459: LTE Timing Advance Evaluation.................................... 38 LTE644: Configurable cell trace content...................................... 38 LTE162: Cell Trace with IMSI.......................................................38 LTE953: MDT (Minimization of Drive Test)...................................38 LTE1340: Trace-based Real Time Monitoring..............................39 LTE1457: Cell trace Configuration via Configuration Management. 39 LTE1687: Basic Layer3 Data Analyzer (L3DA) for LTE................39 Cell trace content......................................................................... 39 Activating cell trace in BTS Site Manager.................................... 40 Viewing and analyzing cell trace data using L3DA...................... 41 Tracing with NetAct...................................................................... 43 IP traffic capturing........................................................................ 43 Monitoring IP traffic using the LTE1460: Local and Remote IP Traffic Capturing feature...............................................................44 Analyzing IP capture files with Wireshark.................................... 45 Sending troubleshooting data to technical support...................... 47 Problems and use cases..............................................................48

4 4.1 4.1.1 4.1.1.1 4.1.1.2 4.1.1.2.1 4.1.1.2.2

LTE troubleshooting use cases.................................................... 51 BTS Site Manager connection problems......................................51 Accessing eNB using BTS Site Manager.....................................51 Problems with BTS Site Manager connectivity to the eNB.......... 51 Checking the PC firewall settings in Windows............................. 52 Checking PC firewall settings in Windows XP Professional.........53 Checking the PC firewall settings in Windows Vista/Windows 7...... 54 Remote connectivity via transport backhaul (m-plane)................ 55 Remote connectivity over Bluetooth (FZM only).......................... 56 Local connection problems.......................................................... 60 Local connectivity via local maintenance port.............................. 61 Cannot ping modules through LMP port...................................... 61 Changing BTS Site Manager local account................................. 62 Configuring Exceed for BTS Site Manager launching..................62 Access to the Linux shell..............................................................63 GPS maintenance access not possible through remote connection ..................................................................................................... 64 Software download/activation problems.......................................65 Incompatible BTS software version .............................................65 Software update to BTS site fails................................................. 66 Software version mismatch.......................................................... 67 Problems with local software download from BTS Site Manager..... 67 (Re)commissioning problems.......................................................70 Checking the BTS commissioning file..........................................70 Cannot save commissioning file or send parameters.................. 70 Verification of normal eNB site operation..................................... 71

3.2.1.9 3.2.2 3.2.3 3.2.4 3.2.5 3.3 3.3.1

4.1.1.3 4.1.1.4 4.1.1.5 4.1.1.5.1 4.1.1.5.2 4.1.2 4.1.3 4.1.4 4.1.5 4.2 4.2.1 4.2.2 4.2.3 4.2.4 4.3 4.3.1 4.3.2 4.3.3

4

DN09185928

Issue: 02


4.4 4.4.1 4.4.2 4.4.3 4.5 4.6 4.7 4.8 4.8.1 4.8.2 4.9 4.9.1 4.10 4.10.1 4.10.2 4.10.2.1 4.10.2.2 4.10.2.3 4.10.3 4.10.4 4.10.5 4.10.6

Issue: 02

Synchronization problems............................................................73 GPS Synchronization................................................................... 73 Timing Over Packet (IEEE.1588) Synchronization.......................73 Fallback solution for phase synchronization................................ 74 Cell does not come on air............................................................ 75 Sleeping cell.................................................................................76 eNB in a reset loop.......................................................................77 Troubleshooting call processing problems .................................. 77 High RRC Setup failure rate/RACH problems..............................78 RAB setup problems.................................................................... 79 Antenna line device problems...................................................... 80 Checking antenna line and MHA .................................................80 Troubleshooting hardware problems............................................81 Performing routine maintenance tasks.........................................81 Troubleshooting Flexi transmission module................................. 85 BTS reset required....................................................................... 85 Dead Peer detected..................................................................... 85 Ethernet interface is not working..................................................86 Identification of faulty units...........................................................87 Identification of faulty FZM unit.................................................... 91 Problems in connectivity between BTS modules......................... 91 FZM Recovery..............................................................................92

DN09185928

5


List of Figures

6

Figure 1

LTE troubleshooting methodology...................................................... 10

Figure 2

Faults view in BTS Site Manager....................................................... 13

Figure 3

System Module LED positions............................................................14

Figure 4

LEDs of the Flexi Multiradio 10 System Module (FSMF)....................17

Figure 5

LEDs of the Flexi Multiradio 10 System Module (FSIH)..................... 18

Figure 6

LEDs of the capacity extension sub-module (FBBA)..........................19

Figure 7

LEDs of the capacity extension sub-module (FBBC)......................... 20

Figure 8

LEDs of the capacity extension sub-module (FBIH)...........................21

Figure 9

LEDs of pico eNB............................................................................... 23

Figure 10

Front panel of the Flexi Zone Controller module ............................... 26

Figure 11

Performance monitoring data useful for troubleshooting....................27

Figure 12

Measurement configuration in BTS Site Manager..............................28

Figure 13

Viewing counters in BTS Site Manager.............................................. 29

Figure 14

Defining symptom data trigger list...................................................... 34

Figure 15

Selecting triggers for eNB snapshot collection................................... 36

Figure 16

Local and Remote IP Traffic Capturing Service window.....................44

Figure 17

IP capture file in Wireshark.................................................................46

Figure 18

Expert Info view in Wireshark............................................................. 46

Figure 19

Disabling Windows Firewall in Windows XP Professional..................54

Figure 20

BTS Site Manager log-in screen.........................................................56

Figure 21

Legacy pairing settings.......................................................................57

Figure 22

Select manual pairing option.............................................................. 58

Figure 23

Enter the legacy pairing code............................................................. 59

Figure 24

Successful pairing ............................................................................. 59

Figure 25

Establishing PAN connection..............................................................60

Figure 26

Local connectivity log-in screen..........................................................61

Figure 27

Enabling the SSH service for Linux shell access............................... 64

Figure 28

Software update failed........................................................................68

Figure 29

Software update report....................................................................... 69

Figure 30

BTS statuses in BTS Site Manager.................................................... 72

Figure 31

IP Connectivity Test in BTS SM..........................................................72

Figure 32

IP Connectivity test result................................................................... 72

Figure 33

Example ToP configuration.................................................................74

Figure 34

LMP IP plug installed..........................................................................82

Figure 35

LMP port IP cap not reinstalled after maintenance.............................82

Figure 36

Connector IP boot correctly installed firmly in place...........................83

Figure 37

Connector IP boot incorrectly installed (IP seals not firmly in place on any edge)............................................................................................83

Figure 38

Connector IP boot incorrectly installed (not pushed all the way in).... 84

DN09185928

Issue: 02


Issue: 02

Figure 39

Cable installed without IP boot........................................................... 84

Figure 40

Flexi Multiradio BTS LTE construction, stacked installation............... 90

Figure 41

Fan mapping in Flexi Multiradio BTS LTE.......................................... 90

Figure 42

Reset button in the BTS Hardware tab ..............................................93

DN09185928

7


List of Tables

8

Table 1

System Module LEDs (from left to right).............................................14

Table 2

LEDs of the Flexi Multiradio 10 System Module (FSMF)....................15

Table 3

LEDs of the Flexi Multiradio 10 System Module (FSIH)..................... 17

Table 4

LEDs of the capacity extension sub-module (FBBA) .........................18

Table 5

LEDs of the capacity extension sub-module (FBBC)......................... 20

Table 6

LEDs of the capacity extension sub-module (FBIH)...........................21

Table 7

Transmission sub-module LED indications.........................................22

Table 8

FZM and FZP LED indications........................................................... 23

Table 9

Flexi Zone Pico Wi-Fi LED states.......................................................25

Table 10

Description of front panel indicator LEDs in a BCN module............... 26

Table 11

Features related to eNB snapshot......................................................32

Table 12

Trace features.....................................................................................37

Table 13

Nokia Case Types.............................................................................. 47

Table 14

Nokia Problem Priorities..................................................................... 48

Table 15

LTE use cases.................................................................................... 48

Table 16

Explanation of NetAct Diagnostic Info values.....................................87

Table 17


Table 18


DN09185928

Issue: 02


Summary of changes

Summary of changes Changes between issues 01 (2014-11-21, RL70) and 02 (2015-09-15, RL70) The Fallback solution for phase synchronizationsection was added. Issue 01 (2014-11-21, RL70) This is the first issue of the document.

Issue: 02

DN09185928

9

LTE troubleshooting methodology


1 LTE troubleshooting methodology Figure 1: LTE troubleshooting methodology presents a general example of the procedures performed and tools used by the troubleshooting personnel. Figure 1

LTE troubleshooting methodology

TROUBLESHOOTING!PROCEDURES MONITORING

• ALARMS • KPIs/!COUNTERS • NETACT • TRAFFICA

COLLECTING DATA • eNB!SNAPSHOT • CELL!TRACE • IP!TRAFFIC CAPTURE

ANALYZING DATA • NETACT • TRAFFICA • PROTOCOL ANALYZER

FINDING ROOT!CAUSE • DATA!CORRELATION • USE!CASES • RECOVERY PROCEDURES

TROUBLESHOOTING!TOOLS The LTE troubleshooting methodology consists of the following procedures: • • •

Monitoring Collecting and analyzing data Finding root cause (with the help of use cases)

The order of actions and the selection of tools might vary depending on the use case and operator preferences. It depends on the nature of a problem if the root cause analysis is performed by the operator (for example, site configuration-related problems) or by Nokia Services (for example, defects inside a product). If it is not possible to find the root cause of the problems, Nokia Services should be contacted.

10

DN09185928

Issue: 02


Monitoring LTE RAN

2 Monitoring LTE RAN Monitoring of LTE RAN network allows the operator to notice potential problems and unwanted behaviors. Network monitoring consists of: • •

hardware and software monitoring (using faults and alarms) performance monitoring (using key performance indicators and counters)

Particular eNB sites can be monitored using the BTS Site Manager. NetAct and Traffica applications allows the operator to monitor the whole LTE RAN network.

2.1 Hardware and software monitoring This section explains how to monitor hardware and software using faults, alarms, and LED indications.

2.1.1 Handling faults and alarms Issues with software or hardware are signalled via fault messages, which in turn raise alarms. Fault messages come either from hardware devices or software components. The faults are system-specific or hardware-specific. The Fault Diagnosis System creates upper network alarms, which are based on an analysis of the faults, and sends them to the BTS Site Manager and iOMS/NetAct. The eNB alarm handling is based on the information of a unit or module state. It prevents sending unnecessary alarms by checking and updating the unit states. The eNB is able to maintain alarm handling after it has been commissioned and is in a configured state. During the start-up, the eNB waits for the real-time clock from the Network Time Protocol (NTP) server, and the fault diagnosis starts after the time has been set, or a time-out has been detected. If the eNB is unable to set the time using the NTP server, the default time (1.1.2004) is used. eNB transmission alarms The eNB transmission alarms (alarm ID 7665) are reported towards iOMS or NetAct. The transmission equipment uses the base station interface. Therefore, the retrieval of alarm information is mapped (using the fault ID) either in the iOMS or in the NetAct to distinguish between the alarms. Troubleshooting faults and alarms Faults and alarms are signaled in several ways: • •

Issue: 02

in BTS Site Manager in iOMS/NetAct

DN09185928

11

Monitoring LTE RAN


•

using LEDs

When starting to solve a problem, first refer to the instructions given in individual alarm descriptions. See the fields 'Fault source' and 'Instructions' in the alarm description. Many problems can be caused by commissioning mistakes, incorrect cabling, or wrong module installation.

2.1.2 Troubleshooting faults and alarms The following procedure is an overview of faults and alarms troubleshooting methodology. Symptoms Fault or alarms is detected. Recovery procedure Follow these instructions to check the BTS faults and LED states.

Procedure 1

Check the faults with OMS Fault Management. a) Start the Fault Management application. b) Check the active faults from the Active Alarms tab. c) Additionally fault history can be checked from the Alarm History tab.

2

Establish a connection to the BTS using BTS Site Manager and read alarm information. For more information see Viewing alarms in BTS Site Manager.

3

Check the unit states signalized by LEDs. Ensure that BTS Site Manager HW view corresponds with the actual BTS HW configuration. If there is a mismatch or grey units, check the SW level and unit states. For instructions, see Checking BTS SW versions and SW download. For descriptions on module LED states see LED indications.

2.1.3 Viewing alarms in BTS Site Manager BTS Site Manager displays the current alarm information, and the following procedure explains how to access this information. Alarm history is also included in the eNB snapshot.

12

DN09185928

Issue: 02


Monitoring LTE RAN

Procedure 1

Check the Faults view in BTS Site Manager (in the BTS Hardware tab). Figure 2

Faults view in BTS Site Manager

B 2

A

Click Show Source to find out which component is responsible for the alarm. Show Source is marked as element A in Figure 2: Faults view in BTS Site Manager. Clicking this button will highlight the faulty unit or open a configuration window where the error occurs.

3

To display the alarm description, click the question mark icon. The question mark icon is marked as element B in Figure 2: Faults view in BTS Site Manager. Clicking this button will open an online help window with detailed information about the fault.

2.1.4 Alarm monitoring in NetAct The NetAct Monitor application allows the user to view and process alarm information. For information on available tools and functions, refer to NetAct operating documentation: • •

Issue: 02

Fault Management Overview and Operations Fault Management Helps

DN09185928

13

Monitoring LTE RAN


2.1.5 LED indications Most hardware components have tricolor LEDs on the front panel to indicate the operational status and fault conditions. It is recommended that you read the information on the LED indications carefully. A blinking red LED does not always require removing of the module.

2.1.5.1

LEDs of the Flexi Multiradio System Module (FSME) See figure below for the FSME System Module LEDs location. Figure 3

System Module LED positions

The LED indications are listed and explained in table below. Table 1

System Module LEDs (from left to right)

LED

Color

FPFB status (one LED for internal and each external power output)

•

Yellow: Stand-by, output disabled

•

Green: Normal operation, output enabled

•

Red: Fault, output disabled

•

Yellow, blinking: Remote controlled, output disabled

•

Red: Fan fault

•

Green: Fan OK

• •

Red: Module self-test or reset (LED red for < 5 seconds) or major alarm or critical alarm Red, blinking: Minor alarm

•

Yellow: Stand-by or blocked

Fan status

System Module status

14

DN09185928

Issue: 02


Table 1

Monitoring LTE RAN

System Module LEDs (from left to right) (Cont.)

LED

Color • • •

RF Module 1 fiber connection status (OPTRF1)

•

Red: no connection

•

Green: connection OK

•

Yellow: not in use


•

Red: no connection

•


•

Yellow: not in use


•

Red: no connection

•


•

Yellow: not in use

System Extension Module • 1 fiber connection status • (OPT-EXT1)

Red: no connection

System Extension Module • 2 fiber connection status • (OPT-EXT2)

Red: no connection

•

•

2.1.5.2

2.1.5.2.1

Green: connection OK Yellow: not in use

Green: connection OK Yellow: not in use

LEDs of the Flexi Multiradio 10 BTS

LEDs of the Flexi Multiradio 10 System Module (FSMF) Table 2

Issue: 02

Yellow, blinking: SW download or configuration ongoing, module non-operational Green: Module operational (the cell can be locked in the RNC) Green, blinking: Module is loading software or parameters or local maintenance access when modules are operational

LEDs of the Flexi Multiradio 10 System Module (FSMF)

LED

Color

EIF2/RF/6

•

Red: no connection

•


DN09185928

15

Monitoring LTE RAN


Table 2

LEDs of the Flexi Multiradio 10 System Module (FSMF) (Cont.)

LED

Color

RF/EXT link 6 status or transport interface 2 status

•

Yellow: not in use

RF/EXT1

•

Red: no connection

RF link 1 status

•


•

Yellow: not in use

RF/EXT2

•

Red: no connection

RF link 2 status

•


•

Yellow: not in use

RF/EXT3

•

Red: no connection

RF link 3 status

•


•

Yellow: not in use

•

Red: no connection

•


•

Yellow: not in use

EIF1/TRS

•

Red: no connection

Transport interface 1 status

•


•

Yellow: not in use

FAN

•

Red: Fan fault

Fan status

•

Green: Fan OK

STATUS

•

•


•


•

Yellow, blinking: SW download or configuration ongoing, module nonoperational Green: Module operational (the cell can be locked in the RNC) Green, blinking: Module is loading software or parameters or local maintenance access when modules are operational

SRIO


• •

16

DN09185928

Issue: 02


Figure 4

Monitoring LTE RAN

LEDs of the Flexi Multiradio 10 System Module (FSMF)

EIF2/RF/6 RF/EXT1 RF/EXT2 RF/EXT3

2.1.5.2.2

STATUS FAN EIF1/TRS SRIO

LEDs of the Flexi Multiradio 10 System Module (FSIH) Table 3

LEDs of the Flexi Multiradio 10 System Module (FSIH)

LED

Color

FAN

•

Red: Fan fault

Fan status

•

Green: Fan OK

STATUS

•

•


•


•

Yellow, blinking: SW download or configuration ongoing, module nonoperational Green: Module operational


• •

Green, blinking: Module is loading software or parameters or local maintenance access when modules are operational

RF1

•

Red: no connection

RF2

•


•

Yellow: not in use

RF3 RF4 RF5 RF6 RF/EXT1 RF/EXT2 RF/EXT3 RF/EXT4

Issue: 02

DN09185928

17

Monitoring LTE RAN


Table 3

LEDs of the Flexi Multiradio 10 System Module (FSIH) (Cont.)

LED

Color

RF/EXT5 RF/EXT6 RF link status SRIO1

•

Red: no connection

SRIO2

•


•

Yellow: not in use

EIF1

•

Red: no connection

EIF2

•


•

Yellow: not in use

SRIO1-2 connection status

Transport interface status

Figure 5

LEDs of the Flexi Multiradio 10 System Module (FSIH)

RFEXT2

RFEXT3/10G

RFEXT1/10G

RFEXT4 RFEXT5/10G

SRIO2 STATUS SRIO1

RFEXT6

FAN RF1-4 RF5 RF6/EIF5

2.1.5.2.3

LEDs of the capacity extension sub-module (FBBA) Table 4

18

EIF1-2

LEDs of the capacity extension sub-module (FBBA)

LED

Color

SRIO

•

Red: no connection

SRIO1 connection status

•


DN09185928

Issue: 02


Table 4

Monitoring LTE RAN

LEDs of the capacity extension sub-module (FBBA) (Cont.)

LED

Color

STATUS

•

Yellow: not in use

•

•


•


•


Operational status

• •


RF/EXT

•

Red: no connection

RF link status

•


•

Yellow: not in use

Figure 6

LEDs of the capacity extension sub-module (FBBA)

SRIO

Issue: 02

STATUS

DN09185928

RF/EXT

19

Monitoring LTE RAN

2.1.5.2.4


LEDs of the capacity extension sub-module (FBBC) Table 5

LEDs of the capacity extension sub-module (FBBC)

LED

Color

RF1

•

Red: no connection

RF2

•


•

Yellow: not in use

SRIO/RF4

•

Red: no connection

RF4 link status or SRIO interface status

•


•

Yellow: not in use

•

•


•


•


RF3 RF link status

STATUS Operational status

• •

Figure 7


LEDs of the capacity extension sub-module (FBBC) CLASS1 LASERPRODUCT

STATUS RF1

20

RF2

DN09185928

RF3

SRIO/RF4

Issue: 02


2.1.5.2.5

Monitoring LTE RAN

LEDs of the capacity extension sub-module (FBIH) Table 6

LEDs of the capacity extension sub-module (FBIH)

LED

Color

STATUS

•

•


•


•


Operational status

• •

Green, blinking: Module is loading software or parameters or local maintenance access when modules are operational.

RF/EXT1

•

Red: no connection

RF/EXT2

•


•

Yellow: not in use

SRIO1

•

Red: no connection

SRIO2

•


•

Yellow: not in use

RF/EXT3 RF/EXT4 RF/EXT5 RF/EXT6 RF1-6 link status

SRIO1-2 connection status

Figure 8

LEDs of the capacity extension sub-module (FBIH)

RFEXT2 RFEXT1/10G SRIO2 SRIO1

RFEXT3/10G RFEXT4 RFEXT5/10G RFEXT6

STATUS

2.1.5.3

LEDs of the transmission sub-module The LED indications of transmission sub-module are listed and explained in table below.

Issue: 02

DN09185928

21

Monitoring LTE RAN


Table 7

Color

Transmission sub-module LED indications

Explanation

Relevant fault

No color The module has no power. Red

Module is faulty.

•

AIS on unit $U, interface $IF

•

BFD Session down in egress direction for session $S BFD Session down in ingress direction for session $S EBER on unit $U, interface $IF

• •

•

License missing for feature $feature on interface $if LOF on unit $U, interface $IF

•

LOMF on slot $U, interface $IF

•

•

LOS on unit $U, [ethernet] interface $IF Missing or non-compliant SFP module on unit $U, interface $I RDI on unit $U, interface $IF

•

Synchronization lost

•

•

2.1.5.4

Red, blinking

Module is degraded.

•

$Protocol Timing source lost on unit $U [, interface $IF]

Yellow

Module is not operational (blocked or uncommissioned).

•

Dead Peer Detected

Yellow, blinking

Module is not operational but SW download or configuration is ongoing.

Green

Module is operational (configured and working).

Green, blinking

Module is operational but SW download or configuration is ongoing.

LEDs of the Flexi Zone Micro/Pico BTS Flexi Zone Pico LED indicators are shown in the figure below.

22

DN09185928

Issue: 02


Monitoring LTE RAN

Figure 9

LEDs of pico eNB BTSStatusLED RFPowerStatusLED WiFiStatusLED TransportStatusLED

The Flexi Zone Micro/Pico BTS LED indications are listed and explained in tables below. Table 8

FZM and FZP LED indications

LED

Color

Description

BTS is booting up, and the Platform SW is starting up. LED is being controlled by HW.

Transport Status LED OFF

Priority ( 1 is highest) 1

Stable RED

In the startup sequence, the 1 Platform SW is up and it has taken the control of LED. This state continues until the TRSW become operational following Site PowerUp or Site Reset. Includes POST (in case of a power on scenario)

Stable RED

TRSW has taken control of Transport Status LED and is Initializing *or* Critical or Major Fault raised on TRSW

2

Stable YELLOW

Critical or Major Fault raised on TRSW AND Bluetooth is ENABLED automatically

2

Blinking RED

MINOR or Degraded alarm exists on TRSW

3

Stable GREEN

TRSW is ready (fully initialized)

4

- No known Critical/Major/Degraded/Minor Transport faults present. RF Power Status LED

Issue: 02

OFF

BTS is booting up, and the Platform SW is starting up. LED is being controlled by HW.

1

OFF

Platform SW has come up successfully.

1

DN09185928

23

Monitoring LTE RAN


Table 8

FZM and FZP LED indications (Cont.)

LED

BTS Status LED

Color

Description

Priority ( 1 is highest)

Blinking GREEN

FPGA has taken control of the RF LED in the startup sequence

1

Blinking GREEN

RF Transmission OFF

1

Stable GREEN

RF Transmission ON

2

Stable RED

While BTS is booting up and Status LED is being controlled by HW

1

Blinking YELLOW

In startup sequence, Platform SW is up and is now controlling the Status LED.

1

Includes POST (in case of a power on scenario) Blinking YELLOW

Startup: Indicates BTSOM has taken control of the BTS Status LED and is performing site initialization related activities with the iOMS (SW version inquiry, SW download, SCF download, HW Configuration upload, Alarm sync, etc).

2

Stable YELLOW

Indicates BTS and/or all CELLs are Blocked/Locked

3

Stable RED

Indicates BTS is Faulty: It signifies that at-least one Critical Fault is currently present on BTS.

4

Note: Includes any type of BTS faults including Transport, UPlane, and C-Plane faults. Blinking RED

Indicates BTS is degraded: At least one Major Fault is currently active on BTS (while no Critical Faults are active)

5

Note: Includes any type of BTS faults including Transport, UPlane, and C-Plane faults. Blinking GREEN

24

DN09185928

Indicates a SW download is in progress during runtime operation (i.e. SW download is occurring outside of startup)

6

Issue: 02


Monitoring LTE RAN

Table 8

FZM and FZP LED indications (Cont.)

LED

Color

Stable RED

Description

Indicates a Critical Failure occurred during Auto Connection (AutoConnectionState is "Disconnected").

Priority ( 1 is highest) 7

If failure is due to an iOMS rejection (unsuccessful AutoConnectionEstablishedReply message was received), condition will persist for 5 min until the AutoConnection process is automatically retried. Note: If failure is due to iOMS connectivity being down (detected by Supervision on iOMS link), a Critical Fault will be active (SET). Blinking YELLOW

Indicates Auto-Connection is in 8 Progress (Until connection to Final iOMS is achieved)

Stable YELLOW

Indicates BTS is Uncommissioned

9

Note: A "4030: EFaultId_NoCommDataAl" will be SET resulting in generation of a "7652 BASE STATION NOTIFICATION" alarm. Blinking GREEN

BTS in Test Dedicated State

10

Stable GREEN

Indicates either 1) a stable 10 condition where at least one CELL is OnAir (indicated in conjuction with "RF Power Status" LED being Stable GREEN), or 2) a transitory condition where the BTS is fully configured and nothing is preventing a CELL from transitioning to onAir (indicated in conjuction with "RF Power Status" LED being Blinking GREEN).

For descriptions of Pico module Wi-Fi LED states, see table below. Table 9

Flexi Zone Pico Wi-Fi LED states Wi-Fi LED state

Issue: 02

Description

Off

Wi-Fi module not present

Blinking red

Wi-Fi module boot in progress

DN09185928

25

Monitoring LTE RAN


Table 9

Flexi Zone Pico Wi-Fi LED states (Cont.) Wi-Fi LED state

2.1.5.5

Description

Stable red

2.4 GHz and 5 GHz fault

Blinking green

2.4 GHz or 5 GHz fault

Stable green

2.4 GHz and 5 GHz on

LEDs of the Flexi Zone Controller Location of LEDs on the Flexi Zone Controller is show in figure below. Figure 10

Front panel of the Flexi Zone Controller module

The Flexi Zone Controller LED indications are listed and explained in table below. Table 10

Description of front panel indicator LEDs in a BCN module

Symbol and name Application-specific LEDs A1 and A2

Color

State

Green

Steady or blinking

Red

Steady or blinking

Blue

Steady or blinking

During module startup both LEDs indicate start (BLUE) and completion (GREEN) of an LMP’s boot sequence. After successful boot-up the LEDs will remain green.

Green

Steady

Both power supplies operate normally.

Red

Steady

One of the power supplies has failed.

Off PSU indicator LED

Off

26

Description Two tri-color application-specific LEDs controlled through an IPMI command.

Power is off.

DN09185928

Issue: 02


Table 10

Monitoring LTE RAN

Description of front panel indicator LEDs in a BCN module (Cont.)

Symbol and name

Color Blue

NS LED

State Steady

Reserved for future use. Default state: Off

Off Processor/add-in card LEDs

Description

Green

Steady

The module application has started.

Red

Steady

The processor add-in card is powered on, but no application is running.

Blue

Steady

MMC is running, but the processor is powered off.

P1-P8

Off

No add-in card inserted into the slot.

2.2 Performance monitoring This section focuses on key performance indicators (KPIs) and counters used to monitor network performance. The graphic illustrates how KPI and counter data can be used for troubleshooting purposes. Figure 11

Performance monitoring data useful for troubleshooting

The KPI descriptions can be found in the LTE Radio Access Operating Documentation/Reference/Key Performance Indicatiors document. The counter

Issue: 02

DN09185928

27

Monitoring LTE RAN


descriptions can be found in the LTE Radio Access Operating Documentation/Reference/Counters document.

2.2.1 Viewing counters in BTS Site Manager BTS Site Manager allows the user to schedule measurements and view counter data. The following procedure explains how to access this information. Before you start There are separate GUIs for viewing radio-related counters (BTS PM) and transportrelated counters (TRS PM). The counter data is also available in the eNB snapshot. The KPIs that are calculated from counter-based formulas can be viewed in NetAct. The user can also define his own KPI formulas.

Procedure 1

Choose either BTS Performance Monitoring or TRS Performance Monitoring. On the left side of BTS Site Manager choose either BTS PM or TRS PM tab.

2

Activate required measurements. In the selected tab switch to the 'Measurement Configuration' and select the required measurement types and define the collection interval. Figure 12

28

Measurement configuration in BTS Site Manager

DN09185928

Issue: 02


Monitoring LTE RAN

3

Switch to the Counters.

4

Select the Object (for example, BTS or a certain cell) and select the Counters to be displayed. It is possible to view the data in a graph or table format. Figure 13

Viewing counters in BTS Site Manager

2.2.2 Performance monitoring in NetAct The NetAct Performance Manager application allows the user to view and process counter and KPI data. For information on available tools and functions, refer to NetAct operating documentation: • • •

Issue: 02

Performance Management Overview Performance Manager Overview Thresholder and Profiler Overview

DN09185928

29

Monitoring LTE RAN


2.2.3 Network monitoring using Traffica This chapter introduces the Traffica application. The LTE1053: Real-time KPI-monitoring with Traffica feature provides real-time performance monitoring for the LTE Flexi Multiradio BTS. The Flexi Multiradio BTS is connected directly to Traffica using a new real-time data interface. This real-time data interface contains the results of ongoing performance measurements (PM-counter values) in real-time. These results are sent every minute. The LTE1340: Trace-based Real Time Monitoring feature offers a real time network monitoring solution based on the Flexi Multiradio BTS cell trace interface (introduced with the LTE433: Cell trace feature). The collected trace data is sent to the Layer 3 Data Collector (L3DC) that transfers the data to Traffica for visualization. Traffica consists of: • •

a Traffica News client for offline analysis of collected data. a Traffic Views client for real-time monitoring (various graph sets for displaying collected data in different time periods).

A Traffica user can configure threshold-based real-time alarms that can be forwarded to the NetAct Monitor application. For more details on Traffica applications and functions, refer to Traffica operating documentation: • •

2.2.3.1

Traffica Principles Traffica Reference Guide

Traffica general use case for LTE troubleshooting Purpose Traffica is a real-time traffic monitoring tool designed to monitor and analyze network traffic. Traffica allows the operator to see how the network functions from the network element level down to individual subscriber information. Before you start To visualize the cell trace data and real-time counter data in Traffica, the following requirements must be fulfilled: • • •

The L3 Data Collector (L3DC) is installed and configured. The LTE433: Cell Trace is activated. The Traffica - L3DC interface is configured.

Procedure

30

DN09185928

Issue: 02


1

Monitoring LTE RAN

Follow network and service performance indicators using Traffic Views. • • •

2

Spot degradations (for example, RRC/RAB setup failure increase). Detect sites with no traffic. Define threshold alarms to automate fault detection.

Analyze problem severity and location using Traffic Views. Use failure graphs to check whether the problem is limited to certain network elements (NEs) or cells.

3

Troubleshoot the problem using linked graphs and Traffic News. • •

Issue: 02

Use drill down graphs to isolate the problem further down in topology. Study call event reasons and call failure reasons to determine the actual fault.

DN09185928

31

Collecting and analyzing troubleshooting data


3 Collecting and analyzing troubleshooting data To efficiently perform the troubleshooting process, a set of supporting data should be collected. In case of call processing and signaling problems, data can be collected using one of the following methods: • • •

saving eNB snapshot cell tracing IP traffic capturing

Additional data can be obtained by: • • • •

drive test data end-user feedback core network indicators problematic site location (for example, from such geographic location tools as Traffica)

If despite having all available data, the operator is not able to solve a problem by himself, Nokia Services should be contacted. Sending troubleshooting data to technical support describes how to contact Nokia in a most effective way.

3.1 eNB snapshot This section explains what is included in an eNB snapshot and how to save a snapshot using different features. Table below list features related to eNB snapshot functionality. Table 11

Features related to eNB snapshot Feature

Release

LTE1099: Event-triggered Symptom Data Collection and Provisioning

RL50

LTE1909: BTS Diagnostics Toolkit

RL60

3.1.1 eNB snapshot content Examples of data included in the eNB snapshot: • • •

32

Active alarms and alarm history Installed SW versions eNB configuration (commissioning parameters)

DN09185928

Issue: 02



Internal log files for different eNB components (including MAC TTI traces) PM counter data for this eNB (starting from the last eNB reset but for a maximum for 24 hours)

• •

3.1.2 Saving a Flexi Multiradio BTS LTE snapshot file for troubleshooting purposes Purpose The snapshot file can be saved in the connected mode; it contains the current status of elements and BTS Site Manager: used HW configuration, logs, alarms, HW and SW version information. Before you start Remember to attach the logs when raising a Resolve case to Tier3 level.

g

Note: Several parts of the eNB snapshot are based on ring buffers which are covering only limited time. Therefore, it is important to save the snapshot as soon as possible after a problem has occurred. Otherwise the log parts showing the root cause of the problem might have been overwritten. Setting snapshot collection triggers (eNB faults, signaling events) helps to automate the process.

Procedure 1

Choose File → Save → Snapshot to open the Save Snapshot dialog box.

2

Enter the file name and define the location for the file to be saved. The default file name is Snapshot__.zip. The default location is the folder where you have saved snapshot files previously or your default working folder, for example \BTS Site Manager Data.

Issue: 02

3

Under Detail Level, select 'Fetch all data from elements'.

4

Enter the description of the situation when the snapshot was taken in the Notes field.

5

Click the Save button.

6

Check that the file name is correct and attach the snapshot to fault report.

DN09185928

33



3.1.3 LTE1099: Event-triggered symptom data collection and provisioning Introduction to the feature This feature introduces automatic fault-triggered BTS symptom data (troubleshooting logs) collection. The triggering BTS faults are defined in the Site Configuration File (SCF) using the NetAct Configurator or using the BTS Site Manager. The logs are available in the OMS and can be uploaded to NetAct.

3.1.3.1

Collecting eNB logs using the LTE1099: Event-triggered symptom data collection and provisioning feature This procedure shows how to configure the LTE1099: Event-triggered symptom data collection and provisioning feature to automate the eNB snapshot collection. Before you start The feature configuration is done during an eNB commissioning.

Procedure 1

Define the list of symptom data collection triggers. Add a new Symptom data trigger list under the MRBTS object. Figure 14

2

Defining symptom data trigger list

Set the Fault ID that should trigger the data collection. Up to 20 faults can be configured.

g

Note: Use the faults that most frequently occur in a certain site or contact Nokia Services for a list of recommended triggers.

3

Define the recovery reset delay. Configure the recovery reset delay parameter under the LNBTS object. If a fault leads to a recovery reset, the eNB might delay it according to the configured value:

34

DN09185928

Issue: 02



• • •

4

'0': no delay '1-9999': delay time '10000': wait for snapshot to be transferred in iOMS

Configure the dwBTSUploadEnabled parameter to enable/disable the data transfer to OMS. It is recommended to provide an empty trigger list if upload is to be permanently disabled.

5

Configure the dwAlarmForUploadCompleted parameter to enable/disable the alarm setting in OMS. If enabled, the 71106 TROUBLESHOOTING DATA RECEIVED alarms is raised once the data is received in OMS

6

Configure the NetActNotifyEnabled parameter to enable/disable the data upload to NetAct.

Result When a fault defined in the symptom trigger list occurs, the symptom data is collected. The 71129 TROUBLESHOOTING DATA CREATION OR UPLOAD FAILED indicates that the data creation or upload to OMS has failed.

3.1.4 LTE1909: BTS Diagnostics Toolkit Introduction to the feature The LTE1909: BTS Diagnostics Toolkit provides the operator, technical support and the Nokia developer with efficient data collection from eNB. The set of data collected, such as: trace data, BTS snapshot data, is sent from the operator to Nokia. This feature supports resolving an error at first occurrence given that all the necessary information is available for detailed analysis. The following are the functions/features of this toolkit: • • • •

Issue: 02

supervises the eNBs via cell tracing triggers the BTS snapshot creation in a timely manner, based on the trigger events in the cell trace content comes with the SW of the Layer 3 Data Collector is easy to use and install

DN09185928

35


3.1.4.1


Triggering snapshot collection using LTE1909: BTS Diagnosis Toolkit With the LTE1909: BTS Diagnosis Toolkit feature, it is possible to trigger an eNB snapshot collection based on selected events.

Procedure 1

Open the L3DA.

2

Go to Filter -> Set snapshot filter for TCP.

3

Configure the triggering options in the opened window. Figure 15

4

Selecting triggers for eNB snapshot collection

Activate the filter.

Result Once the selected event occurs, the eNB snapshot will be collected.

3.2 Cell trace This section lists cell trace features and explains how to use tracing.

36

DN09185928

Issue: 02



3.2.1 Trace features This section lists all features related to cell trace functionality. Table 12

Trace features

Feature

3.2.1.1

Release

LTE163: Subscriber and Equipment Trace

RL20

LTE433: Cell Trace

RL20

LTE459: LTE Timing Advance Evaluation

RL30

LTE644: Configurable Cell Trace Content

RL30

LTE162: Cell Trace with IMSI

RL40

LTE953: MDT (Minimization of Drive Test)

RL40

LTE1340: Trace-based Real Time Monitoring

RL40

LTE1457: Cell trace Configuration via Configuration Management

RL40

LTE1687: Basic Layer3 Data Analyzer (L3DA) for LTE

RL50

LTE1501: Measurement Report (MR) addition to cell trace

RL70

LTE1914: RIP (Received Interference Power) measurement report extension/Support RIP initiated alarms

RL70

LTE1931: Measurement Report addition with UE Uplink SINR

RL70

LTE2054: Measurement Report addition with PDCP KPI counter subset

RL70

LTE163: Subscriber and Equipment Trace This feature provides detailed, subscriber-oriented information at a call-level for one or more specific UEs. The subscriber and equipment trace supports the tracing of IMSI or IMEI numbers. The traces are activated on demand. The operator can activate subscriber and equipment tracing for a limited period for the purpose of specific analysis, for example: • • • •

Issue: 02

root cause determination of a malfunctioning mobile advanced troubleshooting optimization of resource usage and quality RF coverage control and capacity improvement

DN09185928

37


• •


dropped call analysis E2E procedure validation

For more details, see LTE163: Subscriber and Equipment Trace.

3.2.1.2

LTE433: Cell Trace With this feature, it is possible to follow the ongoing connections in a cell and to verify the intended functions within a cell. All UEs in a target cell that are in the connected state are traced simultaneously. This feature is also used for a more extensive analysis of a problem when various performance measurements do not give a clear indication of the problem. For more details, see LTE433: Cell Trace.

3.2.1.3

LTE459: LTE Timing Advance Evaluation With this feature, the Timing Advance (TA) values are added to trace reports. Depending on the configuration, the TA values are added to vendor-specific extension tracing of the subscriber trace (LTE163: Subscriber and Equipment Trace) or the cell trace (LTE433: Cell Trace). For more details, see LTE459: LTE Timing Advance Evaluation.

3.2.1.4

LTE644: Configurable cell trace content This feature allows the operator to configure specific messages to be traced for each selected interface. For more details, see LTE644: Configurable cell trace content.

3.2.1.5

LTE162: Cell Trace with IMSI With this feature, the existing cell trace data reports are mapped with the IMSI/IMEI numbers of the UEs located in the traced cell. This feature extends the scope of the LTE433: Cell Trace feature. The current LTE433: Cell Trace feature remains unchanged. For more details, see LTE162: Cell Trace with IMSI.

3.2.1.6

LTE953: MDT (Minimization of Drive Test) This feature is introduced as an alternative to expensive drive tests performed during network deployment and optimization. It offers a predefined set of MDT profiles available at the NetAct TraceViewer application. The profiles are defined to detect and monitor potential coverage problems. The solution is based on the data collected using the following features:

38

DN09185928

Issue: 02


• • •


LTE433: Cell Trace LTE644: Configurable cell trace content LTE570: Periodic UE Measurements

For more details, see LTE953: MDT (Minimization of Drive Test).

3.2.1.7

LTE1340: Trace-based Real Time Monitoring This feature introduces a real-time network monitoring solution that is based on: • • •

trace data collected from multiple eNBs L3 Data Collector (L3DC) network element that acts as a trace collection entity in tracing Traffica is used for visualizing the collected data

For more details, see LTE1340: Trace-based Real Time Monitoring.

3.2.1.8

LTE1457: Cell trace Configuration via Configuration Management The LTE1457: Cell trace Configuration via Configuration Management feature allows the operator to configure the cell trace using NetAct Configuration Management (CM) and the BTS Site Manager. This solution is an alternative to the existing method using NetAct TraceViewer application. For more details, see LTE1457: Cell trace Configuration via Configuration Management.

3.2.1.9

LTE1687: Basic Layer3 Data Analyzer (L3DA) for LTE The LTE1687: Basic Layer3 Data Analyzer (L3DA) for LTE feature introduces the Layer 3 Data Analyzer (L3DA), which is a tool for analyzing Flexi Multiradio BTS trace data. The LTE L3DA allows the operator to visualize and analyze the cell trace data collected in the L3 Data Collector (L3DC). For more details, see LTE1687: Basic Layer3 Data Analyser (L3DA) for LTE.

3.2.2 Cell trace content Cell trace data consists of: •

S1 Interface: S1AP data – – –

Issue: 02

context management handover signaling UE capability info indication

DN09185928

39


– – – –

E-RAB management NAS transport location reporting error indication

X2 Interface: X2AP data

•

– –

basic X2 mobility error indication

Uu Interface: RRC data

•

– – – – – –

g


RRC connection signaling counter check inter-RAT mobility measurements UE capability UL/DL information transfer

Note: The user plane is not traced.

3.2.3 Activating cell trace in BTS Site Manager This procedure shows how to activate the LTE433: Cell Trace feature using the BTS Site Manager. Before you start

g

Note: To activate the feature on a wider scale, create a plan using the NetAct Configurator.

Procedure 1

Set the LNBTS: Activate cell trace parameter to true. During commisioning • • • •

go to the Radio Network Configuration page expand MRBTS object select LNBTS object set Activate cell trace parameter to true

To activate other trace-related features, set the following parameter values to true: • • •

40

LNBTS: Activate cell trace with IMSI LNBTS: Activate MDT cell trace LNBTS: Activate subscriber and equipment trace

DN09185928

Issue: 02


2


Create an MTRACE object. To create MTRACE object: • •

expand LNBTS object right click CTRLTS object and creat new MTRACE object

The CRTLTS object contains the basic trace parameters. Every active cell trace session equals one MTRACE object. 3

Configure the MTRACE object parameters. To view the cell trace data in another tool than the NetAct, put the appropriate address in MTRACE:Trace collection entity IP address.

4

Set TA tracing to true. • • •

MTRACE object select Cell vendor specific tracing object set Cell TA tracing to true

The Timing Advance information is useful for troubleshooting.

Result The cell trace data can be viewed with the NetAct Traceviewer, L3DA, or Traffica.

3.2.4 Viewing and analyzing cell trace data using L3DA The following procedure shows how to use the L3DA application. Purpose The cell trace analysis platform consists of the following features: •

• •

LTE1340: Trace-based Real Time Monitoring Introduction of the L3 Data Collector (L3DC) and the function of sending cell trace data to Traffica LTE1687: Basic Layer3 Data Analyzer (L3DA) Introduction of the L3 analysis function LTE1909: BTS Diagnostics Toolkit A set of events for snapshot triggering

Typical call processing problems analyzed using the L3DA: • • •

Issue: 02

call setup problems call drops handover failures

DN09185928

41



Before you start To collect the cell trace data, the LTE433: Cell Trace feature needs to be configured.

Procedure 1

Start the cell trace. To view the traces in L3DA, the L3DC IP address needs to be included in the cell trace configuration. The cell trace configuration is done with the BTS Site Manager or the TraceViewer application. The L3DC needs to be selected as the Trace Collection Entity during the cell trace configuration (the L3DC’s IP address put as the tceIpAddress parameter value).

2

Open the L3DA. To manage sacks and profiles, go to Environment > Change sack and profile (SCREEN). Select the one you need and click Load. To configure your log paths (and other preferences) go to Tools > Options.

3

Identify problematic calls by checking the ‘Out cause’ column. Every call scenario is displayed in one row. They can be grouped by clicking on one of the columns in the main window (move the cursor on the column to see its description). The grouping helps to locate common root causes for failed events. Columns useful for troubleshooting: •

Out Cause Examples of abnormal call releases: – – – – –

•

Failure Phase Examples: – – – – – – –

•

RAB Active RAB Setup RAB Access RRC Active RRC Access Source S1 HO Setup Target S1 HO Access

Missing Reply Examples: – – –

42

No UE Reply No EPC Messages No UE Messages Radiolink Failure Re-est Reject

RachPreamble InitialUEMessage rrcConnectionSetup

DN09185928

Issue: 02



– • •

4

securityModeCommand

UE Distance HO Attempts

To view the scenario details right-click on the selected row and choose Trace View or Message view. The Trace view shows the scenario in an L3 view, sequence view, or message view.

3.2.5 Tracing with NetAct The NetAct Traceviewer application allows the user to activate and configure tracing. For information on available tools and functions, refer to the NetAct operating documentation: • •

TraceViewer Overview Tracing Subscribers and Equipment

3.3 IP traffic capturing The capturing of IP traffic is possible as a result of the LTE1460 feature. Introduction to the feature The LTE1460: Local and Remote IP Traffic Capturing feature enables Nokia Technical Support, R&D personnel and as well customers' staff to capture the IP traffic of an eNB and fetch it for later offline root-cause analysis. With this feature it is possible to: • •

• • •

Issue: 02

capture ingress and egress IP traffic such as C-plane, U-plane, M-plane or S-plane traffic Web interface for configuration and capture file retrieval, Layer 3 internal measurement points can be selected and U-plane can be excluded to maximize capturing duration. For local use, trace is streamed to configurable TRS port or LMP. For remote use libpcap file is created, compressed and downloaded by user. allows capturing and storage (compressed libpcap file) of at least 60 s of single UE peak data IPv6 capturing is supported

DN09185928

43



3.3.1 Monitoring IP traffic using the LTE1460: Local and Remote IP Traffic Capturing feature The following procedure shows how to capture the IP traffic using the BTS web interface. Before you start To monitor the IP traffic online, an interface analyzer (for example, Wireshark) needs to be configured.

Procedure 1

Login to the BTS Web interface. In any web browser enter the m-plane IP address of the BTS (https://BTS_MPLANE_ADDRESS)

2

Select Tools > Local and Remote IP Traffic Capturing Service Figure 16

3

Select the measurement point for IP traffic capturing. • •

g g

44

Local and Remote IP Traffic Capturing Service window

Point A is located before the IPsec gateway Point B is located after the IPsec gateway (the ciphered protocols need to be decoded) Note: Capturing at point B is useful when you suspect that the IPSec gateway is not working properly. Note: Capturing at point B makes sense if you suspect problems at the IPSec gateway (compare files taken from A and B). Capturing at point B with Wireshark - need additional decoders to see all information.

DN09185928

Issue: 02


4

g


Decide whether the user plane should be captured. Note: The user plane should be included to troubleshoot backhaul IP problems.

5

Select the Streaming (local) or File (remote) capturing output option. After selecting Streaming, you need to provide the destination MAC Address of the machine you want to see the result on. The File option will generate a pcap log file that can be opened for analysis with protocol analyzer (for example, Wireshark).

6

Optionally, provide a password for the capture file encryption.

7

Click the GEN&DOWNLOAD_PCAP button to start the IP traffic capturing. a) BTS deletes the content of the capturing history data structure. b) BTS raises the 7665 Base station Transmission Alarm alarm to indicate that the IP traffic capturing is ongoing. c) BTS starts the IP traffic capturing.

g

Note: The BTS stops the IP traffic capturing after a timeout of 24 hours has been reached.

Result The IP capturing is visible in the protocol analyzer (Streaming mode) or the pcap file is stored in the eNB (File mode).

3.3.2 Analyzing IP capture files with Wireshark The following procedure shows how to use the Wireshark application to analyze IP traffic. Purpose Protocol analyzers such as Wireshark can be used for troubleshooting the following symptoms: • • •

lost packets transmission delays low throughput

Before you start The IP capturing has been initiated and the pcap file is available.

Procedure

Issue: 02

DN09185928

45


1


Open the generated pcap file using Wireshark. Figure 17

IP capture file in Wireshark

In the main view, the potential problems are highlighted with different colors (for example, errors have been highlighted in red).

2

Go to Analyze -> Expert Info . Figure 18

Expert Info view in Wireshark

This view shows a summary of all errors detected in the analyzed file.

3

Use the Packet Details view to map the packets with certain Network Elements (NEs) and User Equipments (UEs). The IP addresses are available in the protocol view. The TEID identifier or the source and destination addresses can be used to locate certain NEs and UEs.

46

DN09185928

Issue: 02



3.4 Sending troubleshooting data to technical support This is an overview of best practices when contacting Nokia technical support. Details to be included in a problem report Problem reports are used to communicate problems and failures to customer support team. Report only one fault in one problem report. Include the attachments in the compressed format. To make the investigation of a problem faster, include the following information in the problem report: • • •

a title that gives a brief description of the problem (see Problems and use cases) a clear and exact description of the problem itself problem background information: – – – –

– – • • •

software and hardware releases situation in the beginning, for example, the first symptoms of the problem situation after the problem occurred operations you made which possibly caused the failure, for example: hardware, software, parameter, feature or configuration changes, including operations in transport network and third party equipment describe if the problem can be reproduced and what actions are required to reproduce this problem troubleshooting and recovery actions that you made

additional problem specific symptom data detailed information on the other third party products (in case of a multivendor environment) the scale of the problem using Nokia Case Types and Nokia Problem Priorities (see tables below)

Table 13

Nokia Case Types Case type

Emergency

Definition Concerns total and partial outages.

Available problem priorities Critical

Total outage: total loss of voice and data traffic capability. An unscheduled event must be longer than 15 seconds. Partial outage: Loss of greater than 10% of the provisioned capacity for origination and/or termination of voice and/or data traffic. Total loss of one or

Issue: 02

DN09185928

47


Table 13


Nokia Case Types (Cont.) Case type

Definition

Available problem priorities

more critical services. An unscheduled event must be longer than 15 seconds. Trouble Resolution

This case type is for problems which caused by a suspected or an identified defect.

Major, Medium, and Minor

Technical Query

This case type is for technical questions regarding daily network operations and maintenance issues.

Major and Medium

Table 14

Nokia Problem Priorities Problem Priority

Definition

Critical

Problems under Emergency Support service.

Major

Only Total or Partial outages which are not avoidable with a workaround solution.

Medium

Total or Partial outages avoidable with a workaround solution. Partial loss of one or more critical services.

Minor

Minor fault not affecting operation or service quality.

3.5 Problems and use cases The most common problems can be divided into several use case categories. When the data that has been collected is used to create a problem report for Nokia Services, it is recommended to use unified naming. Table 15: LTE use cases lists and explains the most common use cases. For several use cases, you can find a troubleshooting methodology in LTE troubleshooting use cases. Table 15

LTE use cases

Use case category

48

Exemplary symptoms

Tools/Site Manager connection problems

Problems with connecting the BTS Site Manager, BTSLog, L3DC via remote IP address or LMP to the BTS.

Software download/activation problems

Software download/activation fails when started from the NetAct/Site Manager or software fallback fails when performed as a recovery action.

(Re)commissioning problems

Problems with starting or finishing (re)commissioning. Missing commissioning parameters, missing units, or unplanned resets.

DN09185928

Issue: 02



Table 15

LTE use cases (Cont.)

Use case category

Exemplary symptoms

Synchronization problems

The RF carrier is outside the target frequency; the RF phase or GPS sync reference signal is missing or starting to drift.

Cells does not come on air

After commissioning, HW/SW reset, or unblock, the cells will not go to 'On Air' state.

Sleeping cell

According to the BTS Site Manager or NetAct, the cell is 'On Air', but no calls can be setup or handed over to the cell.

eNB in a reset loop

The eNB continuously performs reset after it boots up.

Call processing problem/ High RRC Setup failure rate/RACH problems

Calls cannot be established, RACH storms are visible.

Call processing problem/ RAB setup problems

Certain type of service cannot be activated for an established call.

High call drop rate

Calls are dropping.

Antenna line device problems

Problems with antenna line devices (for example, antenna line devices are not detected, VSWr alarms, RET calibration).

Alarms are missing

The eNB has clearly visible problems, but no alarms are visible in the BTS Site Manager or NetAct. For example, unit resets are seen from logs or the RF carrier is lost, but no alarms are visible.

HO problems

Handover-related mobility is not working properly.

Problems with RF modules. Radio unit connectivity/crash/RP3 problems

Issue: 02

KPI/PM counter problems

PM counters are not visible in the BTS Site Manager or NetAct. PM counters/KPIs are having invalid values, for example, success rate over 100%. PM counters cannot be processed with official tools.

Paging problem

Paging messages are not visible in S1/Iub/Uu interfaces, the paging content is corrupted or IDs are invalid.

UL Signal level/quality problems

Interference or signal levels are abnormal, or the IQ constellation is bad.

DSP crash

A single DSP has crashed.

FSP connectivity/crash

The whole FSP has crashed or reset as a recovery action.

BTS crash

The whole eNB has crashed or reset as a recovery action.

Frequent/toggling alarms

The BTS Site Manager or NetAct shows an abnormal number of certain types of alarms, or a certain type of alarms is toggling between 'on' and 'off'.

Memory consumption problems/faults

Memory consumption is abnormally high or low; memory is leaking.

Throughput problems

The DL or UL throughput is lower than expected or is fluctuating.

CS Fallback/Cell reselection/redirect problems

The interoperability between different RAT technologies is not working fluently. For example, phones keep camping in a bad quality network. Note that the non-HO-related mobility performance is often related to a radio network planning.

Time based problems

Different time stamps do not match the NTP time or are not aligned internally.

Block/unblock problems

When blocking or unblocking units, cells, or eNB via the BTS Site Manager, NetAct or test tools, the commands cannot be executed.

DN09185928

49


Table 15


LTE use cases (Cont.)

Use case category

50

Exemplary symptoms

ANR problems

ANR is creating an excessive number of neighbours. Neighbour objects are not created or their content is empty. Automatically created neighbours cannot succeed in a HO.

S1/Iub/X2 interface problems

Either a link setup fails or link connection is breaking.

Call load related performance issues

Problems related mostly to a high call load coming, for example, from mass events. Performance degradation can be seen from eNB's stability, alarms, or call processing KPIs.

Latency problems

The measurable time events (for example, ping times, DL/UL packet delays, signalling responses) are abnormally long.

Hardware issues

Issues related to faulty unit, not working interfaces or recovering services.

DN09185928

Issue: 02


LTE troubleshooting use cases

4 LTE troubleshooting use cases This chapters contains troubleshooting procedures (use cases) for most common problems.

4.1 BTS Site Manager connection problems This section contains uses cases related to problems with the BTS Site Manager application.

4.1.1 Accessing eNB using BTS Site Manager BTS Site Manager can connect to the FSM/FZM locally or remotely. Local access can be established via the local maintenance port or via Bluetooth (FZM only). Remote access is established through the transport backhaul connection. The PC firewall settings must not prevent the start-up of the FTP server on the PC. If the BTS Site Manager connection cannot be established, correct PC firewall settings have to be confirmed. For instructions, see Checking firewall settings in Windows.

g

Note: For disabling other software manufacturers’ firewall settings, see the productspecific user documentation. When using other than Nokia laptops, it is recommended to remove all firewall and security software including Bluetooth and Wireless LAN.

g w

4.1.1.1

Note: BTS Site Manager login time may take a long time if the computer's temp file folder is full. Delete these files to speed up login. Consult your operating system user documentation, if necessary.

NOTICE: Do not change Windows Date and Time settings after launching BTS Site Manager. Date and Time change causes a BTS Site Manager crash.

Problems with BTS Site Manager connectivity to the eNB The following procedure shows how to troubleshoot a problem with establishing a BTS Site Manager connection. Symptoms Connection with BTS Site Manager cannot be established.

Issue: 02

DN09185928

51



Recovery procedure

Procedure 1

Check your PC’s firewall settings.

2

Check that the network card’s IP address, Subnet mask and Default Gateway match BTS IP settings.

3

Check that the IP address is correct in the BTS Site Manager launch window. Check also that there are no other overlapping BTS Site Manager connections.

4

Check that the username and password are correct.

5

Check that the LMP cable is connected properly (local connection).

6

Check that BTS Site Manager version is correct.

7

In case of local connection, ping the System Module at 192.168.255.1 (for FZM, ping at 192.168.255.129 or 192.168.255.245 if connected through Bluetooth). In case of remote connection ping the BTS public (M-Plane) IP address.

4.1.1.2

Checking the PC firewall settings in Windows If the connection to the BTS Site Manager cannot be established, the PC network settings might not be specified correctly Symptoms Connection to the BTS Site Manager cannot be established. Recovery procedure To check the firewall settings in Windows XP, please refer to Checking PC firewall settings in Windows XP Professional To check the firewall settings in Windows Vista/7, please refer to Checking the PC firewall settings in Windows Vista/Windows 7 For disabling other software manufacturers’ firewall settings, see the product-specific user documentation.

52

DN09185928

Issue: 02


4.1.1.2.1


Checking PC firewall settings in Windows XP Professional Steps

Procedure 1


2

Check PC network settings. a) Select Start → Settings → Network Connections. Right-click on Local Area Connection → Properties. Select the → Internet Protocol (TCP/IP) → Properties. b) If needed, specify network settings to match the following: • •

IP address: 192.168.255.126 Subnet mask: 255.255.255.0

c) Click OK in both windows.

3

Ping the System Module. Check first the connection to System Module by pinging it locally at 192.168.255.1. In case of remote connection use remote IP address. If the ping is successful, try disabling the firewall.

4

Disable the Windows Firewall. a) Select Start → Settings → Network Connections. Right-click on Local Area Connection → Properties. Select the → Advanced- tab → Settings.... b) Select the Off (not recommended) option. c) Click OK.

Issue: 02

DN09185928

53


Figure 19


Disabling Windows Firewall in Windows XP Professional

If the firewall cannot be disabled, allow traffic from the BTS IP address.

4.1.1.2.2

Checking the PC firewall settings in Windows Vista/Windows 7 Steps

Procedure 1


2

Check the PC network settings. a) Select Start → Control Panel→ Network and Internet→ Network and Sharing Center → Manage network connections. Right-click on Local Area Connection → Properties. Select the → Internet Protocol Version 4 (TCP/IPv4) → Properties.

54

DN09185928

Issue: 02



b) If needed, specify network settings to match the following: • •

IP address: 192.168.255.126 Subnet mask: 255.255.255.0

c) Click OK in both windows.

3

Ping the System Module. Check first the connection to System Module by pinging it locally at 192.168.255.1. In case of remote connection, use the remote IP address. If the ping is successful, try disabling the firewall.

4

Disable the Windows Firewall. a) Select Start → Control Panel → Security→ Windows Firewall→ Turn Windows Firewall on or off. b) Select the Turn off Windows Firewall (not recommended) option. c) Click OK. If the firewall cannot be disabled, allow traffic from the BTS IP address.

4.1.1.3

Remote connectivity via transport backhaul (m-plane) Purpose To remotely connect to the eNB, follow this steps:

Procedure

Issue: 02

DN09185928

55


1

Launch the BTS Site Manager application

2

In the Connect tab, enter the m-plane IP address assigned to the eNB.

3

Log in using your credentials or with the default user name/password (Nemuadmin/nemuuser). Figure 20

4.1.1.4


BTS Site Manager log-in screen

Remote connectivity over Bluetooth (FZM only) Purpose To remotely connect to the eNB over Bluetooth, follow this steps:

Procedure 1

g

56

Make sure that Bluetooth is enabled in the BTS. For instructions, see Commissioning Flexi Zone Micro BTS LTE. Note: Ensure legacy default values match the setup parameters.

DN09185928

Issue: 02



Figure 21

Issue: 02

Legacy pairing settings

DN09185928

57


2

Enable Bluetooth interface in the BTS Site Manager laptop, and then choose Add a device.

3

Select the Bluetooth name of the FZM you wish to pair, and then connect.

4

Select the option which requires manual entry of a pairing code. Figure 22

58


Select manual pairing option

DN09185928

Issue: 02


5

Issue: 02


Enter LEGACYOTP as pairing code. Figure 23

Enter the legacy pairing code

Figure 24

Successful pairing

DN09185928

59


6

The operating system will proceed through the pairing process, and then the FZM will be shown under the Bluetooth Devices on the Devices and Printer window of the Control Panel. Once paired, establish connection by selecting Network Connect to establish full connection. Figure 25

t

w

Once full connection is established, assign an IP address to the personal area network (PAN) on the laptop, which is used to connect to the BTS Site Manager. a) Assign IP address (192.168.255.243) to the PAN network interface. b) Assign a subnet mask value of 255.255.255.248. c) DNS address is not required.

8

60

Establishing PAN connection

Tip: If you are monitoring the progress on a BTS Site Manager, you will see Bluetooth alerts in faults window, indicating that Bluetooth is enabled and that a Bluetooth connection has been established.

7

4.1.1.5


Ping 192.168.255.245, the Bluetooth IP address of the FZM, to verify successful connectivity. If ping is not successful, check if the assigned IP address and subnet mask are correct. Also, check the firewall settings. For instructions, see Checking PC firewall settings in Windows.

Local connection problems

DN09185928

Issue: 02


4.1.1.5.1


Local connectivity via local maintenance port Purpose To localy connect to the eNB, follow this steps:

Procedure 1

Directly connect your laptop to the backhaul port assigned as local maintenance port (LMP) in the eNB. IP adress of your laptop must be from the same network as the eNB.

2

Launch the BTS Site Manager application.

3

On the BTS Site Manager login screen, set the IP address to Local.

4

Log in using the default user name and password (Nemuadmin/ nemuuser. Figure 26

4.1.1.5.2

Local connectivity log-in screen

Cannot ping modules through LMP port Follow this procedure if you are unable to ping modules. Symptoms No ping response from FCM, RFs or FSPs.

Issue: 02

DN09185928

61



Recovery procedure

Procedure 1

Check that the feature “Security for Ethernet” is disabled. Parameter actEthernetSec value has to be set to false (0) to allow pinging the modules.

4.1.2 Changing BTS Site Manager local account This procedure shows how to change the BTS Site Manager log-in credentials. Symptoms BTS Site Manager informs that password and user name are expired or you need to change credentials due to security reasons. Recovery procedure

Procedure 1

In the main BTS Site Manager window, choose the Configuration ► Change Local Account menu item to open the Change Local Account dialog box.

2

Enter the new user name in the User name field.

3

Enter the new password in the Password field.

4

Rewrite the password in the Confirm password field.

5

Click the Send button.

6

Click the Close button to close the dialog box.

4.1.3 Configuring Exceed for BTS Site Manager launching The following procedure shows how to configure the Exceed application.

62

DN09185928

Issue: 02



Purpose Exceed application is used to launch BTS Site Manager under Linux. To work properly, Exceed must be configured beforehand. Before you start Exceed configuration procedure needs to be performed on all GUIS.

Procedure 1

Log in GUIS as a domain administrator. Log off all users who are using Exceed session via Windows Task Manager.

2

Access Exceed configuration window. Go to

C:\ProgramFiles(x86)\Hummingbird\Connectivity\14.00\Default User\Exceed and click Exceed.xcfg.

3

Enter Display and Video settings. a) Change Window Manager field to Native. b) Apply the changes by clicking Validate and Apply Changes button.

4

Open Command Prompt. a) Go to C:\ProgramFiles(x86)\Hummingbird\Connectivity\14.00\ Accessories. b) Execute the following commands: 1. HumSettings.exe UPDATELASTMODIFIED=ExceedMP 2. HumSettings.exe REFRESH=ALL USER=ALL

4.1.4 Access to the Linux shell Access to the Linux console shell might be necessary for specific troubleshooting purposes. SSH login must be enabled from the TRS Browser via https. This can be done remotely via M-plane IP address or locally via BT or the LMP address 192.166.255.129.

g

Issue: 02

Note: If the M-plane IP address does not work but the LMP address does, remove any proxy settings set in the browser.

DN09185928

63


t


Tip: SSH is also accessible via Bluetooth (FZM only). Figure 27

Enabling the SSH service for Linux shell access

4.1.5 GPS maintenance access not possible through remote connection Follow this procedure if the GPS maintenance access is not possible through a remote connection. Symptoms Error The BTS is not responding. Check the connection. appears after choosing GPS Maintenance Access in the BTS Site Manager. Recovery procedure

Procedure 1

Enable a GPS port. Type https://x.x.x.x/protected/EnableRemoteFcmTestPorts.cgi into a web browser, where x.x.x.x stands for BTS IP address.

64

DN09185928

Issue: 02



2

Enter login and password.

3

Run GPS Maintenance Access in the BTS Site Manager. Perform the needed actions.

4

Disable a GPS port after finishing the needed actions. Type https://x.x.x.x/protected/DisableRemoteFcmTestPorts.cgi into a web browser, where x.x.x.x stands for BTS IP address.

4.2 Software download/activation problems This section contains use cases related to the most common software issues.

4.2.1 Incompatible BTS software version Follow this procedure to troubleshoot an issue with incompatible software. Symptoms • •

There are grey modules in the Hardware view Fault Incompatible SW version detected is active.

Recovery procedure

Procedure 1

Check that the BTS SW version is the latest available.

2

If the SW upgrade is unsuccessful, BTS Site Manager might not display the BTS SW used. Try a new SW download.

3

If the PC connection or BTS SW download is unsuccessful, ensure the correct PC settings and that the used cable is working and of correct model. You can try to ping the System module, for example when connecting locally by using the Command Prompt: ping 192.168.255.1.

Issue: 02

DN09185928

65



For information on correct PC firewall settings, see Checking firewall settings in Windows. 4

After a successful BTS SW download, launch a new BTS Site Manager session to confirm the use of the correct BTS Site Manager version.

5

Update to a new BTS Site Manager version, if needed.

4.2.2 Software update to BTS site fails A software update can fail if the firewall settings prevent the start-up of the FTP server on the PC. Symptoms • •

BTS Site Manager shows the error message Cannot update SW files. BTS Site Manager shows the error message Error in file server during SW download.

Recovery procedure

Procedure 1

Check the SW Download Report for any errors.

2

Check if there are any active alarms indicating a HW fault.

3

Check the firewall settings. For instructions, see Checking firewall settings in Windows.

66

4

Check the network card’s IP address.

5

Try updating the SW again.

6

If the SW update still fails, collect the eNB snapshot and contact Nokia Services.

DN09185928

Issue: 02



4.2.3 Software version mismatch Follow this procedure if the SW version of the transmission sub-module does not match the SW version of the system module. Symptoms • •

BTS Site Manager shows fault Incompatible SW version detected (0023). Transmission-related settings are greyed out in the BTS Site Manager (only if connected locally).

Recovery procedure

Procedure 1

Check if there are no connectivity problems with the transmission module. For details, see #unique_90.

2

Download a new software release to the BTS. Download a new software release to Flexi Multiradio BTS LTE. After the SW download, analyze the software download report and make sure that the software download is reported as successful.

4.2.4 Problems with local software download from BTS Site Manager Follow this procedure if a software download fails when attempted from a locally connected BTS Site Manager. Symptoms •

Issue: 02

The BTS software update fails as shown in Software update failed.

DN09185928

67


Figure 28

g

Software update failed

Note: During software download the inbound connections from the BTS to the host PC are used. The host PC firewall might be blocking these connections and a file transfer cannot be initiated. •

68


The Software Update Report displayed with View Report button shows that the download of the TargetBD (Target Build Descriptor) xml file has failed as shown in Software update report.

DN09185928

Issue: 02



Figure 29

Software update report

Recovery procedure

Procedure 1

Check the SW Download Report for any errors.

2

Check the firewall settings. The local software update procedure requires that client PC firewall allows traffic on ports from 13062 to 13092. Check that these TCP ports are allowed for inbound and outbound traffic. Additional information about connections between network elements can be found in the Communication Matrix document.

Issue: 02

3

Try updating the SW again.

4

If the SW update still fails, collect the eNB snapshot and contact Nokia Services.

DN09185928

69



4.3 (Re)commissioning problems This section contains use cases related to verification of normal site operation and to the most common commissioning issues.

4.3.1 Checking the BTS commissioning file The following procedure helps in solving problems with the BTS commissioning file. Symptoms • • • • • • • •

wrong cell ID uplink and downlink power are not correct wrong IP setting. problems in start-up problems in transmission and IP functionality problems in HO and cell range functionality problems in peak data rates, modulation or spatial mode problems in attach and detach functions

Recovery procedure

Procedure 1

Check the BTS commissioning file to ensure the correct BTS configuration and settings. Check that the commissioning parameters and settings match the network planning. In case of mismatch correct commissioning with the reconfiguration commissioning option of BTS Site Manager.

2

Check that the cabling and antenna line settings in commissioning match the actual cabling of the BTS.

3

Check that the Transmission settings in TRS commissioning match the settings in Network plan.

4.3.2 Cannot save commissioning file or send parameters The security settings in Windows Vista and Windows 7 may prevent saving and/or sending the commissioning parameters.

70

DN09185928

Issue: 02



Symptoms In commissioning, sending and/or saving commissioning parameters is not possible when using Windows Vista or Windows 7. Recovery procedure Follow this procedure to set access rights to the BTS Site Manager folder:

Procedure 1

Right-click the folder and select Properties.

2

Click on the Security tab.

3

Click Advanced in the lower right-hand side of the view. In the Advanced Security Settings window that pops up, click on the Owner tab and click Edit.

4

Click Other users or groups.

5

Click Advanced in the lower left-hand corner and click Find Now.

6

Scroll through the results and double-click on your current user account.

7

Click OK to all of the remaining windows except the first Properties window.

8

Select your user account from the list up top and click Edit.

9

Select your user account from the list up top again and then in the pane below, check Full control under Allow. You will get a security warning, click Yes.

4.3.3 Verification of normal eNB site operation Follow this procedure to verify if the eNB site is fully operational.

Procedure

Issue: 02

DN09185928

71


1


Open the BTS Site Manager and verify that the BTS statuses. In the bottom right corner of the window, you can find following information: • • •

OAM connection operational state TRS state

Figure 30

2

Select IP Connectivity Test from the BTSSM Tests drop-down menu. Figure 31

3

IP Connectivity Test in BTS SM

Set the Scope to All configured hosts and click Start, and then verify if all configured items indicate Connected when the test is executed. Figure 32

72

BTS statuses in BTS Site Manager

IP Connectivity test result

DN09185928

Issue: 02


4


Verify that UEs are connected to the site and that the site is passing traffic by displaying counters through the BTS Site Manager. See Viewing counters in BTS Site Manager for more information about viewing counters in BTS Site Manager. Important counters that need verification" • •

LTE UE State/Signalling Connection Establishment/SIGN_CONN_ESTAB_COMP (should be non-zero) LTE Cell Resource/DL PRB Usage/PRB_USE_PDSCH (should be non-zero)

4.4 Synchronization problems This section contains procedures for verification of synchronization.

4.4.1 GPS Synchronization Follow this procedure to verify the GPS synchronization. Purpose

g

Note: The example procedure is shown for FWGB, but it applies to every FZM/FSM module.

Procedure 1

Verify GPS synchronization in the BTS Hardware view.

2

Click FWGB System.

3

Click FYGA to see GPS status. Individual satellite signal level and operational status are then displayed.

4.4.2 Timing Over Packet (IEEE.1588) Synchronization Follow this procedure to verify the ToP synchronization.

Issue: 02

DN09185928

73



Procedure 1

On the left side of BTS Site Manager choose IP tab and switch to the Timing over Packet.

2

Verify that all settings are correct. Figure 33

Example ToP configuration

4.4.3 Fallback solution for phase synchronization The following procedure is a fallback solution in case an FDD-eNB in a phase synchronization mode is not able to complete startup due to a missing GNSS receiver control interface. Purpose The problem was visible in the field. As a result of an installation issue with a GNSS receiver or hardware malfunction, the GNSS receiver control interface (containing, among others, time of day) is not available to the eNB. Without this information, the eNB is not able to go to an onAir state. As a solution, the autonomous switching from a phase to a frequency mode before a system frame number alignment was introduced.

Procedure

74

DN09185928

Issue: 02



1

The eNB performs phase mode startup.

2

The eNB detects that the GNSS receiver does not provide a sufficient signal to be used as a phase synchronization reference. The following faults can be raised: • • • • • •

3

4011 ("GPS receiver alarm: control interface not available") 4122 ("GPS receiver alarm: no tracking satellites") 4123 ("GPS receiver alarm: survey in progress") 4124 ("GPS receiver alarm: no stored position") 4125 ("GPS receiver alarm: position questionable") 4126 ("GPS receiver alarm: EEPROM invalid")

The eNB waits for other phase synchronization sources or for the issue with GNSS to be resolved, until the timer expires. Fault 4153 ("Reference clock missing in startup") is raised with additional info that startup will be continued in a frequency mode with deactivated features.

4

If the eNB is not syncHubSlave in RfSharing, the tuning mode is changed to a frequency mode. The eNB deactivates features requiring network-level phase synchronization (for example OTDOA, eICIC, MBMS, SIB8 broadcast) and features related to inter-eNB carrier aggregation.

5

The eNB performs frequency tuning.

Result The eNB goes to an onAir state. When GNSS becomes available, all GPS receiver alarms and fault 4153 are cleared. The eNB raises fault 4001 ("BTS reset required"), informing that a manual reset is required. There is no need to configure the eNB to a phase synchronization mode or manually enable features dependent on the phase synchronization. Once a manual reset is done, those features will be on air automatically.

4.5 Cell does not come on air Follow this procedure to troubleshoot an issue with a cell not coming on air. Symptoms

Issue: 02

DN09185928

75



After commissioning, HW/SW reset, or unblock, the cells will not go to 'On Air' state. Recovery procedure

Procedure 1

Verify that BTS and TRS are commissioned.

2

Verify S1 link connectivity – there will be an S1 link alarm.

3

Verify that site has a valid timing source. Refer to Synchronization problems.

4

Check for configuration-related issues that need to be corrected by recommissioning the site.

5

If problem is not caused by configuration issues, then it is likely an FZM/FSM hardware issue. That means that the unit must be swapped.

4.6 Sleeping cell Follow this procedure to troubleshoot the sleeping cell problem. Symptoms According to the BTS Site Manager or NetAct, the cell is 'On Air', but no calls can be set up or handed over to the cell. No alarms that would indicate the problem have been raised. Recovery procedure

Procedure 1

Check if KPIs are showing traffic in the cell.

2

Check if there is no mismatch in the cell configuration. Check the configuration of: • • •

76

power levels maximum number of RRC connected users neighbor cells

DN09185928

Issue: 02



3

Check if there are no faulty radio circuits or other critical hardware.

4

Check the configuration of broadcast information. UEs might not be able to read the system information required to access the cell.

5

Collect the logs (eNB snapshot, cell traces) and restart the site.

4.7 eNB in a reset loop Follow this procedure to troubleshoot the eNB-in-a-reset-loop issue. Symptoms The eNB continuously performs reset after it boots up. It gets stuck in a continuous reset loop, and it is not possible to gather the log files to analyze the fault. Recovery procedure

Procedure 1

Check all cables and connectors. Ensure the connectors are not loose, and cables are not bent.

2

Reboot the RF modules.

3

Switch the power off and turn it on again.

4

If the problem occurred after an SW upgrade, revert to an older SW version.

5

Manually replace the site configuration file.

4.8 Troubleshooting call processing problems The following generic steps describe a typical call processing troubleshooting procedure.

Issue: 02

DN09185928

77



Before you start You can use following tools for a call processing troubleshooting: KPI data viewer (for example, NetAct Reporting Suite) KPI and counter documentation cell trace data Wireshark logs eNB snapshot

• • • • •

Recovery procedure

Procedure 1

Check which KPIs are indicating bad performance.

2

Check the KPI formula to find out which counters are causing the KPI degradation. The counter description ('Updated' field) helps to determine the signaling scenario that affects the degradation. This information is useful to decide which interface should be further investigated. Details are explained in Performance monitoring.

3

Check when the KPI degradation started. Check if any significant activities (for example, a site reconfiguration or SW upgrade) have been done at the time when the KPI degradation started.

4

Analyze available logs and traces to gather more details. Based on the counter investigation, the following data can be analyzed: • • •

call traces: the analysis can be done using the L3DA interface monitoring: Wireshark can be used eNB snapshot: it can be opened with the BTS Site Manager

4.8.1 High RRC Setup failure rate/RACH problems

g

Note: This is an example of a call processing troubleshooting area along with examples of useful data. Related KPIs • •

LTE_5218d Total E-UTRAN RRC Connection Setup Success Ratio RRC Connection Setup Failure Ratio per Cause (LTE_5229c, LTE_5230c, LTE_5590a)

For KPI descriptions with the counter formulas, refer to the Specifications of LTE RAN Key Performance Indicators (DN0969678) document.

78

DN09185928

Issue: 02



For counter descriptions, refer to the LTE Performance Measurements (DN0944219) document. L3 Data Analyzer indicators • • •

Out Cause: No UE Reply Failure phase: RRC Setup Missing Reply: RrcConnectionSetup

Related parameters • • •

g

Timer T300 (T300) Maximum number of RRC connections (maxNumRrc) Maximum number RRC emergency (maxNumRrcEmergency) Note: Parameter optimization is a complex task; therefore, it is recommended to use the Nokia Network Planning and Optimization (NPO) Services.

For parameter descriptions, refer to the Flexi Multiradio BTS LTE Commissioning, RNW and Transmission Parameters (DN09147584) document.

4.8.2 RAB setup problems

g

Note: This is an example of a call processing troubleshooting area along with examples of useful data. Related KPIs • •

LTE_5017a E-UTRAN E-RAB Setup Success Ratio E-UTRAN E-RAB Setup Ratio per Cause (LTE_5086a, LTE_5087a, LTE_5088a, LTE_5089a)

For KPI descriptions with the counter formulas, refer to the Specifications of LTE RAN Key Performance Indicators (DN0969678) document. For counter descriptions, refer to the LTE Performance Measurements (DN0944219) document. L3 Data Analyzer indicators • •

Out Cause: RAB Setup Failure Failure phase: RAB Setup

Related parameters • • •

g

Max number act DRB (maxNumActDrb) Maximum number of active UEs (maxNumActUE) Max number QCI1 DRBs (GBRs) (maxNumQci1Drb) Note: Parameter optimization is a complex task; therefore, it is recommended to use the Nokia Network Planning and Optimization (NPO) Services.

For parameter descriptions, refer to the Flexi Multiradio BTS LTE Commissioning, RNW and Transmission Parameters (DN09147584) document.

Issue: 02

DN09185928

79



4.9 Antenna line device problems This section contains use cases related to issues with antenna line devices.

4.9.1 Checking antenna line and MHA Follow this procedure to troubleshoot problems with an antenna line and MHA. Symptoms • • •

There are problems with antenna line functionality, for example active alarms. Uplink and downlink power levels are not correct, and do not match with the related commissioning parameters. MHAs are not visible in HW view or Commissioning.

Recovery procedures Use proper instruments for measuring the antenna line, like Site MasterTM Cable and Antenna analyzer, for verifying that connectors, feeders and antennas are installed correctly. Use a spectrum analyzer to check from feeder cable that there are no incoming external interferences causing problems in the RX channel.

Procedure 1

Check that antenna cables (main and diversity) are connected to correct ports.

2

Re-check antenna line connectors (7/16” connectors with moment 25 Nm) for loose connections. Check also cable conditions to see if there is any visible damage.

3

In MHA problems, check first that the commissioning has been done properly. Check that the DC power is going to the MHA. In case MHA is not visible in HW view or Commissioning measure DC voltage from antenna connector with proper tool. Missing DC voltage indicates a problem in RF module and existing DC voltage problem in either MHA, feeder cable or connections between the MHA and feeder cable.

80

DN09185928

Issue: 02


4


Check the VSWR and antenna line faults for example with the Site Master to ensure that connectors, feeders and antennas are ok. VSWR fault indicates if there are problems in the antenna line devices. In that case, check the cables, connectors, MHAs and antennas. The overcurrent fault indicates if there is a short circuit in the antenna line.

4.10 Troubleshooting hardware problems This section contains use cases for troubleshooting hardware issues as well as performing maintenance and recovery.

4.10.1 Performing routine maintenance tasks Follow this procedure during routine site visits to ensure that the base station remains in a good condition. Purpose The Flexi BTS design itself is maintenance-free and therefore has no specified regular maintenance intervals.

Procedure 1

Check whether the maintenance straps inside the module covers are damaged. Make sure there are no visual defects. Replace worn or damaged maintenance straps.

2

Check the fans and air circulation. For modules with fans, keep the fans free of leaves and debris. When cleaning the fans, first remove the power connector from the fan or disable power to the module (depending on the variant). For all modules, keep air inlets and outlets free of obstructions to maintain proper air circulation and prevent overheating of the BTS. Keep the BTS environment clean of dust.

3

Check the connector seals. Check the connector seals visually. Replace worn or broken seals.

Issue: 02

DN09185928

81


4

Make sure that all connector IP seals, plugs and boots are connected correctly. •

•

82


The LMP port or any unpopulated connector openings have IP protection Figure 34

LMP IP plug installed

Figure 35

LMP port IP cap not reinstalled after maintenance

IP seals are firmly fitted around all edges of the IP boot

DN09185928

Issue: 02


Issue: 02


Figure 36

Connector IP boot correctly installed firmly in place

Figure 37

Connector IP boot incorrectly installed (IP seals not firmly in place on any edge)

DN09185928

83


Figure 38

•

Connector IP boot incorrectly installed (not pushed all the way in)

Ensure that cables and connectors are fitted with IP boots. Figure 39

84


Cable installed without IP boot

DN09185928

Issue: 02



5

Check the antenna line and transmission cable connectors.

6

Check the module fixing screws. Check the tightness of the module fixing screws. For RRH, also check the tightness of the solar shield fixing screws. Tighten all loose screws and replace all damaged or missing screws

4.10.2 Troubleshooting Flexi transmission module This section contains uses cases related to the transmission module problems.

4.10.2.1

BTS reset required Follow this procedure if BTS Site Manager displays a pop-up window informing that a BTS will be reset. Purpose If a change to the BTS IP settings has been received from the BTS Site Manager, the BTS needs to be restarted in order to activate them. Recovery procedure

Procedure 1

Reset the BTS manually when all required BTS IP settings have been set. To verify the IP settings, select View ► IP menu item or click the IP button on the left-side menu of BTS SM. Check the settings in the tabs.

4.10.2.2

Dead Peer detected Follow this procedure in case of a dead peer problem. Symptoms BTS Site Manager shows SCTPendpoint failure (36665) and LEDs are red in the transmission module. Recovery procedure

Issue: 02

DN09185928

85



This fault indicates that dead peer has been detected in one of the IPsec associations of the FTM. Possible causes: • • •

a cable is cut or there is no cable connected to the interface the signal is excessively attenuated the port at the connected far end node is switched off

Procedure 1

Check whether the cables are connected to the correct connectors. Ensure the connectors are not loose, and cables are not bent. For details on cabling principles, see Cabling Flexi Multiradio BTS LTE.

2

Check whether the interface at the far-end interface is switched on.

3

Check whether backhaul routers and repeaters are compatible, and whether they are not faulty.

4

Check whether there are active faults at the far end. If there are active faults at the far end, follow the instructions for handling such faults. See Flexi Multiradio BTS LTE Faults for more information.

4.10.2.3

Ethernet interface is not working Follow this procedure in case of issues with an Ethernet interface. Symptoms • •

configured Ethernet interface is not working (for example, "Synchronization lost" message appears) Ethernet interface is enabled and configured, but does not carry any user plane traffic.

Recovery procedure For proper operation of Ethernet interfaces in Flexi Multiradio BTS LTE, the interfaces need to be enabled and there should be no faults active for the interfaces in use. FTIB and FTLB have three Ethernet interfaces but only one should be used. In FTIB interfaces EIF1 and EIF2 cannot be used at the same time.

Procedure

86

DN09185928

Issue: 02


1


Check that the cables are connected to the correct connectors. Make sure the connectors are not loose and cables are not bent. For details on cabling principles, see Cabling Flexi Multiradio BTS LTE.

2

Check that the Ethernet interface is enabled. Verify that the non-operational interface is enabled in BTS Site Manager. Select Configuration -→ TRS Configuration -→ Physical Interfaces. Make sure that the interface is enabled (the In use box is selected at the interface). Click Send to send new parameters.

3

Check that IP configuration is correct.

4

Check that there are no active faults for the Ethernet interface. Verify that there is no active fault for the non-operational interface. In BTS Site Manager, select the Faults view. Make sure that there are no faults active for the Ethernet interface: LOS on unit $U, [ethernet] interface $IF or Missing or noncompliant SFP module on unit $U, interface $I faults. See the Flexi Multiradio BTS LTE Faults document for more details on these faults.

g

Note: The $U parameter stands for unit number and the $I parameter stands for interface number. Both parameters are to be replaced with a corresponding unit and interface values.

4.10.3 Identification of faulty units This section describes how to identify Flexi Multiradio BTS LTE modules and submodules, and how a faulty unit can be located with the NetAct system. The explanation of Diagnostic Info field value is shown in Table 16: Explanation of NetAct Diagnostic Info values. Table 16

Field No.

Explanation of NetAct Diagnostic Info values

Value Type

Meaning

1

Hexadecimal

Not relevant (always 0x64)

2

Hexadecimal

Values from 0x01 to 0x02: 0x01 – master FSM0x02 – extension FSM Values from 0x0B to 0x2B:

Issue: 02

DN09185928

87


Table 16

Field No.


Explanation of NetAct Diagnostic Info values (Cont.)

Value Type

Meaning Value

OptIF

FR place in chain

0x0B(11)

1

1

0x15(21)

1

2

0x1F(31)

1

3

0x29(41)

1

4

0x0C(12)

2

1

0x16(22)

2

2

0x20(32)

2

3

0x2A(42)

2

4

0x0D(13)

3

1

0X17(23)

3

2

0X21(33)

3

3

0X2B(43)

3

4

3

Hexadecimal

Not relevant (always 0x64)

4

Four digits + string

The first 4 digits indicate the BTS fault ID that caused this alarm to be triggered, second string indicate the type of the plug-in unit that is accused of the fault. Possible unit type: FSM, FSP, FT, FF, MHA, TILT, FR, FA, extension FSM, FBIA, FSM FANs.

5

Hexadecimal

Id of unit from filed “Type of unit”. Corresponds to SEM unit number.

6

Hexadecimal

Not relevant

The following examples illustrate the how to read Diagnostic Info. Case 1: Diagnostic info: 64 b 64 1838Antenna 3 0

88

DN09185928

Issue: 02


Table 17



Field No.

Value

Meaning

1

0x64

Not relevant

2

0x0B (11)

OptIF 1 FR in chain 1

3

0x64

Not relevant

4

1838Antenna

BTS fault ID 1838 (VSWR alarm). Fault source Antenna

5

0x03

Id of Antenna 3

6

0x00

Not relevant

Summary: Fault ID 1838 (VSWR Alarm), source Antenna id 3 on first FR in chain connected with BTS via OptIF 1. Case 2: Diagnostic info: 64 0C 64 0133Antenna 02 00 Table 18


Field No.

Value

Meaning

1

0x64

Not relevant

2

0x0C (12)

OptIF 2 FR in chain 1

3

0x64

Not relevant

4

0133Antenna

BTS fault ID 133 (Rx signal level failure). Fault source Antenna

5

0x02

Id of Antenna 2

6

0x00

Not relevant

Summary: Fault ID 133 (Rx signal level failure), source Antenna id 2 on second FR in chain connected with BTS via OptIF 2. The following figure shows how the modules are defined in Flexi Multiradio BTS LTE.

Issue: 02

DN09185928

89


Figure 40


Flexi Multiradio BTS LTE construction, stacked installation

RFModule2

Transmission sub-module SystemModule

RFModule1

PowerModule (optional) Plinth The following figure shows how the fans are defined in Flexi Multiradio BTS LTE. Figure 41

90

Fan mapping in Flexi Multiradio BTS LTE

DN09185928

Issue: 02



4.10.4 Identification of faulty FZM unit This chapter describes how to identify Flexi Micro BTS modules and sub-modules. The FZM is a single field replaceable module. The only components of the FZM that can be swapped in the field are the fiber transport SFP, and the antennas (BT, GPS, LTE).

w

The FZM unit is likely to be installed in a location that is not readily accessible, so ensure that all reasonable troubleshooting steps have been taken prior to replacing a FZM unit. To minimize unwarranted HW replacement, it is recommended that the following must be observed: • • •

Power cycle the FZM if possible to see if the reported faulty condition still exists. Re-commission the FZM with a golden configuration rather than site-specific to establish if the faulty condition is related to some incorrect configuration parameter. Refer to FZM Recovery.

4.10.5 Problems in connectivity between BTS modules Problems with connections between the different modules may be caused by nonoperational Ethernet interfaces, faulty cabling, and non-functional processors. Symptoms Connections between different BTS modules fails and failure indication appears. Recovery procedure

Procedure 1

Check that the cables are connected to the correct connectors. Make sure the connectors are not loose and cables are not bent. For details on cabling principles, see Cabling Flexi Multiradio BTS LTE.

2

Check if any alarms indicate faulty modules.

3

Run a site test to test the Ethernet interfaces. Before you run the site tests, check the cabling and faults. Select the Tests → Site Tests menu item in the BTS Site Manager. For detailed instructions, see BTS Site Manager online help. The test shows the results for the following modules:

Issue: 02

DN09185928

91


• • •


Flexi System Module (FSM): shows test results for each Flexi Multiradio BTS signal processing unit (FSP) and Flexi transport submodule (FT) Flexi Radio Modules (FR): shows test results for uplink and downlink, and for each filter (FF) belonging to the FR Flexi Power Module (FP)

Check for problematic units in the site test results. Change the unit(s).

4.10.6 FZM Recovery This chapter focuses on recovering service to an FZM, especially prior to swapping out an FZM unit. Recovery by resetting FZM using BTS SM If required, the FZM can be resetted using BTS Site Manager. Figure 42: Reset button in the BTS Hardware tab shows location of reset button.

92

DN09185928

Issue: 02



Figure 42

Reset button in the BTS Hardware tab

The FZM should return to the operational state in approximately 4 minutes after reset. Then the BTS SM can be reconnected and normal operation of site can be verified (see Verification of normal eNB site operation.) Recovery from rolling reboot If the FZM is in a continuous reboot sequence, the only recovery action that can be performed is to power cycle the FZM unit. If the condition persists again after the power cycle, then the FZM must be replaced. The only way to power cycle the FZM unit is to physically remove and re-connect the power to the unit. This can be done at the unit itself (which may not be easily accessible) or at the power feed source (for example, fuse or circuit breaker) in the electrical supply. FZM reset button There is a reset button located behind the weather sealed cover where the LEDs are located. The reset button has two functions: • •

Issue: 02

a momentary push of the button will reset the FZM similar to a reset performed from the BTS Site Manager. holding the reset button for longer than 5 seconds will result in a configuration reset.

DN09185928

93



In case of configuration reset, the radio configuration will be restored to factory default, and the transport configuration will be preserved (including IP connectivity and security certificates.) The FZM will still be accessible after a configuration reset.

94

DN09185928

Issue: 02

Troubleshooting LTE RAN NOKIA

Recommend Documents