Debugging Techniques For Embedded Systems

Debugging Debugging Techniques for Embedded Systems using Real-Time Real-Time 1 Software Software Trace Anthony Berent – ARM 2 Abstract Until a few years ago real-time software trace was only available using a Logic Analyzer or In Circuit Emulators. These tools were expensive and difficult to use, and as such were often only used for software debugging when everything else has failed. Also, as processors became more complex and more deeply embedded, it became less and less practical to make the signals required for such debugging accessible to the tools. In the last few years, however, processors, such as ARM’s cores, have included real-time trace facilities. The easy availability of such real-time trace facilities allows new debugging paradigms to be developed. In this paper I describe some of these techniques for debugging embedded systems. I also show that real-time trace can have advantages even for debugging systems that do not have hard real-time requirements.

Introduction Most software engineers spend a significant portion of their time looking for bugs in software. Traditionally this is done using a start/stop debugger. Using such a debugger the engineer can define breakpoints at which the system is to stop, and then examine the state of the system when the program hits a breakpoint. The engineer can, from this, see how the software is failing. Unfortunately such techniques are almost useless for real-time systems, since stopping the system changes its real-time behavior. This causes the problems one is trying to debug to disappear as soon as a breakpoint is introduced. An alternative technique is needed. One such technique is real-time software trace, in which the some hardware captures the sequence of instructions, and possibly data accesses, executed by the processor. The engineer then uses this historical record of the behavior of the processor to investigate the bug. 1

© ARM Ltd. 2002

2

Anthony Berent is a Principal Engineer at ARM Ltd, and is the lead engineer on ARM's Trace Debug Tools. He has over 20 years experience as a software engineer, mainly working with embedded systems, and has spent most of the last 10 years developing debugging tools and techniques for embedded systems.

Debugg Deb ugging ing Techniqu Techniques es for Embedde Embedded d Systems Systems using using Rea Real-Ti l-Time me Software Software Trace Trace

Page Page 1

Until a few years ago real-time software trace was only available using a Logic Analyzer or In Circuit Emulators. These tools were expensive and difficult to use, and as such were often only used for software debugging when everything else has failed. Also, as processors became more complex and more deeply embedded, it became less and less practical to make the signals required for such debugging accessible to the tools. In the last few years, however, processors, such as ARM’s cores, have included realtime trace facilities. The easy availability of such real-time trace facilities allows new debugging paradigms to be developed. In this paper I describe some of these techniques for debugging embedded systems. I also show that real-time trace can have advantages even for debugging systems that do not have hard real-time requirements. Why real-time trace is needed In a traditional start/stop debugging environment the processor, or the application, has to be stopped to find out anything about the state the system. There are two major problems with this in real-time systems: 1. It may be impossible to safely stop the system. For example, in a disk drive, the head may keep moving until told to stop by the processor. If the processor is stopped, so the head is not told to stop in time, the disk will crash. This is an expensive debugging technique. 2. Even if it safe to stop the processor doing so will change the state of the system. In particular, in real-time systems many bugs may depend on the timing of the software relative to external events (e.g. clock interrupts). Stopping the processor will change the relative timing of events, hence causing the bugs to disappear or to metamorphose into different problems. The effect of this is that it is often very difficult to debug real-time embedded systems. This can add considerably to the time taken to complete the development of such systems, and as such adds considerably to the cost of developing such systems. How real-time trace works Real-time trace captures a trace of the instructions executed by a processor, running in real-time, and stores these instructions in a buffer for later analysis. In addition to the instructions executed the data used by those instructions may be captured. It is typically possible to select which instructions or data is captured. It is also normally possible to select a trigger condition, such as execution of a particular instruction, or writing a particular value to a particular location in memory. Much more complex trigger conditions are also sometimes available. When the trigger condition occurs the buffer stops capturing the trace data, either immediately or some time later, hence ensuring that the buffer retains a trace of the system’s behavior around the time the trigger condition occurred.

Debugging Techniques for Embedded Systems using Real-Time Software Trace

Page 2

Traditional real-time trace Traditionally, to obtain real-time trace, one has needed a logic analyzer. The logic analyzer is connected to the processors address and data busses, and traces all accesses to memory (both instruction fetches, and data accesses). This is then used to reconstruct the sequence of instructions executed by the processor.

Trace output

Logic Analyser

Processor

Address bus

Other devices

Data bus

Target System

Figure 1 Traditional real-time trace While this approach can work well it has a number of problems: Cost. Logic analyzers are typically quite expensive, and it is not normally practical to provide them to every software engineer. Access to address and data busses. In modern systems the processor will often be embedded in a chip with many other functions. The processor’s address and data busses may never leave the chip. In these circumstances it is not possible to connect a logic analyzer to the processor’s busses. Complex processors. In a modern processor, with caches, an instruction pre-fetch pipeline, and possibly out of order execution, the order in which instructions are fetched may not reflect the order in which they are executed. It is, in fact, very possible for a processor to execute a significant number of instructions (say in a loop) without accessing external memory at all. As processors become more complex it becomes progressively more difficult to reconstruct the instruction sequence from watching the bus. •

•

•


Page 3

Embedded real-time trace Many modern processors, including most ARM processors, now include an embedded trace macrocell (ETM). The ETM captures the sequence of instructions and data accesses and send a record of them to a buffer, which may be either on or off chip. ARM’s ETM includes features to allow triggers to be set, and to allow the trace to be filtered as it is captured. It also compresses the data so that the buffer memory is used efficiently, and the bandwidth required to the buffer is small.

Microcontroller ARM CPU Macrocell Control

Trace Debug Tools running on host

BREAKPT

Multi-ICE

5 wire JTAG

JTAG Port

Address

Execution Unit

ETM

Data

EmbeddedICE

TAP Trace Port

MultiTrace

Figure 2 ARM's Embedded Trace solution The advantages of this approach are: •

•

Cost. If the chip has an on chip trace buffer then trace can be used with no additional hardware beyond that used for normal debugging. Even if the chip does not have an on-chip buffer the cost of an external trace buffer is considerably less than the cost of a full logic analyzer. Access to the processor. Since the trace is captured within the chip the ETM can easily access the processor.


Page 4

•

Complex processors. The ETM always monitors the data and address busses between the processor and the cache. This means that caches are not a problem. ARM’s more complex processors contain private interfaces to the ETM that provide the ETM with all the information it needs to fully trace the processor.

While this does add something to the cost of the processors, in terms of silicon area, the incremental cost typically is small. For ARM processors the ETM is optional, but we have been finding that most of our partners are now including it in their designs. Other ways of debugging real-time systems. Simulation/modeling – This is a useful method, particularly in the earlier stages of developing the software. It allows full use of traditional start/stop debugging on a software model of the system. It can, however, be slow, and it is often difficult to create a full simulation of the system. Real-time debug monitor running on target. A real-time debug monitor is a piece of software running as part of the target code that allows a debugger to communicate with the system, and to debug its non time critical aspects, while allowing its time critical aspects (e.g. critical interrupts) to keep running. This works well if the problems are in the non time critical parts, but is useless for debugging time critical code. •

•

Using real-time trace to debug real-time systems The fundamental technique for using trace is not very different from using breakpoints for debugging. To debug a problem: Find a point in the program, or a condition, that you expect will give you some information about the bug. Set a trigger on that point, and allow the program to run (if not already running). When the trigger is reached the trace output is generated. Examine the trace output to see what it tells you about the bug. Repeat until the bug is understood. •

• •

•

There are, however, some major differences. Trace gives information over a period. Even simple instruction trace will tell you how the trigger point was reached. In effect, with real-time trace, you can look backwards as well as forwards. Because real-time trace is non-intrusive it can give accurate timing information. Using this timing information you can investigate timing bugs that would be difficult to investigate any other way. You have to chose in advance what information to collect. In a start/stop debugging system, once you reach a breakpoint you can look at the state of any part of the system. Trace systems have limited capacity, and since the system does not stop at the trigger point, you cannot get additional information about the system after collecting the trace. •

•

•


Page 5

The fact that real-time trace is not intrusive means that it is possible to debug parts of the system that are typically not debuggable using breakpoints. For example, one can usefully set trace triggers on interrupt routines, and hence, for example, discover the timing of interrupt routines, and how interrupt routines are interacting with one another. Figure 3 shows an example of trace output. The trigger point is defined to occur when a value is stored to x, and occurs on index 0. As we can see the trace allows the user to see how this point was reached, which is not something that is available using start/stop debugging. The trace buffer will contain hundreds of thousands or millions of instructions leading up to the trace point, giving extensive information about how the trigger point was reached. By filtering the trace to only capture selected instructions one can extend the time period over which trace is captured to seconds or longer. Using the filtering facilities of ARM’s ETMs one can, for example, capture only the code executed when handling a particular interrupt, or exclude all code in a particular source file.

Figure 3 Example of trace output - cycle 0 is the trigger cycle


Page 6

Advantages of real-time trace when debugging non-real-time systems As we saw earlier, one of the advantages of real-time trace is that it can give information about how a particular point in the code, or state, was reached. This makes it particularly valuable for solving bugs that are either intermittent, or where it is not obvious how the program can reach its final, faulty, state. This is true even when the system is not a real-time system, or the bug is not related to its real-time aspects. I, when I am debugging, often find myself asking “How on earth did the program reach that line?”. With traditional start/stop debugging often all one can do is step through the code until the program reaches the line in question. With real-time trace I can set a trigger point on that line of code, and then work backwards through the trace to work out how the program got there. An even worse, but similar, example is data corruption. For example, if a global variable, or an element of a data structure is corrupted it is often very difficult to work out how it is being overwritten. Even once one has found the code that is writing to the location all this typically tells one is that a pointer is incorrect. Using trace one can set a trigger on writes to the corrupted location, and then work backwards to find out why that location is being written to. Another example where trace is valuable is in debugging problems that cause processor exceptions (e.g. data aborts). Once again the ability to work backwards from the trigger is the critical factor here. Using trace one can trigger on the execution of exception vector instructions, and then work backwards through the trace to determine how the exception was caused.


Page 7

Using real-time trace for performance analysis A further use of real-time trace is for performance analysis. Traditional ways of analyzing the performance of software include adding in special function calls to record the time at particular points in the code (instrumenting the code), or using a timer interrupt to sample its state at particular points. Both these techniques can give good approximate measures, but because they require running additional software on the target they, in themselves, change the behavior, and hence the performance, of the target system. Using real-time trace one can get a complete trace of the instructions executed, together with cycle counts for each instruction. One can then analyze this information to give a detailed picture of the behavior of the system over the period in which the trace was collected. Since real-time trace does not in any way modify the behavior of the system, this information is accurate.

3%

1% 2% 1%1%

11%

Func_1

3%

Proc_8

4%

Proc_7 _memcpy_small

5%

11%

_strcmp_loop Func_3 _strcmp_return

5%

_memcpy_aligned_loop Proc_2 Proc_4

5%

11%

__rt_sdiv Proc_3 Func_2

6%

Proc_1 Proc_5 9% 8%

strcmp Proc_6

8%

8%

__rt_memcpy_w

Figure 4 Example performance analysis data from real-time trace


Page 8

Conclusion The use of embedded real-time trace, as provided by ARM’s ETMs, provides new ways of debugging real-time and non-real-time embedded systems. In many cases the availability of real-time trace allows the easy analysis of bugs that would otherwise be difficult or impossible to analyze. As such using processors that include embedded realtime trace can significantly reduce the time taken to debug the resulting systems, and hence produce significant cost savings. In addition real-time trace is one of the few techniques that gives truly accurate performance profiling information. This allows performance bottlenecks to be found and resolved, and hence eases the development of performance critical software. ARM, ARM Powered, StrongARM, Thumb, Multi-ICE, Integrator, PrimeCell and ARM7TDMI are registered trademarks of ARM Limited. ARM7TDMI-S, ARM7EJ, ARM720T, ARM740T, ARM9TDMI, ARM920T, ARM922T, ARM940T, ARM9E, ARM926EJ-S, ARM946E-S, ARM966E-S, ARM1020E, ARM1022E, EmbeddedICE, EmbeddedICE-RT, AMBA, MultiTrace, ModelGen, ARM Developer Suite, RealView, ETM, ETM7, ETM9, ETM10, Embedded Trace Macrocell, Jazelle, PrimeXsys, MOVE and JTEK are trademarks of ARM Limited. CodeWarrior is a registered trademark of Metrowerks Corporation. All other brand names or product names are the property of their respective holders. "ARM" is used to represent ARM holdings plc (LSE: ARM and NASDAQ: ARMHY); its operating company ARM Limited and the regional subsidiaries ARM, INC.; ARM KK; ARM Korea Ltd. Neither the whole nor any part of the information contained in, or the product described in, this document may be adapted or reproduced in any material form except with the prior written permission of the copyright holder. The product described in this document is subject to continuous developments and improvements. All particulars of the product and its use contained in this document are given by ARM in good faith. All warranties implied or expressed, including but not limited to implied warranties of satisfactory quality or fitness for purpose are excluded. This document is intended only to provide information to the reader about the product. To the extent permitted by local laws ARM shall not be liable for any loss or damage arising from the use of any information in this document or any error or omission in such information.


Page 9

Debugging Techniques For Embedded Systems

Recommend Documents