Design Article
Using trace to solve the multicore system debug problem
Aaron Spear, VMware
4/7/2011 1:52 AM EDT
Modern multicore designs often aggregate wildly different hardware and software technologies. Traditional debuggers, which show a snapshot of a portion of the system, do little to uncover issues that arise due to complex interaction of components.
This paper is from a class at the Multicore Expo 2011. Click here for more information about the conference. |
Engineers routinely cobble together proprietary tracing facilities in order to have some chance of catching hard to find defects. What is needed to debug these diverse systems is an approach that can analyze trace data coming from many different collection technologies.
In this paper we outline current multicore development trends, explore the deficiencies in traditional software development tooling when applied to multicore systems. We will also introduce the “Common Trace Format” (CTF), a coming standard for tracing multicore systems over time.
The nature of the multicore debug problem
Debugging and optimizing modern multicore designs is growing in difficulty in proportion to the exponential growth of technology. In 2011, the number of “smart phones” will overtake “feature phones” in the U.S. That means that over half the population has scaled their expectations for the new “normal” embedded system to include:
- Hip touch screen UI with 3D effects
- Extremely responsive
- Reliable/fast Internet connectivity anywhere
- Long battery lifetime
- Available now!
The net result is that the embedded systems designer is handed a set of requirements that are extraordinarily difficult to reconcile. Requirements on power consumption, external connectivity (Cellular, Wi-Fi, Bluetooth), human interface, and time to market have resulted in some general patterns in how devices are designed:
- Low-power requirements force using more cores at often variable clock rates in order to meet the same performance levels.
- Performance critical functionality is pushed into dedicated hardware.
- Open source operating systems, drivers, and software stacks are used (e.g. Linux, Android).
- The control plane and data plane are split into separate processors (with different OSes on each).
Embedded systems are doing more, and doing it in parallel. Debugging an embedded system with a single core is difficult enough, but debugging multiple streams of software activity that are interacting with each other is really hard.
The problem with software debugging
A traditional software debugger, GDB, allows developers to look at snapshots of one program at a time. The developer sets a breakpoint, runs the program, and then sequentially steps through the code from the breakpoint, verifying that state changes as expected. The developer sees the stack of the functions that were called leading up to the next line of source code to be executed. When debugging multiple threads of execution concurrently (multi or single core), most software debuggers add more contexts to the same display. Figure 1 below, for instance, shows Eclipse CDT with GDB debugging three Linux processes, with multiple threads.
Click on image to enlarge.
Figure 1: Multi-process debugging with CDT and GDB 7.2 [1]
Debugging a set of cores that are interacting with each other requires that a debugger be attached to all of them. Depending on the cores and tools being used, it is possible to have a debugger capable of debugging multiple cores concurrently. In some cases hardware may allow synchronous debugging, that is all cores running and stopping together.
This use case tends to be the exception and not the rule however. In many cases, decoupled tools must be used to debug the cores individually. This sounds difficult, but it is in fact much worse than this due to the layering of operating systems and application stacks.
Android, for example, consists of three distinctly different software domains in which to debug: at the bottom is kernel space (which includes the Linux kernel and drivers), in the middle is Linux user process space and a unique JVM, and at the top Java applications and application frameworks.
There are no open source tools that allow you to seamlessly debug from Java down into a C/C++ user process, and then down into Linux kernel code. Debugging everything in the software stack requires three different debuggers for the core running Android, and then additional debuggers for other cores.
The extreme difficulty in simply setting up to debug is further compounded by problems that are impossible for a software debugger to solve.
Some systems have real-time deadlines and they simply cannot be stopped at all. It is also increasingly common for significant portions of functionality to be pushed into dedicated “hardware accelerators”, on chip resources that offload functionality such as networking or graphics from the main processor. These blocks tend to be black boxes inaccessible to software debuggers.
Software debuggers are invaluable for debugging algorithms, but for debugging a system of concurrently running hardware and software components, what is needed is a tool that can trace the state and interaction of software and hardware over time.
Next: Page 2
|
|
|
|
|
