System level software centric power debugging using virtual prototypes

Just the LCD can consume as much power as the CPU in a mobile phone. A software power inefficiency or malfunction can quickly cause a drop of 5x in terms of standby time. Evidence for this is available in the user forums of all the major phone vendors where people have updated their phone with new firmware or applications.

Although power efficiency is a major quality and sign-off criteria for hardware platforms, it is still a poor cousin in the quality checklist for software. There are multiple reasons for this.

The software community still considers power as something that is taken care of by either hardware or the software layer they are building the functions on. Application developers trust that the application framework will make sure that while using services the underlying resources are used in a battery-friendly manner.

But the reality is that a simple RSS feed application that wakes up the phone every 10 minutes for just eight seconds to do some updates via the internet can cut the standby time of a phone in half. Software platforms for mobile applications must be engineered in such a way that smart heuristics make sure only those services required in the active usage scenario are provided. Components must be turned off when not in use.

When porting such platforms, the power control capabilities of the OS and their drivers must be made available to the application framework. During OS development and porting, software engineers have to make sure that all power saving features of the underlying hardware are leveraged and supported by drivers.

With dynamic frequency and voltage scaling of the CPU this can reach deep into the internals of the OS kernel, since it is impacting scheduling through work-load prediction.

Finally, in order to guarantee an efficient use of the battery, all highly distributed software entities for power saving and management have to be vertically integrated and need to cooperate.

Obviously, a minor malfunction or inefficiency in just one software module can trigger a cascade that severely reduces battery mileage. It is also obvious that debugging those cause-effect chains can become a nightmare.

Debugging is extremely problematic since the cause-effect search space is not limited to one software layer. While software looks fine from a functionality and performance standpoint, it could well be that the battery drains four times faster.

Imagine a scenario where the phone is locked and it is located in the pocket of a user who is hiking. After the hike the battery is empty because the accelerometer was not shutdown and constantly woke the phone up when it would have been idle otherwise.

This example shows another challenge for power profiling and debugging. In contrast to software for a desktop PC, the usage scenario plays an important role, as it defines how the phone interacts with the environment. How can you debug your phone however, while it is locked in your pocket? How can you make sure that scenarios are deterministic to compare different implementation options?

Debugging power defects moreover comes with another major issue. In a low power mode, such as suspend, the embedded debug service is likely to be suspended as well. In addition to that, any debug interaction with the device is intrusive and severely tampers the power figures in itself.

A debug daemon would not let the phone go to suspend, require USB/UART/Ethernet drivers, fire additional interrupts, increase the CPU load, change workload prediction and the frequency and voltage control. Furthermore, expensive lab equipment is required to perform a sufficiently fine granular power profiling to determine which of the components is the most critical.

On top of all the technical challenges, software centric power profiling and debugging is an educational challenge. Software engineers may not be able to interpret what it means if 250mW are saved for a specific component configuration. Here, it becomes much more important to provide qualitative results, such as the stand-by time has dropped by 50 percent because the component is used 90 percent of the time in the profiling window.

Virtual Prototype based software centric power analysis and debug

Virtual Prototype (VP) based solutions enable software centric power analysis and debug to address the previously detailed challenges of:

§         Deterministic repetition of real-world scenarios on the developer’s desktop

§         Non-intrusive debugging during suspend and low-power modes

§         Understanding system level hardware and software behavior and interaction

§         Power measurement, debugging and root cause analysis

A VP is a software emulation of an entire system, such as a Smartphone ash shown in Figure 1 below. VPs can be made available early in the development stage of an electronic system, even before hardware is available. A VP is able to run the exact same binary software that the hardware system would execute.

However, execution is performed on simulation models of all the system components, including the application processor, the memories, DMAs, timers interrupt controllers etc. Those models can be instrumented with information that characterize power.

The execution is controlled via scenario scripts that drive the I/O of the VP, and include generating user-input via the touchscreen controller, setting GPS coordinates through the UART or initiating a phone call. Software optimization is made possible by collecting power analysis data along with running other hardware and software traces during simulation, in order to enable root-cause analysis and debugging.

Figure 1 -Virtual Prototype Framework

Software debugging

VPs easily integrate into existing software development flows as state of the art software debugging tools and IDEs—GNU gdb, ARM RealviewTM, and Lauterbach TRACE32TM for example can be connected to the VPs, just like with real hardware. VPs however, do not require an embedded software service executing on the prototype to enable the link to a software debugger. As highlighted earlier, this is a fundamental requirement to debug low power modes, like suspend or idle. Here, the VP framework itself provides debug interfaces for software debug tools.

Using the interface, the debugger is able to read and modify the system state as well as control its operation. In this case debugging is non-intrusive, since the whole system simulation is suspended during debug operations and not only the CPU is under debug.

Because each debug operation happens in zero-time from the software’s point of view, the software is unable to recognize inspection or control by a debugger. When debugging, the user can be sure that he is not accidentally impacting the power profile of the system. This debugging mode is also referred to as stop mode debugging.

Of course VPs also support run-mode debugging, where only a process under debug is stopped and linked to a software debugger and a debug daemon is established via virtualized IO.

Debug and analysis scenarios

Virtual I/O is also essential for the creation of real-world usage scenarios, such as navigating with a “maps” application when walking and using a 3G internet connection. Virtual I/O refers to the ability of a VP interface peripheral, such as USB, UART, Touchscreen IF, to connect either to the real world (e.g. USB), or further (functional) simulation models of other parts of the system (e.g. modem, GPS, accelerometer).

Using VPs it is possible to record and replay those scenarios in a deterministic fashion for debugging, optimization and regression testing. Scenario recording is done via the ability to watch each and every aspect of the system, including even signals and hardware registers next to memory.

Breakpoints and watchpoints can trigger actions (i.e. callback) in a VP, such as reading the interface signal value and writing those, together with a timestamp, into a file on the host for replay. Those actions are expressed using a scripting language, like TCL (Tool Command Language).

The scripting interface provides the ability to peek and poke data from/to a VP, as well as control the execution. It is important to understand that the script is not part of the embedded software.

The script is running on the host in the VP tool framework. Thus, in the replay script, data is pushed back into the VP using simulation time triggered actions. The scenario replay allows the software developer to easily reproduce complex defects during debugging and then compare analysis data for optimization.

Power characterization & analysis

Power analysis data is recorded during simulation by observing the power states of the system components. This observation is again performed using watchpoints triggering actions as shown in Figure 2 below.

Each action re-computes the power level for a component based on querying information, such as internal state registers or signal/pin values. On top of the basic scripting commands an easy-to-use power framework is provided that allows the user to characterize power for components without being familiar with all the details of the scripting command set.

The power data is used to compute energy along with the time that the system is running. The software developer is then able to determine energy for a time interval and component under investigation. This allows finding abnormalities, such as the LCD being turned to low instead of off during suspend or the CPU being loaded more than 70 percent when the phone is idle.

Of course in order to identify those abnormalities the characterization must be accurate. Analysis is conducted on a trend basis and in a qualitative fashion. For this purpose it is sufficient to have ballpark numbers provided by data sheets. A critical piece in the power debug flow however, is the ability to perform a root cause analysis.

Figure 2 - Stimuli, characterization and analysis

Power debugging

Root cause analysis requires more than merely hooking up a software debugger and investigating the execution of software along with the power levels. First, it is a challenge to link the software execution with the power profile.

Power management is a highly distributed function that is performed in many different processes in all layers of an embedded software stack, such as Android. Thus, the tracing of software execution and power is important in understanding the history of the HW/SW system.

This tracing must be performed at a level that is meaningful to the software developer. An instruction level trace is far too detailed, but even tracing on the function level is meaningless in an OS where processes switches happen thousands of times per second (e.g. 1Khz timer interrupt).

A VP helps to abstract the tracing to a level where it becomes useful for the developer. Using an OS-awareness plug in, the VP can show OS processes over time along with the power analysis data as shown in Figure 3 below.

Figure 3 – Power analysis

At the process level the developer is able to understand the relation between software entities in the stack. As shown in Figure 4 below, he or she can determine for example that the battery status animation update every 500msec is triggering a cascade of actions, such as triggering interrupts in the kernel and event queues in the user space. It is important to determine for instance why the Linux “tickles-kernel” power saving feature is not performing well with Android in idle mode.

For a process that has been identified for a power level cause, the user can perform a top down debug analysis looking at functions and even instructions in each process. The developer is able to pinpoint to the exact location in the application or the driver where the state change happens. The OS-awareness of a VP is again realized using breakpoints in the OS that trigger actions in the scripting framework to peek the process ID/name via the virtual memory read interfaces. Thus, OS-awareness in a VP does not require any instrumentation or change of the OS and is solely dependent on function symbol information.

Figure 4 – Tickless Kernel Process Trace & Software Logging

Software logging and printf

Software logs, as well as user defined debug messages, using the well known printf method can be extremely helpful to find the defect cause for a fault symptom. Those messages are typically at a level of abstraction where they nicely describe the function of the software at a detail level targeted for the user/developer.

Furthermore, Android provides logging mechanism for the application layer, the application framework, the run-time environment and the hardware abstraction layer. Unfortunately, the log generated via the Android logger is separated from the logging performed by the Linux kernel, or the logging performed by the user through ad-hoc printfs.

This becomes especially problematic when log messages from the Android’s extended Linux power management (i.e. wake-locks) need to be correlated with the kernel logs or other traces, like the process trace or power trace. VPs allow the unifying of all traces without any change in the embedded software.

Thus, all trace information can be accessed in one environment that is correlated via time stamps. The mechanism behind it is once again based on the VP scripting capabilities. As shown in Figure 5 below, the VP is setting breakpoints at each log function and triggering an action that is reading out the log string.

Figure 5 – Catching Linux Kernel Log Messages

The VP then examines the parameter of the log function, such as the virtual address of the string in memory. The parameter can be inspected for example by reading the CPU register values that are typically used for parameter passing.

This way, arbitrary complex data structures can be de-referenced (e.g. Java object names, etc.) as shown in Figure 4. This is especially useful for software stacks that run applications in Java on an interpreter, since the link between the Java world and the native world can be established to ease debugging.

Summary

In this article we have outlined how Virtual Prototypes (VPs) provide all the necessary elements for a debug solution that can spot and remove power related defects from software. Creating and executing deterministic scenarios for debug and regression has been described. It has also been shown how debugging can be performed in a non-intrusive fashion with global visibility, even during low-power modes, such as idle and suspend.

We have reviewed how system power characterization can be performed and how power can be analyzed over time and component. We have also shown how debugging can be performed at the right level of abstraction using OS-awareness unifying software logging with system traces.

Power debugging is just one formidable application for VPs. A key concept to understand is that debugging capabilities have been added at the VP user level, purely based on scripting capabilities.

VP scripting allows for much more, including user defined debug tools (e.g. memory corruption checker) or system level hardware/software assertions. Such solutions are being actively discussed in the context of the open Virtual Platform User’s group (http://groups.google.com/group/virtual-platform-users).

(Achim Nohl is a Solution Specialist at Synopsys, responsible for Processor Designer and Virtual Platforms. He drives the adoption of Virtual Prototypes for Software Development and the design shift from hardware to software. Achim holds a MSEE in Electrical Engineering from the Institute for Integrated Signal Processing Systems in Aachen. He can be reached at achim@synopsys.com.)