Using Advanced Debug Tools to Ensure Code Quality in Mission Critical Systems



by Barry Lock of Lauterback UK

Embedded systems are now widely used in mission critical systems. The obvious applications might be avionics and medical, and the less obvious might be a diesel engine management system. However, it has been the automotive sector that has driven a great deal of debug tool development in recent years. Complex multicore engine management systems now enable smooth and highly efficient diesel engines. However, a major challenge has been to ensure the integrity of the software, as no major brand can risk a major recall, or reliability issues in the field, or indeed, a catastrophic failure.

This article will explore a few of the latest debug technologies available to engineers that can help ensure Code Quality and Performance in such embedded systems.

Some of the Challenges
To prove the code is meeting the target specification is not easy. Even something that an outsider would think is a simple task, such as proving what code has actually been run (code coverage), is rarely undertaken by development teams. The tracking of obscure or random bugs can be so difficult that when a code alteration “fixes” the problem, the developer moves on without any concrete proof that the original problem was found and fixed.

Another such challenge might be that debugging code properly is often harder than it needs to be because very few teams think about what debug capability will be needed when they define the original system design parameters. As an example; people think about parameters such as measuring best and worst case response times but the operation of the cache and obtaining details of code victims and misses is also invaluable if the system is expected to run at its optimum level. There are many similar interlinked aspects to analysing the operation of the final embedded system and proving it is fit for purpose.

Debug Tools – An Increasingly Essential Investment

“Give me six hours to chop down a tree and I will spend the first four sharpening the axe”

This quote from Abraham Lincoln may be an excellent analogy to debug tools. The more capable the debug environment, the better the development experience for all concerned, resulting in better code, faster development and a reliable in-field performance.

At the start of every project, engineers get to choose their hardware platform, their software solution, compiler and their debug solution. Many of the worlds’ leading developers invest in the very best debug tools they can find. This article will explore some of the functionality that they demand in order to be able to protect their market position, their business and their brand.

Starting Point - Choosing the Right Processor
More important than the choice of tools, compiler or operating system is the choice of processor. In order to get non-instrumented coverage information from the system, a processor with an off-chip trace port is essential. For systems that require safety certification, a good quality trace port is required. The bandwidth at the trace port will prove a very important consideration.

For multi-core devices, the trace port must be capable of providing reliable, error free trace streams from all cores. This will mean that the trace port should be running at a good percentage of CPU clock and be wide enough to get the data off-chip fast enough to prevent overflows.

Required Hardware
Off-chip trace hardware will be required to interface the processor to the development environment. It should be capable of collecting data with the core running at its final clock rate. Many solutions are available, ranging in capability and price. Some include important features such as a logic analyzer and high-speed serial trace functionality.  For older processors without a JTAG port, an In-Circuit-Emulator (ICE) can be used.

Many reference design boards come with low cost debug solutions. These have the advantage of being easy to set up and use, but can prove to have a very limited capability as code development progresses.  The more powerful solutions may be more costly and maybe even require some training, but the longer-term benefits will quickly prove a good return on investment.

Code Quality and Standards
The big changes that debug tools and code analysis developers are seeing, is that it is no longer enough to simply test a system. An important trend has been an increased requirement for engineers to document and prove code behaviour and performance.  In some instances companies are requesting engineers to be Certified and Qualified in Software Writing or mandate some form of specified code quality. With the right choice of debug tools, detailed information about code behaviour can be generated to provide the necessary proof, showing such aspects as the route taken by the code and time taken. In this respect, program flow trace can go a long way to helping companies prove code behaviour and in achieving Safety specifications such as ISO26262.

Long Term Trace Offers Recording and Direct Analysis
Long Term Trace is the name given to streaming trace to a hard drive and thus overcome the limitations on internal buffer size in most debug tools. This provides the ability to collect massive amounts of code performance information from a running embedded system in order for the developer to detect and analyse the most unpredictable and transient of bugs. Demand for long-term trace debugging is being driven by market sectors such as automotive, medical and aerospace, where microprocessor based systems are becoming increasingly complex and in need of more rigorous testing to comply with safety and performance criteria.

One example is to imagine an engine management system. This longer code coverage enables engineers to analyse the software from a cold start, then up to temperature, through acceleration and deceleration and then to shut down. Using Long Term trace technologies, engineers are now able to capture code over much longer periods that can extend to hours or even days. This has been a very important break-through for those developing complex or safety critical code.

High Speed Serial Trace
'High Speed Serial' trace technology is a serial transmission technology that has a transmission rate of over 6Gbits on each of up to 4 lines from the target core. To put it in perspective this data rate is such that it could transmit the entire contents of a DVD in 3 seconds, making it ideal for the collecting of data for the debugging and development of very high speed multicore embedded systems and systems requiring mission critical analysis.

Traditionally, the trace interface which the target processors used to deliver the detailed information on the operation of their inner processes had been a parallel port. Over recent years this design of port has struggled to keep up with the growing flood of information as processors have become more complex, faster and with multiple cores. The step to high speed serial solved these problems and had the side effect of reducing the pin count, which is always good news. For many developers of embedded systems it would have been unthinkable to undertake a development without their valued trace information, so a lot of effort by silicon designers and tool vendors has been made to increase the data throughput of the trace interface.

In recent times, ARM has implemented High Speed Serial Trace with its High Speed Serial Trace Port (HSSTP). This has been followed by AMCC with the Titan, Freescale with the QorIQ processors P4040 and P4080 and Marvell with the SETM3.

Engineers can now source a hardware interface for serial trace; a universal pre-processor has been developed on the basis of the Aurora protocol. Only the firmware and software have to be changed to record any of the alternative protocols. This means that existing systems will need minimal reconfiguration for further variants of serial trace protocols.

An important consideration of this technology is that the large volume of trace data generated obviously requires a correspondingly large trace memory and careful consideration should be given to the tool choice with regards to memory size and the ability to stream to the host.

Multi-Core Debugging
Multi-core development is an area of embedded systems development that is becoming more prevalent.  It started with mobile phone development back in 2003/4, then moved into telecoms (base stations, routers) about five years ago, and today, it is finding its place in the automotive sector as future emissions and fuel usage specifications demand far greater analysis and control.

With multi-core, the data is extracted through a shared trace port. There may well be four or more cores with processes running in parallel and some interacting across core boundaries.  The fact is, two cores are not simply ‘twice’ as complicated to debug as a single core. They are several times more complicated, because of these interactions.

Getting to the point, budget debug technology is not up to the job of multi-core development. For multi-core you will need a powerful trace tool. One with a lot of memory and one that can process data fast. In multi-core you will have a whole new level of problems to avoid, such as race conditions, uncontrolled interactions and out of sequence code behaviour.  Your toolset will need to be able to collect a large amount of accurate data to give you the detail and visibility into the code you will need to be able to understand the behaviour and interactions of the processes running on the cores.

Code Optimisation
In mission critical systems where a fast and predictable performance is essential, Cache Analysis has proven to be an important development.
Cache memory is on-chip memory that can be accessed very quickly by the CPU. Access to cache memory can be in the order of 10-100 times faster than access to off-chip memory. This fast memory acts as a temporary storage for code or data variables that are accessed on a repeated basis, thereby enabling better system performance. However, having cache memory does not necessarily equate to better performance from a system. In order to get the best performance from cache-based architecture, it is important to understand how a cache-based architecture works and how to get the best from particular cache architecture.  Trace technology enables analysis tools to confirm the effectiveness of cache usage and can have an important impact on the overall performance of the software.

Also, these days, safety critical equipment may be a wireless portable device, where battery life and predictable power usage may be an important factor. With this in mind, Power Management is another area that is being influenced by the use of Trace tools.

An Energy Profiler can provide a test set-up that measures, records and analyses the program and data flow of the control software as well as current and voltage gradients. Statistical analyses are run automatically after each program stop. They provide information about minimum, maximum and mean values of the energy consumption of the executed functions. Similarly, the absolute and percentage share of the total energy consumption is calculated for each function. This makes it easy to locate the program parts that use the most energy and enable modification, or simply enable the reliable prediction of battery life and system performance.

Hopefully, this article has provided an insight into how engineers are increasingly relying on Trace technology to not only enable the location of bugs, but to also create better code and provide clearer visibility and understanding of code behaviour. It is strongly recommended that such tools are explored and evaluated at the earliest stage of a new project.

Written by Barry Lock, UK Manager of Lauterbach