Over the last 30 years, approximately, the automotive sector’s use of processors has progressed from 8-bit devices that were very often programmed without benefit of compilation techniques at all (assembler coding) to 32-bit machines with exacting real-time performance constraints.
Also over that timescale, a change in emphasis has taken place in the competitive landscape for embedded tools in the automotive sector. In an earlier market phase, differentiating factors were very much focussed on performance metrics; on code size, on memory usage and on execution speed – the outcome of optimisations carried out during the code compilation process. To some extent, these aspects have been joined by system power usage: this may be less of a key issue in automotive designs compared to other market sectors, but is nevertheless important as processor-based systems proliferate in vehicles and finite electrical power resources are stretched.
But surpassing other aspects, today there is a much greater level of attention paid to correctness of output code, and to safety-related issues.
Over the past three decades, a number of trends have played out in parallel. In the automotive industry, there was a marked increase in quality levels (in general) in the 1980s; and in the 1990s, many previously-mechanical systems were replaced, or assisted, by electronic variants. Electronic fuel injection and anti-lock braking are two notable examples. Microcontroller-based electronic control units (ECUs) became the norm to control subsystems in a car, and the number of ECUs per vehicle multiplied. After the year 2000 complexity escalated rapidly; multi-core processors were required to fulfil the computational and safety requirements of powertrain applications, and there was a corresponding (and continuing) increase in complexity of the algorithms applied in chassis control and ADAS systems.
At the same time, in the wider embedded-code-creation context, awareness of the need for a structured and disciplined approach to software engineering was recognised, as the vulnerability of all types of safety-critical systems to software errors became apparent.
Over that same timescale, the automotive industry has had an equally steep curve to navigate, in terms of increasing product quality. On top of internally generated quality programmes, and competitive pressures, have come external quality standards, to which compliance must be demonstrated. A prime example is ISO 26262.
Quality in code compilation means, at its simplest, absence of bugs. Product quality is measured via exhaustive testing and comparing the results against the expected results and also the results obtained by toolsets of other suppliers. Compilers should generate the correct output in response to (any) legitimate input. Developing a bug-free product requires having mature processes in place, otherwise it is not possible to develop and maintain a product that is representative of the state of the art. Suppliers to the automotive industry typically implement their processes in conformance with ASPICE compliance level 2.
From 1985 to 2010 all compiler vendors applied the same test approach. In the early days, the International Standardisation Organisation (ISO) standardised the C language, and also supplied a test suite to verify the correctness of a compiler implementation. This was a large suite of small test programs that cover all features defined in the ISO-C standard. Part of the specification of the C language is also a list of (nearly 200) “undefined” behaviours, and a smaller but still substantial set of “unspecified” behaviours. The integrated MISRA C and CERT C code checkers can be used to verify possible occurrences of most of those behaviours.
The entire C-compiler community – open-source and commercial alike – had a chastening experience with the release, in 2011 of the open-source tool Csmith (University of Utah) together with the results of many compiler tests the University group ran with the tool. Csmith is a “randomised test-case generator….using differential testing”. It was able to cover a far wider range of “corner cases” of inputs to compilation, and it used different compilers to check each other’s outputs. In short, it inspired a step-change in testing methodology and (in some cases) product quality, almost overnight.
One of the surprising findings that emerged from that episode was that certain compilers showed erroneous behaviour irrespective of the level of optimisation effort requested by the programmer. Nevertheless, verifying correct behaviour through the steps of optimisation, especially intensive optimisation, is a critical aspect of ensuring bug-free compiler operation. Optimisations have the power to alter how the code executes a function; it is essential to ensure that what the code does is unchanged, and is so in response to all possible inputs.
A closer look at ISO 26262
The Standard ISO 26262 “Road vehicles – Functional safety” aims to ensure the safety of electrical and electronic systems in road vehicles. In common with other ISO safety-related standards, it does not aspire to the creation of systems that cannot or will not fail. Rather, it explicitly recognises that failures will occur, and seeks to ensure that the response of the overall system to a failure will in all circumstances result in a safe outcome. It is therefore concerned with ensuring that the design process employed is such as to result in a safe system.
A section within ISO 26262 categorises the likely consequences of a sub-system failure in terms of ASIL (Automotive Safety Integrity Level) from A (least serious) through to D (immediate threat to life). The design processes that build the code for each of the systems listed in below table must be structured to demonstrate compliance with the appropriate ASIL level(s).
The standard describes the qualification methods for each sub-part: highly recommended (++) and recommended (+)
The key word in the preceding paragraphs is “process”. The standard governs the process of design of the automotive product. As an input to that process, a (for example) compiler must be documented to show that it has, itself, been designed following a process that takes account of possible sources of failure, and mitigates – or, more appropriately in the case of a software tool, detects – any such failure. It must also be supported in such a way that the automotive design team can in turn show that their process follows the guidelines of ISO26262; the manufacturer/design house must establish the required level of confidence in the software tools it proposes to use.
Tool Validation to ISO 26262-8 11.4.9
ISO 26262 provides hardly any guidance about tool validation. Section 11.4.9.2 contains all guidance and is limited to three sentences. In an attempt to unify the different tool validation approaches of its customers Altium/TASKING uses, as a framework, the guidance provided by the standard “DO-330 – Software Tool Qualification Considerations”, which is intended to provide tool qualification guidance to the avionics domain, but can also be applied in other domains such as automotive (DO-330 1.2). Guidance for commercial-off-the-shelf (COTS) software tool qualification is provided (DO-330 11.3) where the qualification activities are partly performed by the tool developer and partly by the tool user.
The basis for compiler validation to match ISO 26262-8 11.4.9 addresses the top of the V-model only (Figure below). That is, the requirements phase, and the verification/validation steps. The validation measures must demonstrate that the software tool complies with its specified requirements. The tool requirements specifications cover all features listed in the user manuals, the requirements expressed in tool related standards such as in, for example, “ISO/IEC 9899 – Programming languages – C”, and requirements related to the compiler design and implementation. The verification and validation efforts cover all requirements which fulfil the ISO 26262-8 11.4.9.2.a guidance that the requirements coverage metric shall be 100%. Although ISO 26262 does not provide guidance regarding code coverage, Altium strives to reach near 100% decision coverage. Additionally, the latest academic contributions that advanced the state of the art in compiler testing are applied to uncover errors in highly complex code optimisation algorithms by maximising the path coverage.
Validation of the Software Tool