logo

 

 

slogan

On Target Testing:Has the time come for in-project and on-target C compiler validation?

 

By the wicked Witch of the West
January 2019

 


If your bedtime reading includes ISO/IEC 61508 [functional Safety] Part 3 [software requirements], you will obviously have seen in the annexes A and C reference to a “certificated compiler” as highly recommended for higher SIL level projects. That is becoming most of them these days so certified compilers are being required more often.
What this means is that the compiler has been tested using a recognised conformance test suite, and preferably by an independent tester, to demonstrate conformance with the C language standard. Except for large and highly critical projects, neither end customers nor safety regulators have demanded that such validation be carried out, resting on the alternative provision that a compiler is trusted from “experience in use”. Now, however, changes may be afoot.

 

 

The Author has recently worked on a project to develop a safety controller for equipment to control electric motors. The new controller is built around a safety-rated dual-core lockstep microcontroller (SRDCLM) and provides dual-channelling in software. This immediately hits the issue of common-cause compiler failures, since both cores run the same code. A careful (and commercially highly confidential) software design will offer pretty good mitigation of this problem by use of formal methods tools. Nevertheless the safety case still relies heavily on compiler integrity.

 

In preliminary discussions, the German safety authority asked the project to perform an on-target validation on the C compiler as a form of mitigation for such failures. It is easy enough to understand why this was required. In industrial automation applications, most safety controllers are physically dual-channelled PLCs. Application of SRDCLMs together with formal methods is relatively new in this field and the authority is rightly being cautious.

 

The proposed validation will use the Perennial C validation suite for freestanding C implementations together with the Csmith stress test generation tool. The tester will be an engineer who has for 24 years held registration as a UKAS assessor for compiler testing. Test driver software will be custom- developed to control the test sessions and automate the analysis of results and isolation of failed tests for controlled re-run.

 

The new controller will use only C libraries that are present in a freestanding implementation. Thus the Perennial CVSA Freestanding version with added Csmith stress testing make a good technical fit for the task. As they are relatively straightforward to use, one may well ask why a UKAS assessor will be conducting the tests. The main reason is the need for custom test driver software. Writing a test driver is not difficult in itself. The devil lies in knowing and providing the degree of technical control sufficient to assure the degree of technical integrity that would be required of a UKAS-accredited certification laboratory. Naturally UKAS cannot itself certify such testing since the test system will have been in part developed by one of its own assessors. Moreover, since UKAS certification is not needed, it is more useful to deploy the assessor’s -expertise in test system design.

 

This is the first time in the author’s experience that a safety authority has asked for project-specific on-target validation in what is by most standards quite a small development. The technical rationale is interesting. The compiler vendor is the company that manufactures the SRDCLM. Early consultations with their compiler test staff indicated an exceptionally thorough test regime, running to millions of tests. In fact it is the most rigorous compiler-developer test program of which the author is aware - and she is not lightly impressed by these things. The snag is that all of the developer’s tests are run on a simulator provided by the core designer and not on target hardware itself. (As one would expect, since it would take far too long to run such a large number of tests on-target.) This, combined with the newness of SRDCLMs is probably what motivated the safety authority to require on-target validation.

 

On-target running of the Perennial Suite alone will take the best part of a working week, the total test time being dominated by upload/download speeds to the target on a development kit board. The test driver will be designed for unattended operation and will contain robust self-checking routines as well as means for recovery to a verified configuration after failed tests. The test driver system will be implemented in Tcl/Tk, this option being chosen for its simplicity in comparison with other scripting languages, and its aptness for implementing the driver as an action system permitting straightforward formal modelling..

 

Now comes the big question: Does this mark a point at which on-target compiler validation will start to be required for embedded software at SIL3 and SIL4? Will the increasing use of SRDCLMs be the trigger that makes it the norm rather than something done on big defence and infrastructure projects. Predictions are as ever risky. The assessor who will be doing the tests described above gained UKAS registration in late 1991. At that time it was expected that requirements for compiler validation would grow rapidly. They didn’t. While having conducted many compiler validations for clients, the assessor concerned has done not a single one for a client that was a UKAS-certified testing laboratory. There’s market forecasting for you.

 

It seems reasonable to assume that requirements for compiler validation are most likely to arise to cope with so-called “disruptive” new technologies. Safety controllers based on SRDCLM chips certainly have the potential to replace traditional dual-slice safety PLCs. The cost advantage alone ensures that. This kind of sector-specific paradigm-change may well end up being a turning-point for C compiler validation. On the other hand, this author is not betting her pension on it.

 

The order of the day is a clear heads-up notice to all embedded C users developing safety-critical systems . Watch this space ...

 

PS: Please note that if any readers make enquiries regarding this article, both the author and Phaedrus Systems may be somewhat reticent owing to client confidentiality. In particular, the controller developer, the compiler developer and the safety authority will remain unnamed.