Authoring CodeSonar Compiler Models in Python

You can author custom compiler models in Python using the cscm module provided with CodeSonar.

---

• Overview
• Before you start
• Implementing a Compiler Model in Python
• Installing the Model
• Configurating Templates and Testing
• Example Model
• Python Documentation: the cscm Module

---

Overview

You can author custom compiler models in Python using the cscm module provided with CodeSonar.

These custom compiler models are implemented as Python classes that subclass cscm.CompilerModel or one of the provided CompilerModel subclasses. The remainder of this page describes the requirements for implementing, registering, and installing a compiler model authored in Python. It also includes an illustrated example model.

Note that you will not be able to execute or test these models using a standard Python binary. Follow the compiler model test instructions to test your custom compiler models.

Before you start

Before you start implementing a compiler model:

1. Make sure you have read the Authoring Compiler Models section. In particular, you should have...
   • ... confirmed that you need to implement a custom model (rather than using an existing model).
   • ... understood the kinds of tasks a compiler model needs to accomplish.
   • ... determined the behaviors that need to be modeled for your particular compiler.
2. Choose a base name for your compiler model. The model name can be the same as the compiler executable name (without file extension), but does not have to be. We will refer to this as cmname in the remainder of these instructions. The base name is used in several contexts to identify your model.
   • cmname must be unique among compiler model names.
   • The name() method of your custom compiler model class must return cmname.
   • The name of the source file containing your model must be of the form cscm_cmname.py.
   • Any COMPILER_MODELS rules that map a native compiler executable to your model must specify mapping target pya_cmname.

Implementing a Python Compiler Model

The compiler model adapter module is named cscm. File cscm.py is located in $CSONAR/csurf/src/compmodels.

Python compiler models are implemented as Python classes. A custom compiler model class must...

• ... specify import cscm.
• ... subclass cscm.CompilerModel or one of the provided CompilerModel subclasses.

  For example, cscm.C_CompilerModel provides implementations of get_prepend_flags() and get_append_flags() with some default arguments used in all C/C++ compiler models.

• ... have a class name that is unique among compiler model class names. A useful convention is to use class names of the form CmnameModel, where Cmname is the capitalized form of the compiler model base name.
• ... implement the following cscm.CompilerModel methods.
  • name()
    Return the name of the model. This will be a fairly short token, such as "mycc". This name must be unique among compiler model names (as shown in the main compiler model table).
    If you want to register a single model under multiple names, construct multiple instances of your class, each of which has a different name() return value, then register each instance with cscm.register_compiler_model().
  • verbose_name()
    Return a longer description of the model, such as "Model for the MyCC Compiler v4.3, MyCompany Inc.".
  • __call__()
    Given the arguments passed to a compiler invocation and information about the native compiler, convert the compiler invocation into one or more invocations of a CodeSonar front end.

    In some cases you may wish to have the model invoke the native compiler directly to obtain certain information, such as the set of predefined macros or the compiler version.

  • get_prepend_flags()
    Given information about the native compiler, return a list of flags that should be prepended to the raw arguments passed to the compiler invocation.
  • get_append_flags()
    Given information about the native compiler, return a list of flags that should be appended to the raw arguments passed to the compiler invocation.
• ... be registered by invoking cscm.register_compiler_model().

An illustrated example compiler model is shown below.

Installing the Model

Once you have your model source code, you will make it available to the CodeSonar installation.

The following requirement applies.

• The source file name must be of the form cscm_cmname.py, where cmname is the base model name you selected above.

Installation Steps

The installation steps are as follows.

1. Copy your .py file to $CSONAR/csurf/lib/:

   Windows	cp cscm_cmname.py %CSONAR%\csurf\lib\
   Otherwise	cp cscm_cmname.py $CSONAR/csurf/lib/

   where $CSONAR is the CodeSonar installation directory.

   This naming scheme and location is sufficient to have CodeSonar recognize that the file contains a compiler model named cmname.

2. Set up one or more COMPILER_MODELS rules to map the relevant native compiler executables to your model. The mapping target will be of the form pya_cmname, where cmname is the base model name you selected above.
   For example: COMPILER_MODELS += mycompiler -> pya_cmname

Configuration Templates and Testing

Follow the instructions in section Authoring Compiler Models to:

1. Create configuration templates (if necessary).
   The configuration template names will be of the form pya_cmname.L.S.conf or pya_cmname.A.L.S.conf for language L, target size S, ABI key A.
2. Test your compiler model. Note that you will not be able to execute or test these models using a standard Python binary: custom compiler models must be loaded by specifying COMPILER_MODELS and then tested within CodeSonar.

Example Model

Suppose we wish to write a model for the (fictitious) mst3k compiler, and that we have determined that the following properties and behaviors need to be accounted for in the model.

mst3k...

• ... compiles C and C++ code.
• ... accepts source file extensions .c, .cpp, .cxx, case-insensitive.
• ... only targets Intel i686 32bit architecture.
• ... uses system include directory /path/to/mst3k/include.
• ... accepts the following command arguments.

  -D value	Define a macro.
  -U value	Undefine a macro.
  -I value	Specify an include directory.
  --ENABLE-RTTI	Enable RTTI. (Otherwise disabled.)
  -c value	Compile a file.
  -help	Print help text.
  @value	Read arguments from a file.
  -O	Enable optimization and define macro __OPTIMIZE__
  --version	Print version information.

• ... defines preprocessor macros as follows.
  • Always defined: ALPHA, BETA, GAMMA.
  • in C mode: RUBICK.
  • in C++ mode: BARATHRUM.
• ... does not function if environment variable DISABLE_MST3000 is set.

We choose mst3000 as the compiler model base name.

Example File

The example compiler model source file is cscm_mst3000.py.

Save a copy of this file to a working directory.

cscm_mst3000.py

r'''cscm_mst3000.py

This file implements a CodeSonar compiler model for the fictional
mst3k compiler.

It should be placed in the $GTHOME\csurf\lib directory.

In order to use the compiler model, the following COMPILER_MODELS setting should be used:
    COMPILER_MODELS += mst3k -> pya_mst3000
    COMPILER_MODELS += mst3k.exe -> pya_mst3000
    
Note that all Python adapter compiler models should be identified as
pya_<model_name> in COMPILER_MODELS rules. When CodeSonar encounters the pya_ prefix, it will look for a file called csurf\lib\cscm_<model_name>.py
name() must return <model_name>'''

import cscm
import os
import re
import platform
import traceback
import logging

'''All compiler models should subclass cscm.CompilerModel.'''
class mst3000(cscm.CompilerModel):

    '''All compiler models need to implement name(). The returned name must be unique in the set of compiler model names.'''
    def name(self):
        return 'mst3000'
    
    '''All compiler models must implement verbose_name'''
    def verbose_name(self):
        return 'mst3000'   

    '''This is the equivalent of operator() in the C++ compiler models. 
       The first argument to __call__ is the list of arguments provided to the compiler.
       The second argument is the ParseConfiguration object.'''
    def __call__(self, *args, **kwargs):
    
        '''The list of options that will always be passed to the front end.'''
        fecmds_always = []
        '''The list of options that will be passed to the front end for all compilations in C++ mode.'''
        fecmds_always_cxx = []
        '''The list of options that will be passed to the front end for all compilations in C mode.'''
        fecmds_always_c = []

        native_flags = []
        native_cmd_line = []
        
        '''A list of flags that the compiler recognizes as enabling all warnings
          (IE, -Wall in gcc). If this list is empty, the compiler is assumed to
          enable all warnings by default. '''
        wall_enable_flags = ['-Wall']
        '''A list of all flags seen on the command line that disable reporting of warnings'''
        seen_wall_disable_flags = []
        '''A list of flags that the compiler recognizes as treating warnings as errors
           (IE, -Werror in gcc). '''
        werror_enable_flags = ['-Werror'] 
        '''A list of all flags seen on the command line that disable reporting of errors or
           downgrade errors to warnings.'''
        seen_werror_disable_flags = []
        
        arguments = args[0]
        config = args[1]
        
        '''Macros always predefined: ALPHA, BETA, GAMMA'''
        fecmds_always.append('-DALPHA')
        fecmds_always.append('-DBETA')
        fecmds_always.append('-DGAMMA')
        
        '''Additional macro predefined in C++ mode: BARATHRUM'''
        fecmds_always_cxx.append('-DBARATHRUM')
        
        '''Additional macro predefined in C mode: RUBRICK'''
        fecmds_always_c.append('-DRUBICK')
        
        '''Regex patterns for matching C/C++ source files'''
        dotc = '\\.(c|C)$'
        dotcxx = '\\.[cC]([pP][pP]|[xX][xX])$'
        
        source_files = []
        commands = []
        
        '''Reading an option from the environment.
           If DISABLE_MST3000 is set, the compiler will not function.'''
        disable_mst3000 = os.environ.get('DISABLE_MST3000')
        if disable_mst3000 == '1':
            return cscm.CompilerModelResult()
            
        '''Construct the path for a system include directory provided by the compiler'''
        sincdir = config.get_compiler_path_as_string()
        sincdir = os.path.join(sincdir, "../include")
        sincdir = os.path.normpath(sincdir)

        i = 0
        while i < len(arguments):
            arg = arguments[i]
            i += 1
            
            '''-D, -I, and -U flags behave the same as the front end flags of the same names.
                So we will just push them along.'''
            found = True
            for shortflag in 'DUI':
                shortflag = '-' + shortflag
                # Check for '-X foo'
                if arg == shortflag:
                    native_flags.append(arg + arguments[i])
                    if i < len(arguments):
                        commands.append(arg + arguments[i])
                        i += 1
                        break
                # Check for '-Xfoo'
                elif arg.startswith(shortflag):
                    native_flags.append(arg)
                    commands.append(arg)
                    break
            else:
                found = False
            if found:
                continue
            
            '''--ENABLE-RTTI is equivalent to the front-end flag --rtti'''    
            if arg == '--ENABLE-RTTI':
                # --rtti is only legal with c++ translation units
                fecmds_always_cxx.append('--rtti')
                native_flags.append(arg)
                continue
            
            '''-O causes a macro to be defined, so we translate it into a -D argument'''
            if arg == '-O':
                commands.append('-D__OPTIMIZE__')
                native_flags.append(arg)
                continue
                
            '''-Wno-error disables errors'''
            if arg == '-Wno-error':
                native_flags.append(arg)
                seen_werror_disable_flags.append(arg)
                continue
            
            '''-Wno-unused-parameter disables some warnings'''
            if arg == '-Wno-unused-parameter':
                native_flags.append(arg)
                seen_wall_disable_flags.append(arg)
                continue
            
            '''No effect on the compilation, ignore.'''    
            if arg == '-c' or arg == '-help':
                continue
            
            '''Read additional arguments out of a file and append them to the argument list for processing.'''
            if len(arg) > 1 and arg[0] == '@':
                options_from_file = cscm.read_response_file(arg[1:],False)
                arguments = arguments + options_from_file
                continue
            
            '''If an argument matches the pattern for C/C++ source files, add it to the list of source files.'''
            if re.search(dotc, arg):
                source_files.append((arg, cscm.csl_edgcp_c))
                continue
                
            '''If an argument matches the pattern for C/C++ source files, add it to the list of source files.'''
            if re.search(dotc, arg):
                source_files.append((arg, cscm.csl_edgcp_c))
                continue
                
            if re.search(dotcxx, arg):
                source_files.append((arg, cscm.csl_edgcp_cpp))
                continue
                
        '''Add platform-specific front-end argument.'''
        if platform.system() == 'Windows':
            commands.append('--cs_cygwin_path_translation')
            
        commands.append('--cs_define_edg_macros')
        
        bit_width = 32
        
        '''Obtain the basic front-end arguments for this source language, bit width and parse configuration.'''
        edg_fe_cmds_c = cscm.EDGFrontEnd.cs_and_fe_options(cscm.csl_edgcp_c, bit_width, config)
        edg_fe_cmds_cxx = cscm.EDGFrontEnd.cs_and_fe_options(cscm.csl_edgcp_cpp, bit_width, config)

        cso_native_cmd_line_file = cscm.CmdArgumentNativeCmdLineFile(native_flags, arguments, wall_enable_flags, seen_wall_disable_flags, werror_enable_flags, seen_werror_disable_flags)
        _vnative_cmd_line_file = []
        _vnative_cmd_line_file.append(cso_native_cmd_line_file)
        vnative_cmd_line_file = list(map(cscm.CmdArgument, _vnative_cmd_line_file))
        
        vfecmds_always = list(map(cscm.CmdArgument, map(cscm.CmdArgumentString, fecmds_always)))
        vfecmds_always_cxx = list(map(cscm.CmdArgument, map(cscm.CmdArgumentString, fecmds_always_cxx)))
        vfecmds_always_c = list(map(cscm.CmdArgument, map(cscm.CmdArgumentString, fecmds_always_c)))
        vcommands = list(map(cscm.CmdArgument, map(cscm.CmdArgumentString, commands)))
        
        fecs = []
        
        '''Loop through the list of source files and construct an appropriate FrontEndCommand for them.'''
        for sf in source_files:
            extras = vfecmds_always_cxx if sf[1] == cscm.csl_edgcp_cpp else vfecmds_always_c
            
            edg_fe_ffa = cscm.EDGFrontEnd.first_options(sf[0],'object_file.o')
            cao = list(edg_fe_ffa)

            edg_fe_commands = edg_fe_cmds_cxx if sf[1] == cscm.csl_edgcp_cpp else edg_fe_cmds_c
            
            '''Use cscm.concatenate_vector() to merge vectors of CmdArguments.'''
            cao += edg_fe_commands
            cao += vfecmds_always
            cao += extras
            cao += vcommands
            cao += vnative_cmd_line_file

            fec = cscm.FrontEndCommand( cao, sf[0], sf[1], cscm.FET_EDGCP)
            fecs.append(fec)
                    
        return cscm.CompilerModelResult(fecs)
        

cscm.register_compiler_model(mst3000())

            

Install the Example Model

1. Copy cscm_mst3000.py to $CSONAR/csurf/lib/:

   Windows	cp cscm_mst3000.py %CSONAR%\csurf\lib\
   Otherwise	cp cscm_mst3000.py $CSONAR/csurf/lib/

   where $CSONAR is the CodeSonar installation directory.
2. The modeled compiler has been specified as handling both C and C++ and targeting a 32-bit architecture, so we need two compiler template configuration files. We can just copy the project-compiler defaults:
   cp $CSONAR/csurf/compiler_confs/unknown.c.32.conf $CSONAR/csurf/compiler_confs/pya_mst3000.c.32.conf
   cp $CSONAR/csurf/compiler_confs/unknown.c++.32.conf $CSONAR/csurf/compiler_confs/pya_mst3000.c++.32.conf
3. Add the following rule to your general template configuration file. Remember that the pya_ prefix is required for Python compiler models.

   Windows	COMPILER_MODELS += mst3k.exe -> pya_mst3000
   Otherwise	COMPILER_MODELS += mst3k -> pya_mst3000

Test the Example Model

If you want to practise compiler model testing on the mst3000 model, you can do something along the following lines.

1. Since you (presumably) don't have a native mst3k compiler, make a copy of the null-cc executable to serve this role. Put it in your working directory, or another suitable location that is on your PATH.

   Windows	cp $CSONAR/codesonar/bin/null-cc.exe mst3k.exe
   Otherwise	cp $CSONAR/codesonar/bin/null-cc mst3k

2. Create source files testC.c, testCPP.cxx, testRTTI.cxx to exercise the properties and behaviors noted for the native mst3000 compiler.
   • testC.c should be a C file.
   • testCPP.cxx should be a C++ file.
   • testRTTI.cxx should be a C++ file that makes use of RTTI features.
   The test source files should use an assortment of the preprocessor macros defined by the native compiler.
3. For your first check, see what happens when you test the model on testRTTI.cxx with and without the --ENABLE-RTTI flag.
   • When --ENABLE-RTTI is specified, the model invocation should not have any errors.
     rm -rf CMT* *.conf
      
     codesonar compile -TCMT -CCMT -invoke-compiler no -transparent yes
     -compiler-location mst3k -conf-file mytest.CSURF_COMPILER_CONF_MARKER
     pya_mst3000 --ENABLE-RTTI -c testRTTI.cxx
   • When --ENABLE-RTTI is not specified, the model invocation should have errors, because the RTTI elements in testRTTI.cxx will not be supported.
     rm -rf CMT* *.conf
      
     codesonar compile -TCMT -CCMT -invoke-compiler no -transparent yes
     -compiler-location mst3k -conf-file mytest.CSURF_COMPILER_CONF_MARKER
     pya_mst3000 -c testRTTI.cxx
4. Keep testing with other build flag/source file combinations until you are satisfied that you have tested the model thoroughly.

Cleaning Up

If you don't want to keep the sample compiler model:

1. Edit the general template configuration file to remove the COMPILER_MODELS rule you added for pya_mst3000.
2. Remove file cscm_mst3000.py from $CSONAR/csurf/lib/.
3. Remove your model 'mst3k executable' from your working directory (or other location where you created it).
4. Remove $CSONAR/csurf/compiler_confs/pya_mst3000.c.32.conf and $CSONAR/csurf/compiler_confs/pya_mst3000.c++.32.conf.