Including Python Components in a CodeSonar Project

• Overview
• Including Python Components In Your CodeSonar Project
• Warning Classes
• codesonar python_scan.py
• Example Makefile
• Try the Example Makefile
• Related Links

Overview

This page describes how to extend your CodeSonar-facing build so that your CodeSonar project includes:

• the Python source files involved in your regular software build, and
• analysis results for those files, produced by the Pylint analyzer.

Incorporating Python components follows the standard process for including a tier 3 component. The specializations to that process are summarized in the following table. For detailed instructions, see Including Python Components In Your CodeSonar project, below. We also provide an example Makefile.

Including Python Components In Your CodeSonar Project

Incorporating Python components follows the standard process for including a tier 3 component.

1. Install any required language tools, such as compilers.
   Make sure you have a Python installation.
2. Install the third-party analyzer.
   This feature requires Pylint 1.7 or later. If Pylint is not already available on your system, you will need to install and configure it.
   The Pylint installation must be in your PATH.
3. Install the third-party SARIF converter, if needed.
   There is nothing to do for this step: codesonar python_scan.py will take care of converting the Pylint output to SARIF.
4. Extend the regular build to invoke the third-party analyzer (and SARIF converter, if needed).
   You do not need a separate analysis step.
5. Extend the CodeSonar-facing build.
   Extend your CodeSonar-facing build by adding one or more invocations of codesonar python_scan.py.
   All the Python source files must be present before codesonar python_scan.py is invoked. Different build systems have different mechanisms for specifying this requirement. For example, you might be able to specify that the CodeSonar-facing build depends on the Python source files, or that the CodeSonar-facing build can only take place if the steps required to produce the Python source files have already occurred.

   If you have not already set up a CodeSonar-facing build for your regular software project, you will need to create one.

6. Perform the CodeSonar build/analysis, observing the entire execution of your CodeSonar-facing build.

Diagram: including Python components in a CodeSonar project

The following diagram illustrates the extended software build described above, and its interactions with the CodeSonar build/analysis.

• The CodeSonar-facing build could include the entire regular build for a Python component, or any part of that regular build, but it does not need to.
• If the regular build contains a Python compilation step, that compilation has no consequences for the CodeSonar-facing build.

Note

Warning Classes

The SARIF importer has special handling for warning classes derived from SARIF produced by Pylint. For more information, see Warning Classes Corresponding to Pylint Rules.

codesonar python_scan.py

To analyze Python source files with Pylint and then add the source files and analysis results to a project, CodeSonar observes the execution of a command that incorporates one or more invocations of codesonar python_scan.py.

where:

When CodeSonar processes a codesonar python_scan.py command, it does the following.

1. Compute the set of source files to be analyzed. This will contain every source file F.py that matches the specified combination of -include-sources and -exclude-sources options: those specified directly on the command line and those present in a file specified with @infile.
2. Execute a single Pylint invocation on this set of files.
3. Convert the Pylint results to SARIF.
4. Import the SARIF results into the CodeSonar project, along with every F.py in the set.

-include-sources/-exclude-sources Examples

• Use Pylint to analyze myfile.py.
  When Pylint has finished, import the analysis results and myfile.py.
  codesonar python_scan.py -include-sources myfile.py
• Use Pylint to analyze all files that have names ending in .py and are located directly in srcdir/.
  When Pylint has finished, import the analysis results and all the analyzed .py files.
  codesonar python_scan.py -include-sources 'srcdir/*.py'
  Note that the srcdir/*.py pattern is single-quoted ('srcdir/*.py'). This is so that the shell passes it through to codesonar python_scan.py rather than attempting to expand it. For Windows cmd, use double quotes instead ("srcdir/*.py").
• Use Pylint to analyze all files that have names ending in .py and are located in a direct subdirectory of srcdir/.
  When Pylint has finished, import the analysis results and all the analyzed .py files.
  codesonar python_scan.py -include-sources 'srcdir/*/*.py'
• Use Pylint to analyze all files that have names ending in .py and are located anywhere in the directory tree rooted at srcdir/.
  When Pylint has finished, import the analysis results and all the analyzed .py files.
  codesonar python_scan.py -include-sources 'srcdir/**/*.py'
• Use Pylint to analyze all files that have names ending in .py and are located anywhere in the directory tree rooted at srcdir/, unless they are in the directory tree under srcdir/noinclude/.
  When Pylint has finished, import the analysis results and all the analyzed .py files.
  codesonar python_scan.py \
  -include-sources 'srcdir/**/*.py' -exclude-sources 'srcdir/noinclude/**/*'
  The relative order of -include-sources and -exclude-sources is important: if -exclude-sources were first it would be superseded by the subsequent -include-sources and all .py files under srcdir/ would be imported.
• Use Pylint to analyze all files that are located anywhere in the directory tree rooted at srcdir/, unless they are in the tree under srcdir/noinclude/ and not under a directory named actuallyyes/ in that tree.
  When Pylint has finished, import the analysis results and all the analyzed .py files.
  codesonar python_scan.py \
  -include-sources 'srcdir/**/*.py' -exclude-sources 'srcdir/noinclude/**' \
  -include-sources 'srcdir/noinclude/**/actuallyyes/**/*.py'

Importing Python source files with codesonar python_scan.py

The codesonar python_scan.py subcommand imports Python source files into the CodeSonar project using the same mechanism as codesonar import_sarif.py, because CodeSonar does not have a full front end for Python.

Example Makefile

The Makefile illustrates a case where the Python component is not compiled. The all dependencies are thus all the .py source files in the component, and there are no .pyc targets. If the regular build included a Python compilation step, the CodeSonar-facing build would be the same as shown here.

.PHONY: all clean csonar_facing

all: $(SRCDIR)/python_component.py

# Invoke codesonar python_scan.py to analyze the Python source file with Pylint,
# then import the SARIF results and the source file into the project.
csonar_facing: $(SRCDIR)/python_component.py
	"$(CSONAR)"/codesonar/bin/codesonar python_scan.py -include-sources "$(SRCDIR)/python_component.py"

clean:

Try the Example Makefile

We have provided a minimal sample project so you can experiment with the example Makefile.

1. Sign in to a machine where CodeSonar is installed.
2. Select or create a working directory. The working directory location must be outside both your CodeSonar installation directory and your hub directory.
3. If any of the following are not installed, install them now.
   • Python
   • Pylint
4. If your Pylint installation is not in your PATH, add it now.
5. Save the example files to the working directory (right-click on the link and select Save Target As... or equivalent).
   • Makefile.Python.txt
   • python_component.py
6. cd to the working directory.
7. Rename the Makefile (this simplifies the CodeSonar analysis command later).
   mv Makefile.Python.txt Makefile
8. Edit Makefile to add a setting for SRCDIR at the top (above the .PHONY line). If CSONAR isn't defined in your environment, set that as well.

   SRCDIR=.
   

   or

   SRCDIR=.
   CSONAR=path/to/your/codesonar/installation
   # on Windows, typically CSONAR=C:\\Program Files\\CodeSecure\\CodeSonar
   

9. Build and analyze the CodeSonar project. If your hub is at the default location, your command will be something like:
   codesonar analyze PythonExample make csonar_facing
   For full codesonar analyze command details, see Command Line Build/Analysis.
10. When the analysis finishes, view the Analysis page in the hub GUI.
    • The Warnings tab shows a warning instance for each Pylint result. Click any table entry to view the corresponding warning report.
    • The Files tab has an entry for the Python source file python_component.py. You can click on the file name to view a source listing: as noted above, there is no additional source highlighting or navigation.
    • There are no entries in the Procedures tab: source files imported with codesonar import_sarif.py have file-level internal representation (IR) only.

Other Targets in the Example Makefile

The clean target has an empty recipe because codesonar python_scan.py cleans up its own intermediate files.

Related Links

• Including Tier 3 Components in a CodeSonar Project
• More on codesonar import_sarif.py