Task: Download Warnings from an Analysis with a Python Script

• Permissions Required
• Other Requirements
• The Download Script
• Using the Script
• Modifying the Script
• Writing Other Download Scripts
• Running Scripts Automatically
• Links

Permissions Required

The script requires that special user Anonymous has the following permissions for the analysis A of interest.

• ANALYSIS_READ A
• ANALYSIS_EXISTS A
• ANALYSIS_WARNING_EXISTS A
• ANALYSIS_WARNING_READ A

See Modifying the Script for information on modifying the script to specify credentials for a non-Anonymous user with the required permissions.

Other Requirements

The Download Script

The following script will download the HTML warning reports for:

• all warnings satisfying the default warning filter setting for the authorizing user (Anonymous, in this case),
• from the analysis whose analysis ID matches the second argument passed to the script,
• on the hub specified in the first argument to the script,

and store them under the directory specified in the third argument to the script.

import argparse
import csv
import os
import shutil
import subprocess

# Identify and download the warnings.
def download_warnings(hubaddr, analysis_id, savedir):
    curl_cmd = ['curl']

    def check_page(url):
        http_code = subprocess.check_output(curl_cmd
                                            + ['-w', '%{http_code}',
                                               '-o', os.devnull,
                                               url])
        return (int(http_code.strip())==200)

    analysis_csv_url = f'{hubaddr}/analysis/{analysis_id}.csv'
    out_csv = 'warnings.csv'

    if not check_page(analysis_csv_url):
        print(f'Could not access analysis page for analysis {analysis_id}')
        print('This may indicate one or more of the following.')
        print('- The analysis ID was not specified correctly.') 
        print('- You do not have ANALYSIS_READ permission for the analysis')
        exit(1)
    
    cmdline = curl_cmd + ['-o', out_csv,
                          analysis_csv_url]
    subprocess.check_call(cmdline)
    search_results=[]
    with open(out_csv, 'r', newline='') as csvfile:
        search_results = csv.DictReader(csvfile)

        firstrow=True
        for row in search_results:
            warning_rel = row['url'].replace('.txt','.html')
            warning_url = f'{hubaddr}{warning_rel}'
            if firstrow:
                if not check_page(warning_url):
                    print(f'Could not access Warning Report {warning_url}.')
                    print(f'Make sure you have ANALYSIS_WARNING_READ permission for analysis {analysis_id}.')
                    exit(1)
                firstrow=False
            dl_cmdline = curl_cmd + ['-o', os.path.basename(warning_rel),
                                     warning_url]
            subprocess.check_call(dl_cmdline)

        if firstrow: 
            # No results: report and exit.
            print('No warnings were found for analysis ID', analysis_id)
            print('This may indicate one or more of the following.')
            print('- The analysis ID was not specified correctly.') 
            print('- You do not have ANALYSIS_WARNING_EXISTS permission for the analysis.')
            print('- The analysis has no active warnings.')
            exit(1)


# Set up.
def go():
    parser = argparse.ArgumentParser(
        description=('Download warnings from an analysis on a CodeSonar hub, '
                     + 'as specified by the command-line arguments.'))
        
    parser.add_argument("hub",
                        help="The hub URL.")
    parser.add_argument("aid",
                        help="The analysis ID.")
    parser.add_argument("savedir",
                        help="The save directory.")
    args = parser.parse_args()

    allargs = (args.hub, args.aid, args.savedir)
    if not any([a is None for a in allargs]):
        if os.path.isdir(args.savedir):
            print(f'Output directory {args.savedir} exists, deleting and recreating.')
            shutil.rmtree(args.savedir)
        os.mkdir(args.savedir)
        os.chdir(args.savedir)

        download_warnings(*allargs)

go()

This Python script works as follows.

1. It accesses the specified hub to download the CSV version of the Analysis: Warnings page for the analysis with the specified analysis ID.
   (This is the same file you would download if you navigated to the Analysis: Warnings page in the web GUI and clicked the CSV link. You may like to try this now so you can see how the file is formatted.)
2. For each line in the CSV file (representing a single warning) except for the first (which is a header line), it does the following.
   1. Constructs the URL for the HTML warning report by taking the last entry in the line (representing the URL of the text version of the warning report) and replacing '.txt' at the end with '.html'.
   2. Downloads the URL, storing it in the location under the specified save directory that corresponds to its path.

Using the Script

To use this script with your hub, do the following.

1. Create a directory to save your downloaded warnings in. The remainder of these instructions will refer to this directory as savedir.
2. Copy download_warnings.py to a suitable directory. The remainder of these instructions will refer to this directory as rundir.
   This should not be anywhere under savedir, since the script starts by deleting everything in that directory.
3. Run the script:
   cd rundir
   python download_warnings.py protocol://host:port aid savepath
   where

   protocol	is the protocol for your hub: http or https.
   host:port	is the location of your hub.
   aid	is the analysis ID for the analysis whose warnings you wish to download. You can find the analysis ID: in the URL specified at the end build/analysis command output, which will be of the form http://hub_location/analysis/analysis_id.html, or in file path/to/projname.prj_files/aid.txt, where path/to/projname.prj_files/ is the analysis directory, or by running the following command, where path/to/projname.prj_files/ is the analysis directory codesonar analysis_id.py path/to/projname.prj_files/ or in the Analysis Details section of the corresponding Analysis page.
   savepath	is the path to the savedir directory you created in the first step.

   The script will output information about each HTML file it downloads.
4. Inspect the contents of savedir. It should contain a file called warnings.csv, along with all the HTML warning reports from your specified analysis.
5. Open one of the downloaded HTML files in a web browser to confirm that it contains a warning report.
   Note that links to files that the script did not download (including images) will be broken: this is expected behavior.

Invocation Examples

• Example 1: Using the hub at http://[::1]:7341, download warnings for the analysis with ID 3 and save in directory /tmp/mywarnings:
  python download_warnings.py http://[::1]:7341 3 /tmp/mywarnings
• Example 2: Using the hub at http://[::1]:7341, download warnings for the analysis whose analysis directory is /myprojects/projectX.prj_files/ and save in directory /tmp/mywarnings:
  python download_warnings.py http://[::1]:7341 `codesonar analysis_id.py /myprojects/projectX.prj_files/` /tmp/mywarnings
  • If it is not in your PATH, adjust the command to include the full path to your codesonar executable.
  • On Cygwin, do the following instead.
    python download_warnings.py http://[::1]:7341 `codesonar analysis_id.py /myprojects/projectX.prj_files/ | tr -d '\r'` /tmp/mywarnings

Troubleshooting

curl_cmd=['curl', '-v']

Modifying the Script

You may wish to make one or more of the following modifications.

• Download warnings that satisfy a specific filter
• Download XML warning reports instead
• Read analysis ID from analysis directory
• Provide credentials for a non-Anonymous user

Download warnings that satisfy a specific filter

If an Analysis: Warnings URL is specified without a query string component, the default warning filter setting for the authorizing user (Anonymous, in this case) is applied. To apply a different visibility filter, the URL must include a query string that specifies a filter value.

For example, suppose we want to specify the all visibility filter.

1. Edit download_warnings.py so that the line setting analysis_csv_url is:

   analysis_csv_url=f'{hub}/analysis/{analysis_id}.csv?filter=\"all\"'
   

Download XML warning reports instead

Warning reports can be output in text and XML formats as well as HTML. To download the XML versions, do the following.

1. Edit download_warnings.py so that the line setting warning_url is changed to:

   warning_url = row['url'].replace('.txt','xml')

   (That is, replace the occurrence of .html with .xml).
2. If you want to process the downloaded files automatically, the XML schema at warning_report.xsd is likely to be useful.

Read analysis ID from analysis directory

Instead of specifying the analysis ID on the command line, you can change the script to read the analysis directory from the command line and then read the most recent analysis ID from the analysis directory.

1. In function go(), replace line

   parser.add_argument("aid", help="The analysis ID.")

   with a line adding an adir argument.

   parser.add_argument("adir", help="The analysis .prj_files directory.")
   

2. In function go(), change the definition of allargs to:

   allargs = (args.hub, args.adir, args.savedir)
   

3. Modify the download_warnings() function header, and add two new lines to the very beginning of the function definition.

   def download_warnings(hubaddr, analysis_dir, savedir):
       with open(os.path.join(analysis_dir,'aid.txt'),'r') as aidfile:
           analysis_id=aidfile.read()
   

4. When you invoke the script, specify the analysis directory as the second argument.

   For example: using the hub at http://[::1]:7341, download warnings for the analysis whose analysis directory is /myprojects/projectX.prj_files/ and save in directory /tmp/mywarnings.

   python download_warnings.py http://[::1]:7341 /myprojects/projectX.prj_files/ /tmp/mywarnings

Provide credentials for a non-Anonymous user

If your hub is configured so that special user Anonymous does not have the required permissions, you will need to edit the script to submit credentials for a suitable hub user account.

See CodeSonar HTTP API: Authentication for more information on authentication strategies.

Running Scripts Automatically

You can use your system tools to arrange for the Python script to be run automatically.

5 2 * * * python /path/to/download_warnings.py http://red:7341 `cat /home/projectX.prj_files/aid.txt` /tmp/mywarnings

Links

• Typical CodeSonar Tasks
• HTTP API Documentation
• The CodeSonar Hub
• CodeSonar GUI Reference
• codesonar dump_warnings.py: Download Specified Warnings