Task: Find the Most Recent Analysis of a Named Project with a Python Script

• Permissions Required
• Other Requirements
• The Download Script
• Using the Script
• Modifying the Script
• Writing Other Download Scripts
• Links

Permissions Required

The script requires that special user Anonymous has the following permissions for the project P of interest and its most recent analysis A.

• PROJECT_EXISTS P
• ANALYSIS_EXISTS A

See Modifying the Script for information on modifying the script to specify credentials for a non-Anonymous user with the required permissions.

Other Requirements

The Download Script

The following script will download the CSV project search results for the project (or projects) on the hub specified in the first argument to the script whose name matches the second argument. It will use these results to generate a summary CSV document that lists, for each project found:

• The analysis ID of the project's last analysis, if it can be identified.
• The project's project ID if the last analysis cannot be identified. (That is, either there are no analyses of the project or there are analyses but Anonymous does not have ANALYSIS_EXISTS permission for the last analysis.)

Both CSV files are stored in the directory specified in the third argument to the script.

import argparse
import csv
import os
import re
import shutil
import subprocess


# Find the most recent analysis of the named project and record its
# Analysis ID. Note that there may be multiple projects with the same
# name, in which case information for all of them will be recorded.
def find_last_analysis(hubaddr, project_name, savedir):
    curl_cmd = ['curl']

    project_search_csv = '{0}/project_search.csv?query=project%3D"{1}"&scope=all'.format(hubaddr, project_name)

    result_fname='overall_results.csv'
    search_fname='project_search_results.csv'

    search_cmdline = curl_cmd + ['-o',search_fname] + [project_search_csv]
    subprocess.check_call(search_cmdline)
    search_results=[]

    no_analysis = 'No most recent analysis found. Either: a) the project has '\
                  + 'no analyses or b) you do not have ANALYSIS_EXISTS '\
                  + 'permission for the most recent analysis.'
    num_results = 0
    with open(search_fname, 'r', newline='') as csvfile:
        search_results = csv.DictReader(csvfile)

        with open(result_fname, 'w', newline='') as outfile:
            fieldnames = ('project id',
                          'most recent analysis id',
                          'notes',)
            writer = csv.DictWriter(outfile, fieldnames=fieldnames)
            writer.writeheader()
            for row in search_results:
                result_match = re.search(r'.*/(?P<rtype>analysis|project)/(?P<rid>\d+)\.csv',
                                         row['url'])
                # If the last column of a result line is neither a project
                # URL nor an analysis URL, report a problem.
                if result_match is None:
                    print('Unexpected result {0} in {1}.'.format(str(row),
                                                                 result_fname))
                    continue

                num_results += 1
                # An analysis URL identifies the most recent analysis.
                if result_match.group('rtype')=='analysis':
                    writer.writerow(
                        {'most recent analysis id':result_match.group('rid')})
                    # A project URL indicates that the most recent analysis 
                    # of that project cannot be reported.
                else:
                    writer.writerow(
                        {'project id':result_match.group('rid'),
                         'notes':no_analysis})

    if num_results==0:
        # If no results, report and exit.
        print('No projects with URL-encoded name', project_name, 'were found.')
        print('This may indicate one or more of the following.')
        print('- The project name was not specified correctly (remember to URL-encode).') 
        print('- You do not have PROJECT_EXISTS permission for the specified project.')
        exit(1)

    print('Found {0} project(s) with URL-encoded name {1}.'.format(num_results,
                                                                   project_name))
    print('Your results are in {0}/{1}'.format(savedir,result_fname))
    

# Set up.
def go():
    parser = argparse.ArgumentParser(
        description=('Find the most recent analysis of '
                     + 'a project on a CodeSonar hub, '
                     + 'as specified by the command-line arguments.'))
        
    parser.add_argument("hub",
                        help="The hub URL.")
    parser.add_argument("pname",
                        help="The project  name; must be URL-encoded.")
    parser.add_argument("outdir",
                        help="The save directory.")
    args = parser.parse_args()

    allargs = (args.hub, args.pname, args.outdir)
    if not any([a is None for a in allargs]):
        if os.path.isdir(args.outdir):
            print('Output directory {0} exists, deleting and recreating.'.format(args.outdir))
            shutil.rmtree(args.outdir)
        os.mkdir(args.outdir)
        os.chdir(args.outdir)

        find_last_analysis(*allargs)

go()

This Python script works as follows.

1. It constructs a query string using the project search language to specify that
   • the project name (field-name project) ...
   • ... must case-insensitively match (=, which is URL-encoded as %3D ) ...
   • ... the name specified in the second argument to the script.
2. It accesses the specified hub to download the CSV version of the Project Search Results page for the constructed query string, saving it in the specified output directory with basename project_search_results.csv.

   This is the same file you would download if you did the following.

   1. Navigate to the Home page in the web GUI.
   2. Select projects on the hub from the search domain/scope menu.
   3. Enter project="<pname>" in the search text box, where <pname> is the name of the project you want to find. (Do not URL-escape <pname>.)
   4. Click the Search button.
      (A Project Search Results page will open to show the search results.)
   5. Click the CSV link in the Project Search Results page to obtain the CSV version of the page.
   You may like to try this now so you can see the following.
   • How your search is transformed into a query string in the HTML Project Search Results page URL (in particular, note that URL-escaping has now been applied to <pname>).
   • How the CSV file is formatted.
3. For each line in the CSV file (representing a project whose name is <pname>) except for the first (which is a header line), it does the following.
   1. Inspects the last entry in the line (representing the URL corresponding to the result), extracting the following information.
      1. Whether it is for an Analysis page or a Project page.
         If the former, this is the Analysis page corresponding to the last analysis of the project. If the latter, no last analysis could be found and the hub has fallen back to linking to the project's Project page.
         For more information about CodeSonar URL schemes, see GUI Reference: Page Type Properties.
      2. The "ID" component of the URL: this is an Analysis ID if the URL is for an Analysis page; a Project ID if the URL is for a Project page.
   2. Uses this extracted information to compose and write a line in the result CSV file, which is stored in the specified output directory with basename overall_results.csv.

Using the Script

To use this script with your hub, do the following.

1. Create a directory to save your result files in. The remainder of these instructions will refer to this directory as savedir.
2. Copy find_project_analysis.py to a suitable directory. The remainder of these instructions will refer to this directory as rundir.
   This should not be anywhere under savedir, since the script starts by deleting everything in that directory.
3. Run the script:
   cd rundir
   python find_project_analysis.py protocol://host:port projname savepath
   where

   protocol	is the protocol for your hub: http or https.
   host:port	is the location of your hub.
   projname	is the name of the project whose last analysis you want to find. It must be URL-encoded.
   savepath	is the path to the savedir directory you created in the first step.

   The script will output information about each file it downloads.
4. Inspect the contents of savedir. It should contain two files.
   • overall_results.csv: the result summary computed by the script.
   • project_search_results.csv: the Project Search Results page downloaded from the hub.
5. Open project_search_results.csv to see your results.

Invocation Examples

Using the hub at http://[::1]:7341, find the most recent analysis of the project named "My Favorite Project", saving the results in directory /tmp/csvout:

Troubleshooting

curl_cmd=['curl', '-v']

Modifying the Script

You may wish to make one or more of the following modifications.

• Match project name substrings, rather than matching exactly
• Provide credentials for a non-Anonymous user

Match project name substrings, rather than matching exactly

To change a search to perform substring matching rather than exact matching, we need to edit the corresponding field-condition to change its operator from = to :.
Because of URL-encoding, this becomes a change from %3D to %3A in the query string constructed by the script.

1. Edit the script to change the line setting project_search_csv to

   project_search_csv = f'{hubaddr}/project/project_search.csv?query=project%3A"{project_name}"&scope=all'
   

2. Try running the edited script to see how substring matching affects the results.

Provide credentials for a non-Anonymous user

If your hub is configured so that special user Anonymous does not have the required permissions, you will need to edit the script to submit credentials for a suitable hub user account.

See CodeSonar HTTP API: Authentication for more information on authentication strategies.

Links

• Typical CodeSonar Tasks
• HTTP API Documentation
• The CodeSonar Hub
• CodeSonar GUI Reference