Importing and Exporting Annotations

• Definitions
• Some Use Cases
• Exporting
• Importing
• Example
• Updating Hub Annotation Encoding
• Permissions

Important Note: the CodeSonar Web GUI makes extensive use of JavaScript. Make sure JavaScript is enabled in your web browser.

Definitions

The annotation files imported and exported by CodeSonar contain sets of annotation tuples in CSV (comma-separated value) format compliant with RFC 4180.

Some Use Cases

Use cases for CodeSonar's annotation import/export functionality include the following.

• Mirror annotations across multiple hubs in multiple directions.

  On a regular basis (say, weekly):

  1. Regularly export annotations from each hub.
  2. When exporting is finished, import into each hub the annotations from all other hubs.
  Task: Set Up Automated Annotation Mirroring provides step by step instructions for one approach to this procedure.
• Allow annotations to follow branching and merging in the code base.
  1. Analyze different branches on separate hubs.
  2. When merging branch B into branch A, export all the branch B annotations, then import them into the branch A hub.
  or
  1. Set warning group sharing to OFF before any branches of the project are analyzed (including the trunk).
  2. Analyze different branches under different project names.
  3. When merging branch B into branch A, export the annotations from the project associated with branch B, then import them into the project associated with branch A.
• A lightweight alternative to performing a full hub backup.

  User annotations are typically the most valuable information on a hub: in some cases you may prefer to export annotations instead of carrying out the full backup operation.

  1. Export annotations from the hub home page to capture all user annotations on the hub.
  2. In the event that the hub needs to be restored, import the annotations into the new hub. CodeSonar will attach them to the appropriate warnings once the same projects have been analyzed.
• Perform a one-time annotation encoding update for a hub that has been upgraded directly or gradually from CodeSonar 6.0 or earlier.

  CodeSonar 6.1 introduced UTF8 encoding for all hub annotations. Hubs with ASCII-only annotations will have their annotations automatically transcoded to UTF8 as part of the upgrade from (or past) CodeSonar 6.0. For hubs whose annotations used other encodings, use the Updating Hub Annotation Encoding process described below to ensure that all annotations are correctly transcoded to UTF8.

Exporting

[Jump to How to Export Annotations]

Exporting annotations for a set of warning groups entails writing the corresponding set of nontrivial annotation tuples to a file.

Warning annotations can be exported from several GUI page types, as described in the following table.

How to Export Annotations

1. Determine the set of warning groups for which you want to export annotations.
2. Navigate to the GUI page corresponding to your chosen set of warning groups (see the table above for guidance).
3. Click the Export link (under the table on that page).
4. CodeSonar will export a CSV-format file containing the annotation tuples for your warning groups.

Importing Annotations

[Jump to How to Import Annotations]

Importing annotations for a set of warning groups entails reading the corresponding set of annotation tuples from a file and merging those annotations with a hub's existing annotations.

The availability of annotation import functionality depends on the hub's current setting for warning group sharing, as described in the following table.

Creating A New Warning Group

Given a tuple T=(P,F,I,C,t), we define a new [X-scoped] warning group matching T as a new warning group with:

• Scope as specified by X.

  If X is not specified, the scope of the new warning group corresponds to P: if P=="" then the group has global scope, otherwise the group has project scope and is associated with project P.

• Fingerprint = F
• Priority, State, Finding, Owner, and Notes as specified by C.
  • If the Priority value specified by C is not already available on the current hub, it is added to the hub's Priority set if the importing user has G_PRIORITY_ADD permission, otherwise the Priority for the new warning is set to None.
    State and Finding have analogous behavior.
  • If the Owner value specified by C does not correspond to an existing hub user, a new hub user account with that name is created if the importing user has G_CREATE_USER permission, otherwise the new warning is not assigned an Owner.
• The user who imported the annotations will be noted in the warning group's change history, along with the date and time.
• First Detected=t

Note that a new warning group based on some imported tuple will initially have no instances.

New tuples will only created during annotation import if the user performing the import has G_ANNOTATION_IMPORT permission.

Merging Annotations

When an imported tuple T=(P,F,I,C,t) corresponds to a warning group G that already exists on the target hub, the annotations described in T must be merged with any existing annotations on G.

For a particular property described in T, we say that T has precedence over G for this property if:

• the property has undergone one or more changes in T,
  and
• at least one of the following is true:
  • the same property has not undergone any changes in G,
    or
  • overwrite was enabled for the import operation.
    (Note that it is disabled by default.)

Merging is carried out as follows.

Note on common annotation history

In some situations the annotations on group G and the annotations encoded by tuple T may reflect some shared history. This can be due to

• previous merging, or
• analysis branching, or
• T having actually been exported from G in the first place.

In this case, a warning group is considered to have "undergone one or more Priority changes" if its Priority has changed since the most recent shared Priority change. (State, Finding, and Owner are analogous).

Creating a New Hub User Account

In some circumstances, new hub user accounts will be created during annotation merging.

Properties for these new accounts are initialized as follows.

Note also that the user performing the annotation import must have G_ANNOTATION_IMPORT or G_CREATE_USER permission for new accounts to be created during import. Otherwise, Owner changes that do not correspond to an existing hub user account will be marked with "Property Conflict: Discarded".

How to Import Annotations

1. Are you importing annotations that you exported from a hub running CodeSonar 6.0 or earlier?
   • NO: Go on to the next step.
   • YES: Are you importing the annotations into a hub running a newer version of CodeSonar? (version 6.1 or later)?
     • NO: Go on to the next step.
     • YES: Do the following to ensure that all annotations are encoded consistently.
       1. Ensure that annotations on the destination hub are correctly encoded.
          If necessary, work through the steps for updating hub annotation encoding
       2. Run the following command to update the encoding on the annotations you are importing.
          codesonar transcode_annotations.py path/to/exp_anns.csv path/to/transcoding_output.csv [encoding]
          where:

          path/to/exp_anns.csv	is your existing file of exported annotations.
          path/to/transcoding_output.csv	is the output file produced by changing the encoding of path/to/exp_anns.csv. This is the file you will import into your hub.
          [encoding]	specifies the encoding of the exported annotations: one of (case insensitive) ASCII, Big5, EUC-JP, EUC-KR, EUC-TW, HZ-GB-2312, GB2312/GB18030, IBM855, IBM866, ISO-2022-CN, ISO-2022-JP, ISO-2022-KR, ISO-8859-1, ISO-8859-2, ISO-8859-5, ISO-8859-5, ISO-8859-7, ISO-8859-8, KOI8-R, MacCyrillic, SHIFT_JIS, TIS-620, UTF-16, UTF-32, UTF-8, and windows-1251, windows-1250, windows-1251, windows-1252, windows-1253, windows-1255. If not specified, transcode_annotations.py will attempt to deduce the encoding and prompt you for confirmation.

2. Navigate to the appropriate GUI page.
   • If group sharing is ON, you can only import from the Home page.
   • If group sharing is OFF, choose the project for which you wish to import annotations and navigate to the corresponding Project page.
3. Click the Import link (under the table on that page). A dialog will open:

   

4. Specify an annotation file for importing by clicking Choose File and using the file selection dialog that opens.
5. Decide how you want merge conflicts to be resolved, and select or deselect the overwrite checkbox accordingly.
   • deselect overwrite to resolve conflicts by using the existing hub values. This is the default behavior.
   • select overwrite to resolve conflicts by using the values specified in the imported annotation file.
6. If you have a Template User menu, select the user account that you want to use as the template for any new accounts created during annotation import.
   You will not have a Template User menu if you don't have G_ADMINISTER_USERS permission. In this case, any new accounts created during annotation import will be based on the default template user.
7. Click Import File. CodeSonar will open a page with a streaming progress report.

   

   When the import is complete, the page will display a Continue link.
8. Click Continue to go back to your previous page (which will be either Home or a Project page).

Updating Hub Annotation Encoding

Hubs undergo ASCII→UTF8 annotation transcoding as part of the upgrade from CodeSonar 6.0 (or earlier) to a later version. If some or all of the annotations on your hub previously used non-ASCII encodings, perform the following steps to ensure all existing annotations are correctly encoded. You will need G_ANNOTATION_IMPORT and G_ANNOTATION_EXPORT permissions.

If you have already performed these steps for your hub, you do not need to perform them again.

1. Inspect existing annotations on your hub to see if they are rendering correctly.
   • In particular, look at annotations that were made before the hub was upgraded past CodeSonar 6.0.
   • If annotations can occur in multiple languages, look at examples of each.
   If the inspected annotations are all correctly rendered, you don't need to do anything further.
   If there are problems with annotation rendering, proceed with the remainder of these steps to export, transcode, and reimport the hub annotations.
2. Export all annotations on your hub.

   The export process depends on the warning group sharing setting for your hub.

   • Group sharing ON (this is the default).
     1. Navigate to the hub Home page.
     2. Click the Export link under the table of all projects.
        CodeSonar will export a CSV-format file containing all nontrivial annotation tuples for all global-scoped and project-scoped warning groups on the hub (including those for which no warning instances currently exist). The default name of this file is annotations.csv.
     3. Copy annotations.csv to a working directory.
   • Group sharing OFF.

     For each project in turn:

     1. Navigate to the corresponding Project page.
     2. Click the Export link under the table of analyses.
        CodeSonar will export a CSV-format file containing all nontrivial annotation tuples for all warning groups for which the project has an extant warning instance. The default name of this file is annotations.csv.
     3. Copy annotations.csv to a working directory and change its name to include the project name. For example, if the project name is ProjectX, you might change the name to annotations_ProjectX.csv.
3. Run the following command to update the encoding.
   codesonar transcode_annotations.py annotations.csv annotations_utf8.csv [encoding]
   The optional encoding argument specifies the encoding of the exported annotations (see list of options above). If not specified, transcode_annotations.py will attempt to deduce the encoding and prompt you for confirmation.

   If warning group sharing is OFF for your hub, and so you exported multiple annotation files in the previous step, convert each file separately. For example, if the project named ProjectX has ISO-8859-7-encoded annotations and you saved its exported annotation file as annotations_ProjectX.csv:

   codesonar transcode_annotations.py annotations_ProjectX.csv annotations_ProjectX_utf8.csv ISO-8859-7
4. Delete all annotations from your hub database by visiting the following URL.
   Warning: visiting this URL will instantly delete all comments on your hub.
   /sql.html?sql=update%20cs_warningreport%20set%20comments_text_xml%20%3D%27%27%2C%20comment_index_xml%3D%27%27
5. Restart the hub.
6. Import the annotations back into the hub.

   The Import process depends on the warning group sharing setting for your hub.

   • Group sharing ON (this is the default).
     1. Navigate to the hub Home page.
     2. Click the Import link under the table of all projects.
        A dialog will open.
     3. Next to File, click Browse... and use the file selection functionality to select the annotations_utf8.csv file you produced in step 3.
        • The Overwrite setting doesn't matter because there are currently no annotations on the hub. Leave it unselected.
        • The Template User setting doesn't matter because the annotations were exported from the same hub you are importing back into, so all Owners in the exported annotations must correspond to existing hub user accounts. Leave it set to Default Template User.
     4. Click Import File.
        CodeSonar will open a page with a streaming progress report. When the import is complete, the page will display a Continue link.
     5. Click the Continue link to go back to the hub Home page.
   • Group sharing OFF.

     For each project in turn, do the following.

     1. Navigate to the corresponding Project page.
     2. Click the Import link under the table of analyses.
        A dialog will open.
     3. Next to File, click Browse... and use the file selection functionality to select the annotations_ProjectName_utf8.csv file you produced for this project in step 3.
        • The Overwrite setting doesn't matter because the only annotations on the hub are those you are importing in this process. Leave it unselected.
        • The Template User setting doesn't matter because the annotations were exported from the same hub you are importing back into, so all Owners in the exported annotations must correspond to existing hub user accounts. Leave it set to Default Template User.
     4. Click Import File.
        CodeSonar will open a page with a streaming progress report. When the import is complete, the page will display a Continue link.
     5. Click the Continue link to go back to the Project page.
7. Once you have completed all necessary imports, check the hub annotations to verify that they are now all rendering correctly.
   If there are still rendering issues, contact CodeSecure support.

Example

The following diagram illustrates a typical use case. Project P has two development branches: branch P_A is developed by team A and analyzed on hub A, branch P_B is developed by team B and analyzed on hub B. Exporting annotations between the two hubs once a week allows the teams to avoid duplicating effort in resolving shared problems.

One might reasonably ask why teams A and B need separate hubs at all. In fact, they don't. If the two teams now decide that they wish to share hub A, only two steps remain:

• Ensure that hub A has warning group sharing ON.
• Analyze P_B on hub A.

Permissions

CodeSonar permissions apply to annotation import and export as follows.