cardinal prepare

Correct quality issues within OCDS compiled releases

The prepare command corrects quality issues within OCDS compiled releases in a line-delimited JSON file. Corrected data is written to one file, and quality issues are written to another file as CSV for review.

This command is designed to only warn about quality issues that (1) it can fix and (2) that interfere with the calculation of indicators. For comprehensive quality checking, consider using Pelican.

Command Syntax

cardinal prepare [OPTIONS] --output <OUTPUT> --errors <ERRORS> <FILE>

Arguments

FILE · string · required

The path to the input file (or - for standard input), in which each line is a contracting process as JSON text.

Options

--settings · string

The path to the settings file. Use -s as a shorthand. If not provided, default settings are used.

--output · string · required

The file to which to write corrected data (or - for standard output). Use -o as a shorthand.

--errors · string · required

The file to which to write quality issues (or - for standard output). Use -e as a shorthand.

--verbose · flag

Increase verbosity. Use -v as a shorthand. Can be repeated for more verbosity.

Output Format

Corrected Data

Corrected data is written as line-delimited JSON, with one compiled release per line.

Quality Issues

Quality issues are written as CSV rows with these columns:

Column	Description
line	Line number of the problematic compiled release in the input file
ocid	OCID of the problematic compiled release (for finding it in other systems)
path	JSON path to the field with an issue (useful for sorting and filtering)
array indexes	Index of the problematic array entry (if path contains arrays)
incorrect value	The value that caused the issue (blank if field isn’t set)
error description	Description of the issue (see table below)

Error Descriptions

Error Description	Meaning	Solution
not set	The field isn’t set	Fill in missing values
invalid	The code isn’t valid	Re-map incorrect codes
is zero	The bid’s value is zero	Redact incorrect values

Usage Examples

Basic usage

cardinal prepare --output prepared.jsonl --errors issues.csv input.jsonl

Example Quality Issues Output

Given this input file with a bid missing a status:

{"ocid":"ocds-213czf-1","bids":{"details":[{"id":1}]}}

The errors CSV output would be:

line,ocid,path,array indexes,incorrect value,error description
1,ocds-213czf-1,/bids/details[]/status,0,,not set

Configuration

The prepare command can be configured through a settings file. Initialize one with:

cardinal init settings.ini

Automatic Corrections

These corrections are always applied and cannot be disabled:

Correct Structural Errors

Replaces objects with arrays where OCDS expects arrays:

• /bids/details[]/tenderers
• /awards/suppliers

Normalize ID Fields

Converts ID fields from integers to strings to prevent lookup failures:

• /parties[]/id
• /buyer/id
• /tender/procuringEntity/id
• /bids/details[]/tenderers[]/id
• /awards[]/id
• /awards[]/suppliers[]/id
• /awards[]/items[]/classification/id
• /contracts[]/awardID

Fill in Missing Values

Add a [defaults] section to fill in missing fields:

[defaults]
currency = USD
item_classification_scheme = UNSPSC
bid_status = valid
award_status = active
party_roles = true

Supported fields:

• /bids/details[]/value/currency
• /bids/details[]/items[]/classification/scheme
• /bids/details[]/status
• /awards[]/items[]/classification/scheme
• /awards[]/status
• /parties[]/roles[] (when party_roles = true)

Redact Incorrect Values

Monetary Amounts

Redact placeholder amount values:

[redactions]
amount = 0|99999999

Affects: /bids/details[]/value/amount

Organization IDs

Redact placeholder organization IDs:

[redactions]
organization_id = my-placeholder|dummy-value

Affects:

• /parties[]/id
• /buyer/id
• /tender/procuringEntity/id
• /bids/details[]/tenderers[]/id
• /awards[]/suppliers[]/id

Re-map Invalid Codes

Substitute invalid codelist values:

[codelists.bid_status]
Qualified = valid
Disqualified = disqualified
InTreatment = pending

[codelists.award_status]
Active = active
Cancelled = cancelled

Move Auction Bids

Move non-standard auction bids to the standard location:

[modifications]
move_auctions = true

Moves bids from /auctions[]/stages[]/bids[] to /bids/details[].

Prefix Organization IDs

Add prefixes to organization IDs to match /parties[] entries:

[modifications]
prefix_buyer_or_procuring_entity_id = DO-UC-
prefix_tenderer_or_supplier_id = DO-RPE-

Affects:

• /buyer/id
• /tender/procuringEntity/id
• /bids/details[]/tenderers[]/id
• /awards[]/suppliers[]/id

Standardize Unconstrained Values

Split and standardize text values:

[modifications]
split_procurement_method_details = -

For example, “Open Bidding - Emergency” becomes “Open Bidding”.

Affects: /tender/procurementMethodDetails

Replace Incorrect Award Statuses

Change award status based on contract statuses:

[corrections]
award_status_by_contract_status = true

An award’s status is set to cancelled if all related contracts are cancelled.

Workflow

Before running prepare, follow the data preparation steps in the main workflow documentation.

1. Initialize settings

   Create a settings file:

   cardinal init settings.ini
   
2. Run prepare command

   Process your data:

   cardinal prepare --settings settings.ini \
     --output prepared.jsonl \
     --errors issues.csv \
     input.jsonl
   
3. Review quality issues

   Open issues.csv in a spreadsheet application to review problems. Most issues are repetitive and can be fixed at once with configuration changes.

4. Adjust configuration

   Update settings.ini to fix the quality issues identified in the previous step.

5. Iterate

   Repeat steps 2-4 until you’re satisfied with the results.

Next Steps

Indicators Command

Calculate procurement indicators from prepared data

Coverage Command

Analyze field coverage in your data