cardinal coverage

Count the number of times each field is non-empty in line-delimited JSON

The coverage command analyzes field coverage in line-delimited JSON files. It counts how many times each JSON field contains non-empty data, helping you understand data completeness and identify missing fields.

Command Syntax

cardinal coverage [OPTIONS] <FILE>

Arguments

FILE · string · required

The path to the input file (or - for standard input), in which each line is JSON text.

Options

--verbose · flag

Increase verbosity. Use -v as a shorthand. Can be repeated for more verbosity.

How It Works

The command walks the JSON tree and counts non-empty nodes. A node is considered empty if it is:

• An empty string: ""
• An empty array: []
• An empty object: {}
• Null: null
• A node containing only empty nodes

The result is a JSON object where:

• Keys are JSON paths
• Values are counts of non-empty occurrences

Path Notation

Path Format	Represents
""	A line in the file
/	An object
[] (at end)	An array element
Other paths	Object members

Usage Examples

Basic usage

cardinal coverage data.jsonl

Example Input and Output

Given this input file example.jsonl:

{"phoneNumbers": [{"type": "home", "number": "212 555-1234"}, {"type": "office", "number": "646 555-4567"}]}

Running the command:

cardinal coverage example.jsonl

Produces this output:

{
  "/phoneNumbers[]/type": 2,
  "/phoneNumbers[]/number": 2,
  "/phoneNumbers[]/": 2,
  "/phoneNumbers[]": 2,
  "/phoneNumbers": 1,
  "/": 1,
  "": 1
}

Interpreting Results

From the example above:

Path	Count	Meaning
""	1	One line in the file
/	1	One root object
/phoneNumbers	1	The phoneNumbers field appears once
/phoneNumbers[]	2	Two array elements in phoneNumbers
/phoneNumbers[]/	2	Both array elements are objects
/phoneNumbers[]/type	2	Both objects have a type field
/phoneNumbers[]/number	2	Both objects have a number field

Use Cases

Data Quality Assessment

Identify which fields are consistently populated vs. frequently missing

Schema Validation

Verify that required fields are present in your OCDS data

Data Completeness

Measure how complete your data is before running indicators

Field Discovery

Discover what fields exist in unfamiliar datasets

Analyzing OCDS Data

For OCDS compiled releases, coverage analysis helps identify:

Critical Missing Fields

Fields required for indicator calculations:

• /ocid
• /buyer/id
• /tender/procuringEntity/id
• /bids/details[]/tenderers[]/id
• /awards[]/status

Bid Information Completeness

Check if bid data is well-populated:

• /bids/details[]/value/amount
• /bids/details[]/value/currency
• /bids/details[]/status
• /bids/details[]/date

Award Information

Verify award data availability:

• /awards[]/value/amount
• /awards[]/suppliers[]/id
• /awards[]/items[]/classification/id

Tender Information

Assess tender data coverage:

• /tender/tenderPeriod/startDate
• /tender/tenderPeriod/endDate
• /tender/procurementMethod
• /tender/procurementMethodDetails

Recommended Workflow

1. Run coverage analysis

   Analyze your raw OCDS data:

   cardinal coverage input.jsonl > coverage-raw.json
   
2. Review missing fields

   Examine the coverage report to identify fields with low counts. These may need to be filled in via the prepare command configuration.

3. Configure prepare command

   Update your settings file to fill in missing values:

   [defaults]
   currency = USD
   bid_status = valid
   
4. Re-run coverage after preparation

   Compare coverage before and after:

   cardinal prepare --settings settings.ini --output prepared.jsonl --errors errors.csv input.jsonl
   cardinal coverage prepared.jsonl > coverage-prepared.json
   

Edge Cases

These edge cases are not expected in typical OCDS data but are documented for completeness.

Duplicate Member Names

If a member name appears multiple times in an object, only the last occurrence is counted:

Input:

{"a": 1, "a": ""}

Output:

{}

Empty Member Names

If a member name is empty, its path is the same as its parent object’s path:

Input:

{"": "value", "other": "value"}

Output:

{"/": 2, "": 1}

Member Names Ending with []

If a member name ends with [], its path can collide with array notation:

Input:

{"a": [1], "a[]": 2}

Output:

{"/a[]": 2, "/a": 1, "/": 1, "": 1}

Output Format Details

The output is a single-line JSON object. For easier reading, you can pipe it through jq:

cardinal coverage data.jsonl | jq .

Pretty-printed output:

{
  "/phoneNumbers[]/type": 2,
  "/phoneNumbers[]/number": 2,
  "/phoneNumbers[]/": 2,
  "/phoneNumbers[]": 2,
  "/phoneNumbers": 1,
  "/": 1,
  "": 1
}

Tips for Large Files

For very large datasets:

Sample the data

head -n 1000 large-file.jsonl | cardinal coverage - > sample-coverage.json

Split by time period

# Analyze recent data only
grep '"2024-' large-file.jsonl | cardinal coverage - > 2024-coverage.json

Increase verbosity

# See progress information
cardinal coverage -vv large-file.jsonl > coverage.json

Next Steps

Prepare Command

Fix missing fields identified by coverage analysis

Indicators Command

Calculate indicators once data coverage is sufficient