Skip to main content

CSV ingestion

Comma-separated values (CSV) is a text file format commonly used for data exchange. You can upload CSV files to the DoiT Cloud Navigator to:

  • Create new datasets.

  • Add new records to an existing dataset.

  • Overwrite records.

Syntax and conventions​

When preparing CSV for the DoiT DataHub, follow the syntax and conventions listed below:

  • Separate values with commas.

  • Add each data record as a new line, using Carriage Return and Line Feed (CRLF) as the line break.

  • The first line is a header row.

Header row syntax: usage_date[,id],DIMENSION_KEY...,METRICS_TYPE...

  • Field names in the header row must use the correct format; DoiT DataHub doesn't modify the field names.

  • usage_date: The timestamp (RFC3339) of the data record, corresponding to the time property in the DataHub Events schema.

  • id: The unique identifier of the record, corresponding to the id property in the DataHub Events schema.

    • If you prefer the UUIDv4 identifier automatically generated at ingestion time, do not include this property in your CSV.
    • To overwrite an existing record in a dataset, make sure to include the correct id in your CSV.

  • DIMENSION_KEY: The keys of dimensions in your dataset. At least one dimension key must be present.

    • For fixed dimensions, see Allowed keys for fixed dimensions.
      • Fixed dimensions must be prefixed with fixed, for example, fixed.billing_account_id, fixed.sku_description.
      • Fixed dimensions are case insensitive.
    • For label dimensions, use the format label.CUSTOM_DIMENSION_KEY. For example, label.foo. Label dimensions are case sensitive.
    • For project_label dimensions, use the format project_label.DIMENSION_KEY. For example, project_label.app. Project_label dimensions are case sensitive.

  • METRICS_TYPE: The types of metrics in your dataset. At least one metric type must be present.

    • Use the format metric.METRICS_TYPE. For example, metric.cost, metric.usage, metric.num_cameras.
    • Basic metrics (metric.cost, metric.usage, and metric.savings) are case insensitive, all other metric types are case sensitive.

  • Fields in the header row can be arranged in any order. The order is preserved when you preview the ingested CSV.

Warning

Make sure to sanitize your data, for example, mask personally identifiable information (PII), before sending it to DoiT.

Limitations​

  • When uploading a CSV file, the maximum size is 100MB. Large files take longer to process.

  • The CSV file must not contain records with timestamps (usage_date) older than one year.

  • Currently, the DoiT Cloud Navigator does not accept compressed files.

Example CSV​

Below is a simple example CSV:

Month,Territory,Cost
2024-03-01,AMER,$70000
2024-04-01,AMER,$70000
2024-05-01,AMER,$70000
2024-03-01,EMEA,$7500
2024-04-01,EMEA,$7500
2024-05-01,EMEA,$7500
2024-03-01,APAC,$3700
2024-04-01,APAC,$3700
2024-05-01,APAC,$3700

You can also download a sample CSV with more columns.

Upload CSV file​

Once you formatted the CSV file correctly, you can upload it when creating a new dataset or updating a dataset.

  1. In the Import data dialog, select Upload CSV file, and then select Continue.

    Import data options

  2. Prepare file: Make sure your CSV file complies to the Syntax and conventions. Select Next to continue.

  3. Upload CSV: Select the CSV file to upload. This step also validates the CSV.

    Upload CSV step 2

  4. Preview: In this step, you verify that the data will be ingested with the desired dimensions and metrics.

    Upload CSV step 2

After you select Confirm upload, DoiT Cloud Navigator will start processing the data. When the data has been uploaded successfully, it may take up to 15 minutes for the data to be available in the DoiT Cloud Navigator.

Last updated on