data.gov.sg
  • Introduction to data.gov.sg
  • 🆕What's new
    • Release Notes: Feburary 2025
    • Release Notes: October 2024
    • Release Notes: September 2024
  • USER GUIDE
    • Content quality guidelines
      • Data types
      • Data principles
    • For data consumers
      • How to search for data
      • How to raise a data request
      • How to embed the dataset explorer on your site
    • How to search for real-time APIs
    • For data owners
      • How to onboard as an admin
      • How to login
      • How to publish data
      • How to unpublish and/or delete datasets
      • How to edit datasets
      • How to generate API keys
      • Write APIs
  • DEVELOPER GUIDE
    • API overview
    • Collection APIs
    • Dataset APIs
Powered by GitBook
On this page

Was this helpful?

  1. USER GUIDE
  2. For data owners

Write APIs

PreviousHow to generate API keysNextAPI overview

Last updated 4 months ago

Was this helpful?

Government Admin users can use Write APIs to automatically and programmatically publish data to data.gov.sg without needing to log into the UI


Prerequisites

  • You must be a government officer

  • Your email has admin permissions to manage datasets

Step 1 - Generate API Keys:

  • Create an API key from the .

  • Store the API key in a secure location. It will be used later.

  • Step by step details can be found here: How to generate API keys

Step 2 - Verify API Connectivity:

Run a get request against the above endpoint, using your admin api key in the header:

A 200 response means that you're connected.

Common issues:

  1. Check your API URL - did you use the correct URL above?

  2. Check your API key - did you use the correct API key? Try regenerating the API key and testing again

  3. Are the network configurations set up to connect to the internet? data.gov.sg is an internet platform and require internet connectivity

Step 3 - Publish Dataset:

  • To push data, an existing dataset must exist. The dataset can be created from the admin dashboard.

  • The dataset must already contain existing data.

  • The dataset must be published.

  • For more details on publishing data: How to publish data

Using Write APIs involves three steps:

  • Generating upload link

  • Uploading file to link

  • Polling for upload status

Step 4 - Create Upload / Append URL:

If you are creating a new dataset or replacing an existing dataset completely, then generate an upload link:

If you are appending rows to an existing dataset, then generate an append link. Please ensure the column data types and names are all the same and no new columns have been added. This append API only supports appending rows to an existing dataset and not appending columns:

Step 5 - Upload Tabular File:

It’s important to ensure that you are sending a PUT request with only the file data. While POST-ing form-data is allowed, the file received will not be the same and may cause errors.

Common issues:

  1. Check that you're using a PUT request

  2. Check that you're using the generated upload url instead of the one provided above

Step 6 - Get Latest Ingestion Status:

The validation and ingestion process of DataGovSG is asynchronous and long running. The upload request from the previous section will return before the process is complete. In order to get the status for validation and ingestion, manual polling is required.

Here are the possible validation and ingestion statuses:

Order
Status
Description

1

Pending

The validation and ingestion process has been started

2

Pending Validation

The dataset has been queued for validation

2.1

Validation Failed

The dataset contains invalid values causing validation failures. Please review the reported errors, make the necessary changes and try again.

2.2

Validation Passed

The dataset has been validated successfully, but not yet ingested.

3

Pending Ingestion

The dataset has been queued for ingestion.

3.1

Ingestion Failed

The dataset failed to be ingested. Please try again later.

3.2

Ingestion Success

The dataset has been ingested successfully.

Raw API Schema:

Updated as of 20th June 2024

Find the datasetId within the dataset url For example, for this url: The datasetId is the following: d_07c63be0f37e6e59c07a4ddc2fd87fcb Hence, the endpoint used would be:

For further queries feel free to reach out to us:

https://api-production.data.gov.sg/v2/admin/api/auth/whoami
https://beta.data.gov.sg/datasets/d_07c63be0f37e6e59c07a4ddc2fd87fcb/view
https://api-production.data.gov.sg/v2/admin/api/datasets/d_07c63be0f37e6e59c07a4ddc2fd87fcb/upload-link
Contact the Data.gov.sg team
admin dashboard
16KB
v2-write-api-specification.yml

Who Am I

get
Header parameters
x-dgs-admin-api-keystringOptionalExample: your-api-key
Responses
200
Successful response
application/json
Responseobject
get
GET /v2/admin/api/auth/whoami HTTP/1.1
Host: api-production.data.gov.sg
Accept: */*
200

Successful response

{
  "user": {
    "userId": "167",
    "email": "admin@data.gov.sg"
  }
}
  • Prerequisites
  • Step 1 - Generate API Keys:
  • Step 2 - Verify API Connectivity:
  • GETWho Am I
  • Step 3 - Publish Dataset:
  • Step 4 - Create Upload / Append URL:
  • GETGet Create Upload URL
  • GETGet Create Append URL
  • Step 5 - Upload Tabular File:
  • PUTUpload Tabular File
  • Step 6 - Get Latest Ingestion Status:
  • GETGet Latest Ingestion Status
  • Raw API Schema:

Get Create Append URL

get
Path parameters
datasetIdstringRequiredExample: d_263ee33e4d309a0bba75a1fd30ab7035
Query parameters
md5stringOptional

(Optional) MD5 hash of the file to be uploaded. The MD5 Hash should be encoded in base64. The base64, MD5 hash should then be URI encoded.

Example: FkXeisMYBP6UpdSiY8kOsA%3D%3D
Header parameters
x-dgs-admin-api-keystringOptionalExample: your-api-key
Responses
201
Created
application/json
Responseobject
get
201

Created

Get Create Upload URL

get
Path parameters
datasetIdstringRequiredExample: d_263ee33e4d309a0bba75a1fd30ab7035
Header parameters
x-dgs-admin-api-keystringOptionalExample: your-api-key
Responses
201
Created
application/json
Responseobject
get
201

Created

Get Latest Ingestion Status

get
Path parameters
datasetIdstringRequiredExample: d_263ee33e4d309a0bba75a1fd30ab7035
Header parameters
x-dgs-admin-api-keystringOptionalExample: your-api-key
Responses
200
**Legend** | Order | Ingestion status | Description | | ----- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | 1 | Pending | The dataset validation and ingestion process has been started. | | 2 | Pending Validation | The dataset has been queued for validation. | | 2.1 | Validation Failed | The dataset contains invalid values causing validation failure. Please review the reported errors, make the necessary changes and try again. | | 2.2 | Validation Passed | The dataset has been validated successfully, but not yet ingested. | | 3 | Pending Ingestion | The dataset has been queued for ingestion. | | 3.1 | Ingestion Failed | The dataset failed to be ingested. Please try again later. | | 3.2 | Ingestion Success | The dataset has been ingested successfully. |
application/json
Responseobject
get
200

Legend

Order Ingestion status Description
1 Pending The dataset validation and ingestion process has been started.
2 Pending Validation The dataset has been queued for validation.
2.1 Validation Failed The dataset contains invalid values causing validation failure. Please review the reported errors, make the necessary changes and try again.
2.2 Validation Passed The dataset has been validated successfully, but not yet ingested.
3 Pending Ingestion The dataset has been queued for ingestion.
3.1 Ingestion Failed The dataset failed to be ingested. Please try again later.
3.2 Ingestion Success The dataset has been ingested successfully.
GET /v2/admin/api/datasets/{datasetId}/append-link HTTP/1.1
Host: api-production.data.gov.sg
Accept: */*
{
  "url": "https://s3.ap-southeast-1.amazonaws.com/attachments.data.gov.sg/c/6047/private/d_2a6070fa301695904e1a626434189e59/VW50aXRsZWQgRGF0YXNldA/d_2a6070fa301695904e1a626434189e59.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAU7LWPY2WH4V4RGEJ%2F20231129%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Date=20231129T053810Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHYaDmFwLXNvdXRoZWFzdC0xIkYwRAIgdQy7ZfOZaDbQIEsS5VqAd76mSGitIgNSHxm3C6sP9SwCIFS5kaaPMcCYQpVv9UdgGzwLZyujx1S844oiCHZxT3yhKqcDCM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQBBoMMzQyMjM1MjY4NzgwIgxKiGm5wh31GOszXoIq%2BwLwXu11Ykcov8rg20fsos4%2FcGfR8JtKuUvTtjKC4IBP46v5FzBKPIlMT%2ByfrMvOYZV4kqSiN5%2F%2FeOuGMwk6tfNGY0PphSWfTnWCEVT5wap%2BFRsW7OTPFy9G3ih0X1h7NDYpK%2F1Jkh9pM3%2FyW4x7FQUnXaptNP3k%2FWArdBG3kIvH2kaOiBU9TlqvRhP0xd7dEXoqARY%2BZiOIFBoA2WknNPxQU82mzavOFj9hmeSyi5rvsFa%2BG47m02g6F46vIicnDzQSDgO7UAusH9pQPO40jXvspYAYknW7BQ6EnMcUL61fUWPCaS8zaLdvRNnb6gisasL8RC6gJsJeAPLRhijUxD8KsVDWtt3ZBho7YR5tQbdg%2FAIVmPL4xQe%2B05N9dHt5wG9ws8A2789F3SgF0yZW9tzc5IfNIxsixotOrLkZRUBXmzujAVe51sQM98dx93Y2%2B%2F9mGbQY7700ShGjAyErYvdl2CYJutJ9jI1yswaJkFyR8MkxmSf26vUY7Of0MMCcm6sGOp4BNi7cp9OFj6enS%2Fn7jhlhdFbkn4zBa%2F0e4ywK05Vqzr7mRguUmMawsUh9i%2Booc%2Bnn90PA7C6YvzZ3IBxyJLWxmzkRDpdlThbkQwxpREJ2c%2FzaPNPdUTrDvp28qAPV3bUe%2FWJTKY7Lh72U2AIRFlKUoTCpQ6ihvgIZH1GGPJsomcajXaguK%2F5yLAqFxY%2B7org%2FJlFwMXfZUSzuObK%2BHsE%3D&X-Amz-Signature=c3444a1b8e9d73d8ac7fd64da4a963860a3e9353ae956f08687a78640dedb834&X-Amz-SignedHeaders=host&x-id=PutObject"
}
GET /v2/admin/api/datasets/{datasetId}/upload-link HTTP/1.1
Host: api-production.data.gov.sg
Accept: */*
{
  "url": "https://s3.ap-southeast-1.amazonaws.com/attachments.data.gov.sg/c/6047/private/d_2a6070fa301695904e1a626434189e59/VW50aXRsZWQgRGF0YXNldA/d_2a6070fa301695904e1a626434189e59.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAU7LWPY2WH4V4RGEJ%2F20231129%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Date=20231129T053810Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHYaDmFwLXNvdXRoZWFzdC0xIkYwRAIgdQy7ZfOZaDbQIEsS5VqAd76mSGitIgNSHxm3C6sP9SwCIFS5kaaPMcCYQpVv9UdgGzwLZyujx1S844oiCHZxT3yhKqcDCM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQBBoMMzQyMjM1MjY4NzgwIgxKiGm5wh31GOszXoIq%2BwLwXu11Ykcov8rg20fsos4%2FcGfR8JtKuUvTtjKC4IBP46v5FzBKPIlMT%2ByfrMvOYZV4kqSiN5%2F%2FeOuGMwk6tfNGY0PphSWfTnWCEVT5wap%2BFRsW7OTPFy9G3ih0X1h7NDYpK%2F1Jkh9pM3%2FyW4x7FQUnXaptNP3k%2FWArdBG3kIvH2kaOiBU9TlqvRhP0xd7dEXoqARY%2BZiOIFBoA2WknNPxQU82mzavOFj9hmeSyi5rvsFa%2BG47m02g6F46vIicnDzQSDgO7UAusH9pQPO40jXvspYAYknW7BQ6EnMcUL61fUWPCaS8zaLdvRNnb6gisasL8RC6gJsJeAPLRhijUxD8KsVDWtt3ZBho7YR5tQbdg%2FAIVmPL4xQe%2B05N9dHt5wG9ws8A2789F3SgF0yZW9tzc5IfNIxsixotOrLkZRUBXmzujAVe51sQM98dx93Y2%2B%2F9mGbQY7700ShGjAyErYvdl2CYJutJ9jI1yswaJkFyR8MkxmSf26vUY7Of0MMCcm6sGOp4BNi7cp9OFj6enS%2Fn7jhlhdFbkn4zBa%2F0e4ywK05Vqzr7mRguUmMawsUh9i%2Booc%2Bnn90PA7C6YvzZ3IBxyJLWxmzkRDpdlThbkQwxpREJ2c%2FzaPNPdUTrDvp28qAPV3bUe%2FWJTKY7Lh72U2AIRFlKUoTCpQ6ihvgIZH1GGPJsomcajXaguK%2F5yLAqFxY%2B7org%2FJlFwMXfZUSzuObK%2BHsE%3D&X-Amz-Signature=c3444a1b8e9d73d8ac7fd64da4a963860a3e9353ae956f08687a78640dedb834&X-Amz-SignedHeaders=host&x-id=PutObject"
}

Upload Tabular File

put

Note that this uses the URL provided the response of "Upload Dataset Data Step 1"

Query parameters
X-Amz-AlgorithmstringOptionalExample: AWS4-HMAC-SHA256
X-Amz-Content-Sha256stringOptionalExample: UNSIGNED-PAYLOAD
X-Amz-CredentialstringOptionalExample: ASIARQYHRAAPWDRCHVCT%2F20231020%2Fap-southeast-1%2Fs3%2Faws4_request
X-Amz-DatestringOptionalExample: 20231020T064001Z
X-Amz-ExpiresintegerOptionalExample: 3600
X-Amz-Security-TokenstringOptionalExample: IQoJb3JpZ2luX2VjELf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDmFwLXNvdXRoZWFzdC0xIkcwRQIhAMVvV07i01mD2JdX6WJvSY3G1pj5iKXWhdqd4cf9THaCAiB0BiBDhrsgZ3PEqRYrwsvjE3h%2Bb2nWfPfoYgsnsKf%2FZiqdAwjQ%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAAaDDEwNDcwNTYyMjA0NyIMYieEEU5FGLr%2BCxjxKvECiy6xz1X1ftvSEdIhxpqa0Ok41TMEZJgYNqg6%2FPzoz4c%2FWs7oJWn%2FPXsgrRNJt7lFQ%2F0tcPSGa9osWYau0y6buIyz3cVFgcpa%2BSlte9B5iQkcpLltI9W6wlw1BZ%2BjNrsjtkMPV27GGiLoz8gfKCelITJHHuzR0RE55a9CIeJzLXPOwikDzvKPzFYL3QMH7T4luXXYyPg50hBIyA8xBwKurI4WDOWTI7YqfjsP%2F6VGhbpcUg9V49OCYZWh60xGs0yNUC7r5uwPdK2SerDaD0bMdp8ugrc8wg%2B6guDBUOyRLPKTKgObVjC1eb%2BhMIGiPrhiOUTsKjgf5qmRvy%2B7oVNImtmxH%2FBvyo%2BIyW0lDW%2BQ31oLv97wEZdHPLMwp4bNrFcAACRk16scfaOsqjFTKdPb3Ue9VmIZ0cpNZKe2E9rJ6tmeODWdSxvix1rkc8DZ61SD7tKdw2pQExs16HTWmYnEFmxiZBf275yVi9om8h%2B4kgLNMLzByKkGOp0BqHjxUEHQcSlTogZxqnD9Tq1z5ogXoSNjSTOaF3wUy6zuoUiBGOHgVD4XIfUKL0KdO1Rk1XDcG4RpPYaahGahBb7dmM51ZXpx7ZoN%2FVK1tKesZ8B7MwOKibsBAJ4UGuaqizSGLpfLDnGxJ5RvRxwei7JRXpaya7pcggEGPQhk3QkU9JZTaEWnTqr7bAF%2FTR8f9x%2BH0A%2BC1KMDxw9mBQ%3D%3D
X-Amz-SignaturestringOptionalExample: 039ec79400db61bdbb858945438f0617bfeb56a7a8fcd968859afd287e518efa
X-Amz-SignedHeadersstringOptionalExample: host
x-idstringOptionalExample: PutObject
Body
stringOptionalExample: "name,title,email\njanedoe,SWE,jane@open.gov.sg"
Responses
200
OK
text/plain
Responsestring
put
PUT /attachments.data.gov.sg/c/6047/private/d_2a6070fa301695904e1a626434189e59/VW50aXRsZWQgRGF0YXNldA/d_2a6070fa301695904e1a626434189e59.csv HTTP/1.1
Host: api-production.data.gov.sg
Content-Type: */*
Accept: */*
Content-Length: 53

"\"name,title,email\\njanedoe,SWE,jane@open.gov.sg\""
200

OK

text
GET /v2/admin/api/datasets/{datasetId}/ingestion-status HTTP/1.1
Host: api-production.data.gov.sg
Accept: */*
{
  "version": 0,
  "userId": 1,
  "updatedAt": 1699953464415,
  "status": "NA",
  "failures": null
}