There are two types of links you can create depending on your use case:
Important notes:
The maximum file size is 1GB.
If you want to have a dataset that is more than 1GB, use the Get Create Append URL and ensure each file uploaded is less than 1GB.
The pre-signed url generated will expire after 1 hour
If you're replacing an existing dataset completely, use the Get Create Upload URL method
If you are appending rows to an existing dataset, use the Get Create Append URL method.
Please ensure the column data types and names are all the same and no new columns have been added. This append API only supports appending rows to an existing dataset and not appending columns
You'll need to make a direct PUT request to the S3 pre-signed url returned in the response of the previous step to upload your CSV file.
The url should start with https://s3.ap-southeast-1.amazonaws.com/
For example: https://s3.ap-southeast-1.amazonaws.com/attachments.data.gov.sg/c/6047/private/d_2a6070fa301695904e1a626434189e59/VW50aXRsZWQgRGF0YXNldA/d_2a6070fa301695904e1a626434189e59.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAU7LWPY2WH4V4RGEJ%2F20231129%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Date=20231129T053810Z&X-Amz-Expires=3600&...
Javascript
Python
It’s important to ensure that you are sending a PUT request with only the file data. While POST-ing form-data is allowed, the file received will not be the same and may cause errors.
Common issues:
Check that you're using a PUT request
Check that you're using the generated S3 pre-signed url, instead of the one above
Set the header as Content-Type: text/csv when uploading the CSV file
The pre-signed URLs expire after 1 hour, so you'd need to generate them again
The dataset validation and ingestion process has been started.
2
Pending Validation
The dataset has been queued for validation.
2.1
Validation Failed
The dataset contains invalid values causing validation failure. Please review the reported errors, make the necessary changes and try again.
2.2
Validation Passed
The dataset has been validated successfully, but not yet ingested.
3
Pending Ingestion
The dataset has been queued for ingestion.
3.1
Ingestion Failed
The dataset failed to be ingested. Please try again later.
3.2
Ingestion Success
The dataset has been ingested successfully.
The validation and ingestion process of data.gov.sg is asynchronous. The upload request from the previous section will return before the process is complete. In order to get the status for validation and ingestion, manual polling is required.
Here are the possible validation and ingestion statuses:
Order
Status
Description
1
Pending
The validation and ingestion process has been started
2
Pending Validation
The dataset has been queued for validation
2.1
Validation Failed
The dataset contains invalid values causing validation failures. Please review the reported errors, make the necessary changes and try again.
2.2
Validation Passed
The dataset has been validated successfully, but not yet ingested.
3
Pending Ingestion
The dataset has been queued for ingestion.
3.1
Ingestion Failed
The dataset failed to be ingested. Please try again later.
3.2
Ingestion Success
The dataset has been ingested successfully.
Common questions:
What happens when there are concurrent writes?
If there is an existing dataset upload that has not completed, subsequent dataset uploads will automatically fail. This is to prevent multiple writes happening at the same time, that will cause unexpected behaviour
To prevent this, poll this endpoint and wait for Ingestion Success status before uploading another file
What happens if there is one or more rows that fail validation?
We currently take the conservative approach and will reject the dataset upload if any of the rows fail validation. In these cases, we recommend to fix the dataset before retrying.
The API will not return detailed explanations on the validation error. For further debugging and investigation, please try uploading the dataset via the UI, which will show more explicit error messages to help with debugging.
