Amazon Redshift how to copy from s3 and set a job_id

By : gbegley

Amazon Redshift provides the ability to load table data from s3 objects using the "Copy" command. Is their a way to use the copy command, but also set additional "col=CONSTANT" for each inserted row.

I want to set a job_id (which is not in the source data) on each copied row, and I think it would be a shame to have to execute a few million inserts just so each row has a job attribute, when "copy" gets me 99% of the way there with much better performance.

Maybe there is a more clever solution?

By : gbegley


It seems there is no option to do post/pre processing with the COPY command itself. Therefore your best option seems to be to do preprocessing to the files you intend to COPY into Redshift, add the jobid and then load them into Redshift.

By : arikfr

This video can help you solving your question :)
By: admin