Output Data to Amazon S3
Use S3 buckets as input to your pipeline or as an output target
A batch deployment pipeline supports connecting to an Amazon S3 bucket to use as an input source or an output destination. Users can configure an S3 bucket to be synced for input data and also have the option of writing their output to an S3 bucket.
Plainsight Pipelines do not support writing data to publicly accessible cloud buckets. By default, S3 buckets block public access. See this guide for more information on blocking public access to your S3 storage:
Add output destinations under the Outputs tab.
To add an Amazon S3 bucket as an output source:
- 1.Navigate to the "Outputs" tab of your batch pipeline.
- 2.Click the "Add Output" button on the far right. If you do not see this option, ensure you are in "Edit" mode.
- 3.Select the Amazon S3 option
Fill in the fields with the bucket's connection information:
- 1.Enter a name for your output in the Output Name field. This is how it will display in your list of outputs. Example,
AWS output. - 2.Enter your Bucket Name. This is the globally-unique name for your bucket, so be sure this is spelled correctly with no trailing spaces. Example,
my-chess-images - 3.Enter an optional folder path in the Folder Path field. The folder must exist when the output is configured.
- 4.If your bucket is private you will need to authenticate it by entering an Access Key ID and a Secret Access Key in their respective fields.
- 5.Click "Save Output" at the bottom to add this destination to your pipeline. Each time your pipeline runs, it will output data to this bucket.
Configure an S3 bucket as an output destination for your pipeline.
The pipeline will output a JSON file for each prediction as well as the processed image file (if "Include output images" was selected).
Example JSON output format:
{
"source": {
"type": "gcs",
"bucket": "my-test-bucket",
"prefix": "images/more_cars",
"name": "00133.jpg",
"url": "https://storage.googleapis.com/my-test-bucket/images/more_cars/00133.jpg"
},
"output": {
"image": "gs://my-private-bucket/Output/721_cars/gcs/my-test-bucket/images/more_cars/00133.jpeg",
"prediction": {
"car": [{
"type": "rectangle",
"children": {},
"data": {
"min": [0.0, 0.2958550516764323],
"max": [0.9144365692138672, 0.9760575358072917]
},
"score": 0.9966015815734863
}, {
"type": "rectangle",
"children": {},
"data": {
"min": [0.855058822631836, 0.41045003255208334],
"max": [0.9956594848632813, 0.5383407084147135]
},
"score": 0.8756466507911682
}]
}
}
}
Removing an output destination will stop pipeline output from being written to that location.
Removing an output destination cannot be undone.
To remove an output destination:
- 1.Navigate to the "Outputs" tab.
- 2.Under the Connected Outputs table, find the output you want to remove and click "Remove" under the Actions column.
- 3.You will be prompted to confirm the deletion. Click "Remove Output" to permanently remove the output destination from your pipeline.
Click to remove this output destination.
Last modified 7mo ago