Output Data to Google Cloud Storage
Use GCS buckets as input to your pipeline or as an output target
A batch deployment pipeline supports connecting to Google Cloud Storage (GCS) to use as an input source or an output destination. Users can configure a GCS bucket to be read for input data and also have the option of writing their output to a GCS bucket.
Plainsight Pipelines do not support writing data to publicly accessible cloud buckets. Ensure that your GCS output bucket prevents public access. See this guide for more information:
To add a GCS bucket as an output destination:
Add output destinations under the Outputs tab.
- 1.Navigate to the "Outputs" tab of your batch deployment.
- 2.Click the "Add Output" button on the far right. If you do not see this option, ensure you are in "Edit" mode.
- 3.Select the Google Cloud Storage option.
Fill in the fields with the bucket's connection information:
- 1.Enter a name for your output in the Output Name field. This is how it will display in your list of sources. Example,
GCS output - 2.Under Output Data, check the "Include output images" checkbox to include the image data that was processed in your pipeline. By default, references to the input asset will be included. If there are image transformations, a signed URL to access the image will be provided.
- 3.Enter your GCS Bucket Name. This is the globally-unique name for your bucket, so be sure this is spelled correctly with no trailing spaces. Example,
my-images - 4.Enter an optional folder path in the Folder Name field. This folder must exist when the output is configured. Example:
- 5.Enable the "Private Bucket" toggle. This will allow you to upload a Service Account Key JSON File with credentials to access a private bucket. More info
Configure a GCS bucket as an output location for your pipeline.
Click "Save Output" at the bottom to add this destination to your pipeline. Each time your pipeline runs, it will output data to this bucket.
The pipeline will output a JSON file for each prediction as well as the processed image file (if "Include output images" was selected).
Example JSON output format with predictions for a bounding box model:
{
"source": {
"type": "gcs",
"bucket": "my-test-bucket",
"prefix": "images/more_cars",
"name": "00133.jpg",
"url": "https://storage.googleapis.com/my-test-bucket/images/more_cars/00133.jpg"
},
"output": {
"image": "gs://my-private-bucket/Output/721_cars/gcs/my-test-bucket/images/more_cars/00133.jpeg",
"prediction": {
"car": [{
"type": "rectangle",
"children": {},
"data": {
"min": [0.0, 0.2958550516764323],
"max": [0.9144365692138672, 0.9760575358072917]
},
"score": 0.9966015815734863
}, {
"type": "rectangle",
"children": {},
"data": {
"min": [0.855058822631836, 0.41045003255208334],
"max": [0.9956594848632813, 0.5383407084147135]
},
"score": 0.8756466507911682
}]
}
}
}
Removing an output destination will stop pipeline output from being written to that location.
Removing an output destination cannot be undone.
To remove an output destination:
- 1.Navigate to the "Outputs" tab.
- 2.Under the Connected Outputs table, find the output you want to remove and click "Remove" under the Actions column. If you do not see this option, ensure you are in "Edit" mode.
- 3.You will be prompted to confirm the deletion. Click "Remove Output" to permanently remove the output destination from your pipeline.
Click to remove this output destination.
Last modified 7mo ago