2D Semantic Segmentation
Learn how to create an example 2D semantic segmentation project using the KITTI open source dataset
Last updated
Learn how to create an example 2D semantic segmentation project using the KITTI open source dataset
Last updated
This quickstart will walk you through uploading data into Aquarium, starting with a standard open source dataset for a 2D semantic segmentation task.
Before you get started, there are also pages with some background and information around key concepts in Aquarium.
The main steps we will cover are:
Create a project within Aquarium
Upload labeled masks
Upload inference masks
By the end of this guide, you should have a good idea of how to upload your data into Aquarium and explore your semseg dataset!
To follow along with this Quickstart guide here are some things you'll need:
Download the quickstart dataset
The dataset contains the raw images, masks, and an end-to-end example upload script
Ensure you have installed the latest Aquarium client
pip install aquariumlearning
A development environment running a version of Python 3.6+
The aquariumlearning package requires Python >= 3.6.
Aquarium provides a python client library to simplify integration into your existing ML data workflows. In addition to wrapping API requests, it also handles common needs such as efficiently encoding uploaded data or using disk space to work with datasets larger than available system memory.
You can install and use the library using the following code block:
To get your API key, you can follow these instructions.
This quickstart leverages the KITTI open source dataset. This dataset contains images captured from from a car driving around a mid-size city with up to 15 cars and 30 pedestrians are visible per image.
For this 2D Semantic Segmentation, we'll be pixel level classification. The KITTI dataset has a lot of labeled classes, and a smaller amount of predicted classes. The predicted classes are:
road
sidewalk
building
wall
fence
pole
traffic light
traffic sign
vegetation
terrain
sky
person
rider
car
truck
bus
train
motorcycle
bicycle
You can download the quickstart dataset (334MB) at this link. Once downloaded and unzipped, you should be able to open the 2d_semseg_ingest.py file, add your API key, and run the file to upload data to your org! The quickstart folder will be formatted
There is a file in the zip named semseg_img_list.json, this file has a list of all the names of the images that are going to be uploaded and exists as a utility to make it easier to loop through data for upload.
There is a complete upload script included in the downloadable quickstart dataset as well as here in the docs. The following sections break off individual pieces of the process to discuss them in depth, but for a complete end to end example please refer to the whole script.
Projects are the highest level grouping in Aquarium and they allow us to:
Define a specific core task - in this case, segmentation of road images (2D Semantic Segmentation)
Define a specific ontology/class map
Hold multiple datasets for a given task
You can click here for more information on defining projects and best practices!
In the examples below, you'll see a reference to a ./classnames.json on line 40, this json is a simple list of the classnames we will be using in our project. That json file is available in the downloadable quickstart files.
For a semantic segmentation use case, the image you are segmenting is uploaded for reference and rendering.
In the code block below, we highlight the function .add_image() used to add both the original full quality image that is being segmented as well as the compressed version of that image. The compressed version of the image is uploaded with the preview_url parameter and allows for quicker and more efficient viewing of the image when using Aquarium.
The function .add_image() is called on an object of type LabeledFrame. We will cover this in more detail in the next sections. See the complete upload script at the bottom of this guide for an example of a semantic segmentation ingestion script.
The following code block highlights the .add_image() function:
Aquarium supports both image masks and numpy arrays to represent labeled data, where each pixel value is represented by a number that maps to a class id. For example, say your total segmentation model identifies 6 unique classes. Your image would be a greyscale image or a numpy array where each pixel would be represented by a value 0-5, where each numeric value maps to the corresponding order in which you defined your class map.
For the quickstart example, we leverage image masks as our labels! To upload the image masks, you first need to create a LabeledDataset object to represent all your labels. Then we create a Frame object that we add all of our labels and inferences too. We'll break down the code in more detail below, but want to point out normally when uploading (and like you'll see in the completed upload script) data is normally uploaded in some kind of loop:
This is an example of the greyscale image that is being uploaded to Aquarium:
Now that we have created a Project and a LabeledDataset, let's also upload those model inferences. Inferences, like labels, must be matched to a frame. For each labeled frame in your dataset, we will create an inference frame and then assign the appropriate inferences to that inference frame.
The inferences just like labels can be represented as a greyscale image or a numpy array. Same concept apply to inferences as we discussed above for labels.
Creating InferencesFrame objects and adding inferences will look very similar to creating LabeledFrames and adding labels to them.
Important Things To Note:
Each InferencesFrame must exactly match to a LabeledFrame in the dataset. This is accomplished by ensuring the frame_id property is the same between corresponding LabeledFrames and InferencesFrames.
It is possible to assign inferences to only a subset of frames within the overall dataset (e.g. just the test set).
Breaking down some of what you'll see in the provided ingest script:
At this point we have created a project in Aquarium, and uploaded our labels and inferences. The data has been properly formatted, but now as our final step, let's use the client to actually upload the data to our project!
Now that we have the datasets, using the client we can upload the data:
With the code snippet above your data will start uploading! Now we can monitor the status within the UI!
When you start an upload, Aquarium performs some crucial tasks like indexing metadata and generating embeddings for dataset so it may take a little bit of time before you can fully view your dataset. You can monitor the status of your upload in the application as well as your console after running your upload script. To view your upload status, log into Aquarium and click on your newly created Project. Then navigate to the tab that says "Streaming Uploads" where you can view the status of your dataset uploads.
Once your upload is completed under the "Datasets" tab, you'll see a view like this:
And congrats!! You've uploaded your data into Aquarium! You're now ready to start exploring your data in the application!
Putting it all together here is the entire script you can use to replicate this project. You can download this script and the necessary data here. If you need help getting your API key, you can follow these instructions.
Now that you have uploaded data, time to explore and understand your data better using Aquarium!
