2D Object Detection

Learn how to create an example 2D object detection project using the RarePlanes open source dataset

Overview

This quickstart will walk you through uploading data into Aquarium, starting with a standard open source dataset for a 2D object detection task.

Before you get started, there are also pages with some background and information around key concepts in Aquarium.

The main steps we will cover are:

• Create a project within Aquarium

• Upload labeled data

• Upload inference data

By the end of this guide, you should have a good idea of how to upload your data into Aquarium and explore a dataset full of satellite images of runways and planes!

Example data loaded into Aquarium from the RarePlanes dataset

Prerequisites

To follow along with this Quickstart guide here are some things you'll need:

• Download the quickstart dataset

  • The dataset contains the raw images, data, and an end-to-end example upload script

• Ensure you have installed the latest Aquarium client

  • pip install aquariumlearning

• A development environment running a version of Python 3.6+

• API Docs for reference

RarePlanes Dataset Ingestion

This quickstart uses a subset of an open sources This quickstart uses a subset of an open source dataset called RarePlanes, sourced from CosmiqWorks. This dataset contains satellite imagery and aircraft labels.

For this 2D Object Detection task, we'll be detecting the aircraft and classifying them by wing type. There are four possible wing type classes/categories:

• straight

• delta

• swept

• variable-swept

You can download the quickstart dataset (334MB) at this link.

# Overall data structure
├── images
│   ├── 1_104005000FDC8D00_tile_8.png
│   ├── 1_104005000FDC8D00_tile_13.png
│   └── 1_104005000FDC8D00_tile_14.png
├── inferences.json
└── labels.json
└── classnames.json

# All images are mirrored online at
# https://storage.googleapis.com/aquarium-public/quickstart/rareplanes/imgs/<filename>.jpg

# Format of labels.json
[
    {
        "image_name": "114_1040010049AD0900_tile_153.png",
        "image_url": "https://storage.googleapis.com/aquarium-public/datasets/rareplanes/train/PS-RGB_tiled/114_1040010049AD0900_tile_153.png",
        "image_id": 1166,
        "detected_plane_data": [
            {
                "bbox": [
                    160.64238835126162,
                    67.3610782045871,
                    181.12383367866278,
                    113.94228099286556
                ],
                "plane_details": {
                    "area": 911.7884419186188,
                    "area_pixels": 10964.798453198293,
                    "canards": "no",
                    "faa_wingspan_class": 4.0,
                    "id": 13086,
                    "image_id": 1166,
                    "is_plane": 1.0,
                    "length": 48.12729927156121,
                    "loc_id": 114.0,
                    "location": "Richmond International Airport, 1, Richard E Byrd Terminal Drive, Henrico, Henrico County, Virginia, 23250, USA",
                    "new_area": 911.7884419186188,
                    "num_engines": 2.0,
                    "num_tail_fins": 1.0,
                    "propulsion": "jet",
                    "role": "Large Civil Transport/Utility",
                    "role_id": 3.0,
                    "segmentation": [
                        [
                            160.64238835126162,
                            104.35532528162003,
                            284.9430585205555,
                            67.3610782045871,
                            341.7662220299244,
                            141.3495723567903,
                            250.02048928290606,
                            181.30335919745266,
                            160.64238835126162,
                            104.35532528162003
                        ]
                    ],
                    "wing_position": "mid/low mounted",
                    "wing_type": "swept",
                    "wingspan": 37.89318834269945
                }
            }
        ]
    },
    ...
]

# Format of inferences.json
[
       {
        "image_id": 1209,
        "image_url": "https://storage.googleapis.com/aquarium-public/datasets/rareplanes/train/PS-RGB_tiled/4_1040010019345B00_tile_827.png",
        "image_name": "4_1040010019345B00_tile_827.png",
        "predictions": [
            {
                "bbox": [
                    422.0789489746094,
                    29.705419540405273,
                    512.0,
                    118.0540771484375
                ],
                "confidence": 0.9869634509086609,
                "category_name": "swept"
            }
        ],
    },
        ...
      ]
    },
    ...
]

Uploading the Data

Python Client Library

The aquariumlearning package requires Python >= 3.6.

Aquarium provides a python client library to simplify integration into your existing ML data workflows. In addition to wrapping API requests, it also handles common needs such as efficiently encoding uploaded data or using disk space to work with datasets larger than available system memory.

You can install and use the library using the following code block:

!pip install aquariumlearning

import aquariumlearning as al
al_client = al.Client()
al_client.set_credentials(api_key=YOUR_API_KEY)

To get your API key, you can follow these instructions.

Projects

Projects are the highest level grouping in Aquarium and they allow us to:

• Define a specific core task - in this case, pet breed detection (2D Classification)

• Define a specific ontology

• Hold multiple datasets for a given task/ontology

You can click here for more information on defining projects and best practices!

In the examples below, you'll see a reference to a ./classnames.json on line 11, this json is a simple list of the classnames we will be using in our project. We have example files available to download for each quickstart project.

import string
import random
import json

# Project names have to be globally unique in Aquarium
PROJECT_NAME = 'Rareplanes_Quickstart'

with open('./classnames.json') as f:
    classnames = json.load(f)

# primary_task field defines what kind of ML task this project will be
al_client.create_project(
    PROJECT_NAME, 
    al.LabelClassMap.from_classnames(classnames), 
    primary_task="2D_OBJECT_DETECTION"
)

Labeled Datasets

Often just called "datasets" for short, these are versioned snapshots of input data and ground truth labels. They belong to a Project, and consist of multiple LabeledFrames.

In most cases a Frame is a logical grouping of an image and its structured metadata. In more advanced cases, a frame may include more than one image (context imagery, fused sensors, etc.) and additional metadata. Now let's create our LabeledDataset object and add the LabeledFrames to it:

# read in the label data
with open('./labels.json') as f:
    label_entries = json.load(f)

# define your Labeled Dataset
labeled_dataset = al.LabeledDataset()

# loop through your labeled data to created a Labeled Frame
for entry in label_entries:
    # example file name 1_104005000FDC8D00_tile_8.png
    # Create a frame object, using the filename as an id
    frame_id = entry['img_name'].split('.png')[0]
    frame = al.LabeledFrame(frame_id=frame_id)

    # Add an image to the frame
    image_url = entry['image_url']
    frame.add_image(image_url=image_url)

    # for each frame, there can be multiple detected labels
    # which in this case means multiple detected planes
    for idx, identified_plane in enumerate(entry['detected_planes']):
        # create a unique id for each label
        label_id = frame_id + '_gt_' + str(idx)

        # bbox needs to be uploaded in top, left, width, height format
        # usr_attrs adds metadata to each label
        # top, left, width, height in pixels
        frame.add_label_2d_bbox(
            label_id=label_id,
            classification='plane',
            top = identified_plane['bbox'][1],
            left = identified_plane['bbox'][0],
            width = identified_plane['bbox'][2],
            height = identified_plane['bbox'][3],
            user_attrs= identified_plane['plane_details']
        )

    # Add the frame to the dataset collection
    labeled_dataset.add_frame(frame)

Inferences

Now that we have created a Project and a LabeledDataset, let's also upload those model inferences. Inferences, like labels, must be matched to a frame within the dataset. For each LabeledFrame in your dataset, we will create an InferencesFrame and then assign the appropriate inferences to that InferencesFrame.

Creating InferencesFrame objects and adding inferences will look very similar to creating LabeledFrames and adding labels to them.

We then add the InferencesFrame to an Inferences object and then upload the Inferences object into Aquarium!

Important Things To Note:

• Each InferencesFrame must exactly match to a LabeledFrame in the dataset. This is accomplished by ensuring the frame_id property is the same between corresponding LabeledFrames and InferencesFrames.

• It is possible to assign inferences to only a subset of frames within the overall dataset (e.g. just the test set).

# read in inference data
with open('inferences.json') as f:
    inference_entries = json.load(f)

# define Inference Dataset
inference_dataset = al.Inferences()
for entry in inference_entries:
    # Create a frame object, using the same id as the label
    frame_id = entry['image_name'].split('.png')[0]
    inf_frame = al.InferencesFrame(frame_id=frame_id)

    # for each image, we need to loop through each predicted plane
    for idx, identified_plane in entry['predictions']:   
        x1 = identified_plane['bbox'][0]
        y1 = identified_plane['bbox'][1]
        x2 = identified_plane['bbox'][2]
        y2 = identified_plane['bbox'][3]

        # inference data formats bbox as [x1,y1,x2,y2] 
        # aquarium needs top, left, width, and height
        # so we need to do a little bit of math
        width = abs(x1 - x2)
        height = abs(y1 - y2)

        # Add the inferred classification label to the frame
        inf_label_id = frame_id + '_inf_' + idx.toString()

        # we add a 2D inference bounding box object to each inference frame
        inf_frame.add_inference_2d_bbox(
            label_id=inf_label_id,
            classification=identified_plane['category_id'],
            confidence=identified_plane['confidence'],
            top = y1,
            left = x1,
            width = width,
            height = height
        )
    
    # Add the frame to the inferences collection
    inference_dataset.add_frame(inf_frame)

At this point we have created a project in Aquarium, and uploaded our labels and inferences. The data has been properly formatted, but now as our final step, let's use the client to actually upload the data to our project!

Submit the Datasets!

Now that we have the datasets, using the client we can upload the data:

# the name of our Labeled Dataset that will show up in Aquarium
LABELED_DATASET_NAME = 'rareplanes_labels'

# now let's upload our labeled dataset to Aquarium
al_client.create_dataset(
    PROJECT_NAME, 
    LABELED_DATASET_NAME, 
    dataset=labeled_dataset
)

# the name of our Inference Dataset that will show up in Aquarium
INFERENCE_DATASET_NAME = 'rareplanes_inferences'

# now let's upload the inferences to the Aquarium project
al_client.create_inferences(
    PROJECT_NAME, 
    LABELED_DATASET_NAME, 
    inferences=inference_dataset, 
    inferences_id=INFERENCE_DATASET_NAME
)

With the code snippet above your data will start uploading! Now we can monitor the status within the UI!

Monitoring Your Upload

When you start an upload, Aquarium performs some crucial tasks like indexing metadata and generating embeddings for dataset so it may take a little bit of time before you can fully view your dataset. You can monitor the status of your upload in the application as well as your console after running your upload script. To view your upload status, log into Aquarium and click on your newly created Project. Then navigate to the tab that says "Streaming Uploads" where you can view the status of your dataset uploads.

Once your upload is completed under the "Datasets" tab, you'll see a view like this:

Your uploaded project complete with labeled data displayed on the left, and your inference data on the right

And congrats!! You've uploaded your data into Aquarium! You're now ready to start exploring your data in the application!

Completed Upload Example Script

Putting it all together here is the entire script you can use to replicate this project. You can download this script and the necessary data here. If you need help getting your API key, you can follow these instructions.

#!pip install aquariumlearning

import aquariumlearning as al
al_client = al.Client()
al_client.set_credentials(api_key=YOUR_API_KEY)

import string
import random
import json

# defining names for project and the datasets as they'll show up in Aquarium
# project names must be unique, and within a project, datasets must also have unique names
PROJECT_NAME = 'Rareplanes_Quickstart'
LABELED_DATASET_NAME = 'rareplanes_labels'
INFERENCE_DATASET_NAME = 'rareplanes_inferences'

# define the filepaths we'll be working with
classnames_filepath = '2D_Rareplanes/classnames.json'
labels_filepath = '2D_Rareplanes/labels.json'
inferences_filepath = '2D_Rareplanes/inferences.json'

# load in your classname file
with open(classnames_filepath) as f:
    classnames = json.load(f)

# load in your label data
with open(labels_filepath) as f:
    label_entries = json.load(f)

# load in your inference data
with open(inferences_filepath) as f:
    inference_entries = json.load(f)

# primary_task field defines what kind of ML task this project will be
al_client.create_project(
    PROJECT_NAME, 
    al.LabelClassMap.from_classnames(classnames), 
    primary_task="2D_OBJECT_DETECTION"
)

# defines our dataset we will be uploading
label_dataset = al.LabeledDataset()

# loop through each of our labels
for entry in label_entries:
    # example file name 1_104005000FDC8D00_tile_8.png
    # Create a frame object, using the filename as an id
    frame_id = entry['image_name'].split('.png')[0]
    frame = al.LabeledFrame(frame_id=frame_id)

    # Add an image to the frame
    image_url = entry['image_url']
    frame.add_image(image_url=image_url)

    # for each frame, there can be multiple detected labels
    # which in this case means multiple detected planes
    for idx, identified_plane in enumerate(entry['detected_planes']):
        # create a unique id for each label
        label_id = frame_id + '_gt_' + str(idx)

        # bbox needs to be uploaded in top, left, width, height format
        # usr_attrs adds metadata to each label
        frame.add_label_2d_bbox(
            label_id=label_id,
            classification=identified_plane['category_name'],
            top = identified_plane['bbox'][1],
            left = identified_plane['bbox'][0],
            width = identified_plane['bbox'][2],
            height = identified_plane['bbox'][3],
            user_attrs= identified_plane['plane_details']
        )

    # Add the frame to the dataset collection
    label_dataset.add_frame(frame)

# create our inference dataset we will be uploading
inference_dataset = al.Inferences()

# loop through all the inferences we have
for entry in inference_entries:
    # Create a frame object, using the same id as the label
    frame_id = entry['image_name'].split('.png')[0]
    inf_frame = al.InferencesFrame(frame_id=frame_id)

    # for each image, we need to loop through each predicted plane
    for idx, identified_plane in enumerate(entry['predictions']):   
        x1 = identified_plane['bbox'][0]
        y1 = identified_plane['bbox'][1]
        x2 = identified_plane['bbox'][2]
        y2 = identified_plane['bbox'][3]

        # inference data formats bbox as [x1,y1,x2,y2] 
        # aquarium needs top, left, width, and height
        # so we need to do a little bit of math
        width = abs(x1 - x2)
        height = abs(y1 - y2)

        # Add the inferred classification label to the frame
        inf_label_id = frame_id + '_inf_' + str(idx)

        # we add a 2D inference bounding box object to each inference frame
        inf_frame.add_inference_2d_bbox(
            label_id=inf_label_id,
            classification=identified_plane['category_name'],
            confidence=identified_plane['confidence'],
            top = y1,
            left = x1,
            width = width,
            height = height
        )
    
    # Add the frame to the inferences collection
    inference_dataset.add_frame(inf_frame)

# now let's upload our labeled dataset to Aquarium
al_client.create_dataset(
    PROJECT_NAME, 
    LABELED_DATASET_NAME, 
    dataset=label_dataset
)

# now let's upload the inferences to the Aquarium project
al_client.create_inferences(
    PROJECT_NAME, 
    LABELED_DATASET_NAME, 
    inferences=inference_dataset, 
    inferences_id=INFERENCE_DATASET_NAME
)

What Now?

Now that you have uploaded data, time to explore and understand your data better using Aquarium!

Get to know some of the different views and features within Aquarium better:

Viewing Your Dataset

Analyzing Your Metadata

Querying Your Dataset

Organizing Your Data