Uploading Data

Getting your data into Aquarium

Overview

There are three kinds of datasets you can upload into Aquarium.

• Labeled Data

  • Labeled datasets include all of your ground truth labels for your dataset

• Model Inference Data

  • Inference sets include all of the related data from your model's predictions/results for your dataset

• Unlabeled Data

  • Unlabeled datasets consists of unlabeled datapoints with or without model results/predictions

To upload data into Aquarium, there are a few main steps to get going.

• Create a project in Aquarium for the specific ML task you're working on

• Format your labels, metadata, and inference/predictions into frames

• Add frames into a dataset

• Upload the dataset into Aquarium

How Do I Upload My Data?

Prerequisites

• You'll need to setup an account with Aquarium before you can upload your data

• Install the aquariumlearning package

• Ensure you have at least a version of Python >= 3.7

Python SDK

Data uploads are done through our Python SDK.

Aquarium provides a python client library to simplify integration into your existing ML data workflows. In addition to wrapping API requests, it also handles common needs such as efficiently encoding uploaded data or using disk space to work with datasets larger than available system memory.

We have quickstart guides and detailed documentation available to demonstrate how you can upload data for a few demo datasets.

We also have step-by-step guides on how to upload different kinds of data into Aquarium:

• Upload Labeled Datasets

• Upload Model Inferences

• Upload Unlabeled Datasets

Monitoring Upload Status

After starting an upload, you'll be able to view the status of your uploads in the web app.

If you go to the Project Details page, you'll see a Streaming Uploads tab:

Each upload ID corresponds to a subset of your dataset/inference set (with the associated frame count + label count).

To view more details on which specific frames/labels are present in a given upload, you can click on the Status (e.g. DONE). A pop-up will appear with the following info:

In the case of a failed upload, you can debug via the Errors section (which exposes frame-specific debug logs), and download this info to determine which frames/crops may need to be re-uploaded.

To access the Errors section, click on the Status for each row:

Where to click on the row in order to view the Errors section

Example of the Errors section

If you are running into an error and the error logs are not sufficient to understand how to fix the issue, please reach out to the Aquarium team and we can help resolve your problem.