Querying Your Dataset

Use the Query Bar to slice and dice your frames, labels and inferences by metadata and model performance.

Introduction to the Query Bar

The Query Bar, located at the top of every screen, is the first class way to slice and dice datasets in Aquarium.

Click in to the Query Bar to see the full scope of available options, but the upfront highlights are:

• Use simple query language to filter frames, labels or inferences by their metadata.

• Query based on frame level performance metrics (Precision, Recall, F1)

• Chain query parameters together using logical ANDs, ORs, and up to one layer of parenthesis.

• Save the point-in-time results of a query by clicking the Add to Segment button, directly underneath the query bar.

Once a query is applied, the result is available everywhere in Aquarium. Switch between the Grid, Histogram, Embedding, and Metrics views to quickly dive deeper into your filtered frames and crops.

Frame Level Queries

Metadata Queries

Using the query bar, you can filter your dataset based on any system generated or user submitted frame-level metadata.

Common frame level metadata queries include things like assigned split, data collection facility, labeling provider, etc.

To query metadata fields for labels and inferences, you have to explicitly call update_dataset_object_metadata_schema when uploading the dataset in order to search those fields using the query bar.

Frame metadata queries follow the pattern:

frame.<metadata_field> <operator> <value>

Filtering a dataset to all frames assigned the "Training" split.

Frame-level metadata queries return:

• All frames that match the criteria,

• All crops within the returned frames.

Contained Object Queries

In addition to direct frame metadata queries, it is also possible to query frames based on the objects contained within them.

Contained object queries are useful for things like finding dense frames and finding frames with at least one object exhibiting a property.

Contained object queries follow the patterns:

frame.<sub_element_type>.<metadata_field>.<aggregation> <operator> <value>

Filtering a dataset to images with at least 10 cars.

frame.<sub_element_type>.attribute.<metadata_field> <operator> <value>

Note that frame level queries on object metadata (e.g. frame.label.attribute.width > 50) will return all frames with at least one object matching the criteria.

Filtering a dataset to images with at least 1 inference result with confidence greater than 0.95

Contained object queries return:

• All frames that match the criteria,

• All objects within the returned frames.

Metric Queries

Provided an inference set has been uploaded, it is possible to query frames based on their performance metrics.

Frame-level metrics accessible via the query bar include Precision, Recall, F1, Support, False Positive count, False Negative count, Confusion count.

Frame-level metric queries follow the pattern:

frame.inference_metrics.<metric>.<class_mix> <operator> <value>

Note that all metrics queries imply confidence and IOU thresholds. For transparency, these are automatically added to the Query Bar once a metric query is applied.

Edit the IOU and confidence thresholds via the Display or Metrics settings, and remember to update the Query Bar after editing either parameter.

Filtering a dataset based on frames with a high Precision score.

Frame-level metric queries return:

• All frames that match the criteria,

• All objects within the returned frames.

Object Level Queries

In addition to writing queries directly at the frame level, it is also possible to directly query the objects within frames.

Direct object queries behave differently than frame queries. Importantly,

• in order to be compatible with the Metrics View, all object level queries applies at a specific IOU and Confidence threshold.

• object queries apply an implicit frame level query first (i.e. first filter to frames that have at least one object that match the query, then filter to objects within those frames that match the object level query).

• labels and inference results are independently queryable. If you expect both labels and inference results to be returned by a query, configure two separate parameters and chain them with an OR statement.

Object level queries are useful for limiting the Grid, Embedding or Metrics views only to specific classes or to objects based on object-level metadata (e.g. distance from camera).

Filtering a dataset to only Van labels.

Object-queries return:

• All frames with at least one object that matches the criteria

• For the Grid View

  • Emphasized objects that match the criteria, with other objects deprioritized but still visible in the preview. Configure this behavior with the Show Adjacent Crops setting in Display Settings.

• For the Embedding and Metric Views

  • Objects that match the criteria, and if it exists, the paired label/inference for the matched object (e.g. with label.class = car applied, the results will include all car labels and any inference matched to a car label at the current IOU and Confidence threshold).

Notes about Object Level Query Interactions with the Metrics View

Object level metrics filters via the Query Bar are an experimental feature in Aquarium! Be aware that the methodology and presentation may update over time. Contact the Aquarium team if you have questions or feedback.

Label-only Queries

Queries of the pattern label.class = car will only return labels that match the criteria, and if they exist, the inferences paired with those labels for metrics calculations.

This is analogous to only returning a ground truth row of the confusion matrix in Aquarium.

Metrics with only a label filter applied.

Inference-only Queries

Queries of the pattern inference.class = vehicle will only return inferences that match the criteria, and if they exist, the labels paired with those inferences for metrics calculations.

This is analogous to only returning an inference column of the confusion matrix in Aquarium.

Metrics with only an inference filter applied.

Combined Label and Inference Queries

Queries of the pattern inference.class = vehicle AND label.class = car will only return matched pairs of inferences and labels where both conditions are met. This query will not return unmatched car labels or unmatched vehicle inferences.

This is analogous to only returning a label / inference intersection of the confusion matrix in Aquarium. In general this type of query is more useful for looking at examples in the Grid or Embedding views and it's preferable to interact with the confusion matrix directly in the Metrics view.

Queries of the pattern inference.class = vehicle OR label.class = car will return

• all labels that meet the label condition and the labels' paired inferences, if they exist

• all inferences that meet the inference condition and the inferences' paired labels, if they exist

This is analogous to returning both the row and column of the confusion matrix in Aquarium.

Note that for object level metadata queries on range values, the OR behavior in Aquarium does not enforce assigning each label / inference pair to exactly one query result bucket and does not prioritize either the label or the inference criteria match, but evaluates both equally.

This reflects a true logical OR, but in certain cases may lead to the same inferences shown in multiple, similar query results and metrics that are unintuitive when compared across queries.

Putting it All Together

In addition to individual frame and object level queries, the Query Bar also supports logical operations and groupings of query parameters.

Chain any query statements together the AND and OR operators.

Query statements can also be grouped with parenthesis, which will ensure grouped statements are evaluated first.

Note that currently parenthesis may only be one layer deep (i.e. no nesting parenthesis).

The example below uses logical operators and grouping to identify specific examples of confused car labels within otherwise high-performing frames.

Querying frame metrics, labels and inferences.