Aquarium
  • Getting Started
    • Intro to Aquarium
    • Key Concepts
    • Account Setup and Team Onboarding
    • Quickstart Guides
      • 2D Classification
      • 2D Object Detection
      • 2D Semantic Segmentation
    • Announcements
    • Python Client API Docs
  • Data Privacy
    • Data Sharing Methodologies
      • Generate Local Credentials from AWS
      • Generating Access-Controlled URLs
      • Granting Aquarium Read Access to an AWS S3 Bucket
    • Anonymous Mode
  • Integrating With Aquarium
    • Creating Projects in Aquarium
    • Uploading Data
      • Labeled Datasets
      • Model Inferences
      • Unlabeled Datasets
    • Exporting Data
      • Batch Exports
      • Webhooks
    • Updating Datasets
  • Working In Aquarium
    • Managing Projects
    • Viewing Your Dataset
    • Analyzing Your Metadata
    • Querying Your Dataset
    • Organizing Your Data
    • Inspecting Model Performance
    • Analyzing Model Inferences
    • Finding Similar Elements Within a Dataset
    • Comparing Models
  • Common End-To-End Workflows
    • Assess Data Quality
    • Collect Relevant Data
    • Evaluate Model Performance
  • Python SDK
    • Python Client API Docs
    • Working With the SDK
    • Code Snippets and Examples
      • Segments
      • Confusion Matrix Scripting
      • Collection Campaign Scripting
  • Advanced Concepts
    • Adding Custom Embeddings
    • Dataset Checkpoints
    • Collection Campaign Classifier
    • Embeddings
    • URL formatting
    • Metrics Methodology
    • Complex Label Class Maps
    • Webhooks
      • Integrating with Labeling Using Webhooks
    • Custom Metrics
      • Stratified Metrics
    • Troubleshooting
      • Troubleshooting Common Web Issues
Powered by GitBook
On this page

Was this helpful?

  1. Python SDK
  2. Code Snippets and Examples

Confusion Matrix Scripting

PreviousSegmentsNextCollection Campaign Scripting

Last updated 2 years ago

Was this helpful?

In addition to exploring model failures in the UI, you can also script directly against the entries that make up the confusion matrix.

Full python client docs are , and a brief example follows.

PROJECT = 'YOUR_PROJECT'
INF_ID = 'experiment_1'
DATASET_ID = 'dataset_name'

al_client = al.Client()
al_client.set_credentials(api_key="YOUR_API_KEY")

metrics_manager = al_client.get_metrics_manager(PROJECT)

# Specify your query here. The union of queries will be returned.
# See the python client docs for the exhaustive list
queries = [ 
    # All Confusions
    metrics_manager.make_confusions_query(),
    
    # A specific Query
    metrics_manager.make_cell_query('gt_class', 'inf_class'),
    
    # A full row / column
    metrics_manager.make_confused_as_query('inf_class')
]

confusions_opts = {
    'confidence_threshold': 0.5,
    'iou_threshold': 0.5,
    'queries': queries,
    'ordering': metrics_manager.ORDER_CONF_DESC
}
confusions = metrics_manager.fetch_confusions(DATASET_ID, INF_ID, confusions_opts) # type: ignore

print('num_results: ', len(confusions['rows']))
print(confusions['rows'][0])
available here