Active Vision Dataset Benchmark

Welcome to the wepbage for the Active Vision Dataset Benchmark. Here we will define several benchmark tasks, as well as provide any data you may need in addition to the standard AVD downloads.

AVDB Evaluation

Labels for the test data are here: Test Labels (4MB)

Evaluation code here (LINK). You can use this to evaluate your system on the training and validation data.

AVDB Test Data

We provide 7 new scans of scenes as test data for the various tasks in AVDB. The ground truth labels for each task are withheld. Test Data (9 GB)

Task 1: Active Object Search (AOS)

Evaluation code here (LINK)

Here, a system is tasked with navigation close to a target object in a given scene. We provide an OpenAI Gym style environment to help with training on this task here (LINK). A small amount of additional data is needed to be added to each training scene from AVD, available in the link below. Data for AVD Training Scenes (4 MB)
This includes:

*each scene's world coordinates are in an arbitrary coordinate frame (i don't know where the origin is or why it is there) but are pretty close to being in millimeter units.

Task 1a: AOS in Known Environment

In this setting, the system will be testing in a scene it has encountered during training. The set and location of objects in the test scan will be different than in the training scan. There are three scenes in the AVDB_test set that also have scans in the AVD training data: Home_002_2, Home_008_2 and Office_001_2. For this task it is acceptable (and likely) that a different model be trained for each test scene. The idea is that a robot living in your house should be able to learn something about the static parts of the environment to help it navigate and find objects.


Task 1b: AOS in Unknown Environment

In this setting, the system will be tested in 4 unknown environments, Home_009_1, Home_012_1, Home_017_1, Office_002_1.


Task 2. Object Instance Detection

This task is the same as traditional object categroy detection on dataset like PASCAL VOC and MSCOCO, but with object instances instead. In fact we use the MSCOCO evaluation code.

Evaluation code here (LINK), though you can probably directly use the code provided from MSCOCO.

Task 2a.Known Object Instance Detection

Again, this task is the same as regular object detection. Train/Val/Test set here (recommended, test set is fixed)(LINK). Train and validation can both be used in training final model for evaluation on test set. LEADERBOARD (LINK)

Task 2b.Few-Shot Instance Detection (Class Incermental Learning)

This task is similar to regular object detection, but only a few examples of some of the instances are given. 10 instances are chosen as "few-shot instances", list here (LINK). The other 17 instances (LINK) have many examples as in Task 2a. The Train/Val/Test set here (recommended, test set is fixed)(LINK) is the same as in Task 2a. We host leaderboards for {1,2,5,10} training examples per "few-shot instance".

***IMPORTANT NOTE**** All 10 "few shot instances" in this task as present throughout the training data, with thousands of labels. In order for an entry to be valid for this task, these instances must not be treated as foreground during training. Valid options include simply ignoring the few-shot instance's labels, blacking out the few-shot instances, or many more. Any method submitting for this task must make code publicly available so this property can be checked.