Back to blog posts

Generating Initial Conditions for Enhanced Scenario Coverage

Wed Jun 22 202238 views

Maybe you have a large amount of labeled data (we do too). And maybe you can extract from this an empirical distribution of initial conditions or even full replay trajectories. But what if you could multiply that data infinitely, in particular by learning a model that can generate realistic initial conditions, multiplying your scenario coverage literally right from the start. Inverted AI researchers have invented a model that can do just that.

Let us explain what we mean via a short quiz. In the examples below some of the vehicle arrangements are labelled training data. Some are samples from our INITIALIZE cloud API. Can you tell which are which? (the answer appears at the bottom of this post)

initial-conditions-content-1.png

Note that vehicles that overlap onto the black are in the training data too so that doesn't help. Red lines indicate vehicle fronts.

This model is a "permutation invariant normalizing flow built on equivariant continuous dynamics" and is the subject of a paper just released on arXiv and under review at NeurIPS 2022.

An intuitive sense of how it works can be gathered from watching its internal flow do its job, starting from randomly placed vehicles and "flowing" them to final conditional locations. Here the condition is the underlying map image.

initial-conditions-content-4.gifinitial-conditions-content-2.gifinitial-conditions-content-3.gifinitial-conditions-content-5.gifinitial-conditions-content-7.gifinitial-conditions-content-8.gifinitial-conditions-content-9.gifinitial-conditions-content-10.gif

Because this model is a normalizing flow it can be used for outlier detection, a fact that we exploit in our own data labelling pipeline, and you might too. For instance thresholding the conditional likelihood of a configuration under this model is a good way to detect mislabeled images.

initial-conditions-content-11.png

LP stands for "log probability." There is a large, detectable difference in conditional log probability for test samples that have been modified to have, for instance, as shown in blue, even a single vehicle mislabeled in terms of orientation.

The use case of such a model for autonomous simulation should be obvious, but, if for some reason it is not, here are some samples from Initialize used to generate initial conditions for a scenario in CARLA.

initial-conditions-content-12.pnginitial-conditions-content-14.pnginitial-conditions-content-13.pnginitial-conditions-content-15.pnginitial-conditions-content-16.pnginitial-conditions-content-17.pnginitial-conditions-content-19.pnginitial-conditions-content-20.png

These are all completely synthetic and, for those familiar with the CARLA autopilot, are traffic configurations that would simply not occur under the traffic manager.

Integrating INITIALIZE into your simulator via our cloud API is straightforward. Contact us for support and details.

The underlying research and model are also useful for object detection. This was demonstrated in the paper on the CLEVR dataset where the flow was trained to conditionally detect objects. Here the flow transforms boxes in space so that they region propose bounding boxes for the objects in the scene.

initial-conditions-content-21.png

We promised the answer key for the quiz above. Here it is.

initial-conditions-content-22.png

Data means training data. All of the others are synthetic. The last column are samples that have more vehicles on that map than ever appeared in the training data!

Interested in the technical details? Check out the arXiv paper just released.