Decision testing
Description
People can test how an automated decision changes by trying different data inputs before committing to the result.
For example, when buying car insurance, someone could test how parking their car in different places change the cost of the policy before they purchase it.
IF thinks this pattern is useful because it exposes how automated systems make decisions without overwhelming someone with technical information. This pattern works best when combined with ways to feedback or get support if the result doesn’t look right.
Advantages
- A low-friction way to help users understand more about how automated decisions are made that doesn’t require lots of previous knowledge.
- Helps someone test the output before they commit to a course of action.
- This pattern allows people to test a model using dummy data. (They don’t necessarily have to share any real data about themselves.)
Limitations
- People might create their own theories for why outcomes change, unless it’s combined with other explanation methods. This could work against the intent of the pattern by undermining understanding and explainability.
Examples
-
Flock, a drone insurance company, uses this pattern in their app to help pilots understand the risk number generated by Flock’s algorithm, and act on it by using it to select different insurance plans.
-
ImageNet Roulette was an experiment published by the AINow Institute that used an algorithm trained using images of people from ImageNet to classify photos uploaded by users. The authors said:“...ImageNet contains a number of problematic, offensive, and bizarre categories. Hence, the results ImageNet Roulette returns often draw upon those categories. That is by design we want to shed light on what happens when technical systems are trained using problematic training data.”
Related patterns
Was this pattern useful?