The painted door test: How to verify your business idea

It all starts with an idea. You want to start a company, introduce a new product to the market or test a new feature on your digital platform. Of course, every such endeavour takes time and work. And what if it turned out that your idea wasn’t worth pursuing? You would waste precious resources.

Clearly, you need a more straightforward way to test your (product/business) idea. And that’s where the painted door test (sometimes referred to as the fake door test) saves the day.

What is the painted door (or fake door) test?

The idea behind the fake door test is very simple. Just like a door painted on a wall exists but cannot be opened, the painted door test is based on creating a non-functional version of your:

• Business
• Product
• Feature
• Service

What's the point of that? To see how many people would be interested in it. If you painted a door on a building, you could expect, at least in theory, that some people will try to go through them. It works the same in the digital world. You create a fake version of your product and see how many people will be truly interested in it by trying to interact with it (e.g., by entering a specific website, clicking a particular button or performing a specific action). Thanks to analytics platforms such as Google Analytics, you can estimate how much traffic your future idea will get.

Naturally, there are a few things to consider during such a test.

What do you need to know about the fake door test?

To explain what the fake door test is all about, let’s use a quick e-commerce example. Imagine putting a link or a button on your site to download a mobile app or to use a new payment option in the checkout section. Of course, you haven’t developed such a mobile app nor introduced the said payment option, so after clicking the button or selecting the option, the customer will be politely informed that this feature is not yet available and will perhaps come soon. [1]

image source: https://crstanier.medium.com/a-product-managers-guide-to-painted-door-tests-a1a5de33b473

You must remember that you can't let the users make too many steps before you inform them that this option isn't available. In the best scenario, it should be just one click. The juice is worth the squeeze, though. Thanks to such a test, you can verify how many people would download the app and how many would use the new payment option. You can even run some trial ad campaigns to see how much traffic your new product/idea can get.

Why is testing before implementation so important?

„Fail fast and with low cost”

If a feature is going to prove not impactful or even harmful to your business, you surely want to find that out as early in the process as possible – preferably even before committing any serious resources to implementation. That’s where the painted door approach comes in.

Testing as clue part of every step of project

In the software industry, we’re used to thinking that testing comes after implementation, even if it is minimal. With a painted door test, you don’t need to have a working feature: You can test the very idea of a feature in a real-world scenario to see if your customers would resonate with it.

And unlike the more traditional way of performing user/market research, the painted door test allows you to gauge customers' reactions for a fraction of the development or implementation cost.

Qualitative and quantitative user research can be expensive, and its results can be difficult to translate into tangible conclusions. Oftentimes, there’s also a substantial difference in how people respond when asked in a survey about a feature versus how they actually react when interacting with the said feature in real life.

The fake door test helps you deal with this issue. There are no surveys and no focus groups. Not only this way of testing is more effective, it’s also significantly cheaper!

What should you test?

Shortly put, everything that requires a lot of time or effort to get it running. The painted door test is actually very flexible, and you can use it not just in product/web development but even in:

• Sales and marketing (see how many people go to a landing page promoting your new product)
• HR (check how many potential candidates you will get for a new position in your company)
• SEO (run trial ad campaigns to see which keywords are the most attractive to potential customers) and many more business areas.

The six fake door test best practices

There are a few things to remember when running a painted door test:

• Make the subject of your test as legitimate as possible. So, if you run an e-commerce business, put the fake payment button in the actual checkout section. If you want to test a new feature of your digital platform, put it in the dashboard.
• Be upfront, and don’t confuse your users. That’s especially important when you meddle with their financial or personal data. If you test a payment method or a paid feature, you have to clearly communicate that the given solution is not available yet and that no money was taken from your users’ accounts. If you need to store some data concerning your fake door test, inform your customers that it is stored safely (and ensure that it’s true!).
• Test one thing at once. Imagine a situation when you run an online store and test many features simultaneously. Customers enter your website, and they see many options that don't work. That's a recipe for many lost customers and poor user experience. It's like A/B testing (which we're going to discuss in a moment) – you ought to test one thing at a time.
• Don’t require too much action from your customers: Don’t ask them to fill out any forms or leave their personal data. Just use an analytics tool that will measure the number of clicks on the fake button. At this point, you just need to verify whether there is any interest in your idea. Later on, you can opt for other tests.
• Ask for feedback: To get extra value from this experiment, you can give your customer a chance to provide more feedback. For example, you can give a 1–5-star rating on how much they would like to use this new payment method.
• Use additional solutions: You can combine a painted door test with other testing solutions, e.g., heatmaps. With a heatmap, you can verify not just how many people click on the test button but how many of them even look at the given area on your website or in your app.

image source: https://www.hotjar.com/product/heatmaps/

How does the painted door test differ from standard A/B testing?

The truth is that A/B testing is not so different from the painted door test. If you’ve run some A/B tests in the past, you’ll be able to quickly implement the painted door test as well.

You can, of course, run a painted door test by applying code changes, setting events via GTM and checking the results in the analytics feature, but there are simpler methods. One such tool that allows you to harness the full potential of A/B testing is Optimizely Feature Experimentation. It's easier and you can additionally segment the users to whom you want to target the test.

Optimizely Feature Experimentation is far more than just an A/B testing tool – it is a feature flagging and experimentation platform for websites, mobile apps, chatbots, APIs, smart devices and anything else with a network connection.

It allows you to deploy code behind feature flags, experiment with A/B tests and roll out (or roll back) new features immediately. All of this functionality is available with minimal performance impact via easy-to-use, open-source SDKs.

Check more about A/B testing in our article: What Are the Benefits of A/B Testing for your Business? 

The painted door test in Optimizely experimentation

Now, let’s see how a fake door test could be easily set up in Optimizely Full Stack. This platform comes with ready-made modules that can be used to run the painted door test. Here’s how it looks.

Create a flag and add a Boolean variable to it like in the screen below:

In the variations tab, edit the "On" variation and set its value variable to True. Then, in the events tab, create an event to be tracked when the option is selected:

Finally, add the A/B test type rule to the desired environment and set up the experiment according to your needs. In this example, only 50% of customers will be included in the experiment; 50% will see the new payment method and 50% won't. That's it! This way, you can use Optimizely to run a fake door test.

Summary

Before you decide to pursue your business idea, you ought to make sure that it will work and try to minimise the risk of failure. With a painted door test, it’s much easier, especially in the digital world.

NoA Ignite will help you verify your business assumptions before you decide to invest time and money in a new endeavour. We can assist you with many different testing forms, including the painted door test, research, prototype testing with real users etc.

If you want to know more about how we validate our assumptions, book a free consultation with our team!

---

(1) https://crstanier.medium.com/a-product-managers-guide-to-painted-door-tests-a1a5de33b473

Bibliography:

1. How to Avoid Building the Wrong Product? Make Your Product Roadmap Your Experimentation Roadmap
2. https://crstanier.medium.com/a-product-managers-guide-to-painted-door-tests-a1a5de33b473