15 beta flag best practices

Software feature flags – also called beta flags – are an important component of any feature release program. With most development teams continuously delivering new updates, beta flags can be an invaluable way to test new features and experiences, and gather customer feedback for further improvement.

What is a feature flag? They are markers that designate new features available in beta either openly or to a closed group of users for testing and feedback.

These new features might be a little rough around the edges (which is why they’re “flagged” as being in beta), but are complete enough to deliver value to some users.

So, let’s take a closer look at some best practices for beta flags (and programs).

1. Start with users

New features often come about as a result of customer feedback. If a customer offers up really valuable (or passionate) feedback and ideas, you might reach out to them for further info. Set up a discovery call to hear more about their use case and what they had in mind for your product.

While in the early discovery phase, create low-fidelity prototypes using basic interactive wireframes. Then conduct usability testing to see how a user might interact with the new feature. Because wireframing is quick and inexpensive, it’s a great way to test and pivot ideas.

After testing your low-fidelity prototypes, but before development, you should create high-fidelity prototypes for usability testing. While slower than wireframing, a visual design will help you test the intuitiveness of the product or feature, and ensure you get this right before developing it.

2. Before beta release, do browser-based usability testing

After developing the feature, conduct further usability testing in the browser. In this format, it’s as close to reality as possible, so the results are very valuable. Practically speaking, however, you’d likely only be able to make small tweaks based on feedback, as the goal is to get your feature to beta release for user testing as soon as possible. For example, in 2019, our team at Tiny performed an internal browser-based test for our Advanced Tables feature and made several changes before beta releasing it.

3. Choose between a closed or open beta release

When deciding whether to do a closed or open beta release, consider what data you need to collect. 

For targeted testing from a small subset of users, do a closed beta release and release protected code to a single group. A closed beta is an invite-only beta group and not open to the public. This approach could be useful if you need to test with a specific segment (for example, testing a new tables feature with customers who use a lot of tables).

This exclusivity can help build closer relationships with specific users, but it’s also trickier to recruit them, compared to a public release. It’s also worth considering the extra work involved in changing your technical processes to securely deliver unreleased code to specific users.

For wider testing, use an open beta release that’s open to the public. This means that the updates get pushed to the entire user base for feedback and bug reports ahead of a formal release. These updates are shown with a beta tag so that it’s clear that the product or feature isn’t yet fully released. By reaching more users more quickly, bugs are likely to be found more quickly and the release is both simpler and lower cost. After all, the only thing you need to do is update the website and doc labels with the open beta flags and add a release number. This labelling is important to set users’ expectations, especially if your reputation is built on solid releases.

The default lifecycle for new features includes an open beta. Building in iterative functionality likely won’t require an open beta unless it’s a significant update.

4. Avoid multiple concurrent open betas

Ideally, try to avoid having multiple features in open beta at any time, unless absolutely necessary. This could impact the overall user experience and overwhelm your development team, with the potential for more bugs and feedback requests. If you need to test multiple features in beta at the same time, it’s better to run these as closed betas.

5. Update your version number as you move out of beta

As you prepare to move out of beta, you’ll need to update the version number. Follow Semantic versioning standards with MAJOR.MINOR.PATCH numbers. If it’s a new feature, the first official release number will change from 0.X.X to 1.0.0. Then future updates post-release will depend on whether it’s a major, minor, or patch update.

6. Set a time limit for your beta flag

Features should not sit in beta indefinitely. Set yourself a reasonable deadline for releasing the full version of your feature. At the end of this period, your product and engineering teams’ aim should be to move the feature to the next stage in the maturity lifecycle so that it becomes part of the natural product landscape. For example: Planned/Scoped → Open Beta → 1.0 Release → Further Iterations.

The beta status is removed as part of major or minor product releases - not on patch releases.

At Tiny, we usually give our open betas 6 months before we bump the version up to 1.0.0. for the full release, then schedule quarterly minor updates (with the right to make exceptions, where needed). 

The 6-month timeframe is practical because:

• It’s usually enough to get a decent amount of feedback
• It gives the marketing team breathing space to promote the open beta as well as the wider release
• It enables the engineering team to tackle additional projects

In some cases, a feature might leave open beta sooner, especially if it’s smaller or simpler. Or if it’s more complex, the feature might stay in open beta for longer than 6 months. Beta status is removed on major or minor releases (not patch releases).

7. See your beta flag as an intent to deliver

Whether you go with a closed or open beta, it’s important to remember that you have an underlying commitment to ship. While your feature will likely be iterated on due to feedback and testing, the core elements of your feature are already determined. It’s not just about delivering on your promise to customers. Your sales and marketing team will also feel comfortable talking about the new feature to customers when they are confident about the end result.

8. Set requirements for your beta

It’s important to define what an acceptable beta looks like for your team. Usually, it won’t meet the same standards as your fully released features. But the tradeoff is that you’ll be able to get it to your users (and start getting feedback) sooner. 

For example, at Tiny, we consider an open beta to be pre-release software. Even with minimal features, it should include enough context (via docs or website information) for a customer to integrate it into their application. When our software is in open beta:

• We don’t require translations (English-only is fine)
• It doesn’t need to comply with accessibility standards (although accessibility is important to us and we do ensure our released software meets the standards)
• It may have some known shortcomings

9. Set exit criteria for your beta

While your feature will likely stay in beta for a set period of time, it’s also worth setting exit criteria so you can work towards getting your feature ready for full release. Exit criteria will be defined on a case-by-case basis, depending on the feature.

10. Provide acceptable support for your beta release

Because your open beta is considered “pre-release software”, you can provide a different level of support. It’s a good idea to define this in your license agreements (to see an example, take a look at Tiny’s license agreements) so that customers know what they can expect.

11. Update your terms and agreements to cover betas

Your customer agreements should address issues specific to pre-release software and beta features. Your agreement might state that:

• You don’t promise or guarantee that pre-release software will be announced or made available in the future
• Beta features may contain bugs, errors, and other problems that could lead to failures and/or data loss
• The pre-release software does not represent the final product
• The software is provided “as is”, disclaiming any warranty or legal obligations
• Customers who perform research/development with pre-released software is done at their own risk

12. Provide reasonable levels of support for beta features

Although you might not be legally required (in your terms and conditions) to provide the same level of support for beta features, support is still important. The point of having a beta is to gather feedback and track down high priority bugs ahead of a full release. Reasonable levels of support means that you’ll respond to customers’ questions and concerns, and work on improving the software. But it does also mean that we can’t guarantee there won’t be changes to APIs, and some bugs might not be considered as critical.

13. Promote open beta features in your marketing

Beta is the perfect time to start promoting your new features because you can experiment with different messages, illustrations, and landing pages. Ensure that key places are labelled with the beta status, like the pricing table, product pages, and docs pages. But it’s a good idea to keep the beta tag off digital artefacts (like product images) so the move to version 1.0 is as straightforward as possible.

14. Name your beta program carefully

What’s in a name? The name of your beta program is actually quite important as it can set users’ expectations for what you’ll deliver. At Tiny, we’ve discussed a number of options:

• Beta Program - This is the most common and easy-to-recognize name, but it does indicate that you’ll provide early access to code (which may not be the case)
• Discovery Program - This term is more generalized, which is useful for covering a broad range of research
• Insights Program - This is also broad, while putting the focus on the customer experience and use cases
• Explorer Program - This is another broad one that helps to set the tone for your brand and beta product experience
• Technical Preview - Rather than saying “beta” which sets the expectation that it’s ready to use (but likely has a few kinks), a “Technical Preview” tells customers that the feature is a work-in-progress and things may change, but they can try it out and provide feedback

At Tiny, the name used is "Insights Program". This may change to Beta Program once closed-beta access to code is available for early testing.

15. If your beta fails, it’s okay to retire the feature

Finally, it’s important to define what you’ll do if a beta doesn’t go to plan and you ship a dud. Let’s say, you launch an open beta and despite the best intentions, planning, and testing, you realize that it’s not workable. Perhaps it’s not financially or strategically viable to maintain the product or feature. In that case, it’s okay to give your customers notice and sunset the feature so you can move on and focus on other features.

Create better products sooner with betas

Ready to launch your own beta program or feature? While there are a number of best practices to keep in mind and you do need to lay some groundwork to maximize the chances of a successful beta, the benefits are well worth it.

A beta program is one of the best ways to test out new features, recruit participants, gather feedback, and build better products in less time - all while building a strong community around your product.

Whether you go with an open beta or a closed beta approach, lowering the bar with a beta flag ensures you can ship new capabilities to real customers much sooner. Use the insights from these customers to fine-tune the features before removing the beta flag and doing a full release, all while generating greater interest from potential customers who might need the capabilities right away or like being the first in line to try new things.

Tiny Insights in the future: the product roadmap

While Tiny Insights is our own beta program here at Tiny, the most recent initiative is a TinyMCE product roadmap.

We've launched the TinyMCE Roadmap so you can see what exactly we're working on. 

You can also have your say and contribute to what comes next in the future of TinyMCE.

Insights are hugely valuable for our internal teams here at Tiny. Our Product & Design team has information on what's needed from the community, and for our Engineering team, it means smaller loops for bug tracking and usability concerns.

As a flow-on effect, our customers are better informed, more included in our efforts, and end up with more intuitive products that solve real-world problems.

Have your say on the TinyMCE Roadmap

We’d love to hear from you. Check out the TinyMCE Roadmap. We regularly send out email updates with a summary of recent testing efforts, along with details about any new opportunities coming up.

There’s new ideas on the roadmap for 2022 and beyond, so if you want to be among the first to see about what’s happening at Tiny (and get the chance to have your say) and contribute to the TinyMCE Roadmap!