These days it seems only fashionable to be in Beta. We all remember trying to get a coveted invitation to join the Beta for Pinterest or Inbox by Google. Beta has become a gimmick for some companies. It is being used as a marketing tool where scarcity can be employed to fuel demand, build buzz and create virality. Beta has become the new “Tickle Me Elmo”.
The History of Beta
But the truth is that Beta existed long before Pinterest, Google or Apps. In fact, the term “Beta” originated in the 1950’s when Martin Belsky, a software project manager at IBM, first used it.
Beta is typically the first time that software is available to real users outside the company that developed it. These days some companies such as Google seem to keep software in perpetual Beta adding new features and functionality on a continuous basis.
Advantages of Beta
1. Allows Faster Pace of Development
Users do not expect Beta releases to be bug free so this allows companies to keep an ambitious development schedule.
Technology is moving so fast that waiting for a perfect version is not practical.
2. Companies get Valuable Feedback from Customers
Apps can be released to the market sooner so they can get customer feedback which can help to shape the App and identify bugs.
The more people who use the App and report bugs the better the App will become.
3. Customers Experience Benefits from the App
Customers can begin using a Beta version early so they can immediately benefit from the functionality it offers to improve aspects of their business or life.
Customers can request features or suggest improvements to make the app more efficient.
4. Tests App on a Wide Range of Devices
You need a very large test group to cover the myriad of Android and iOS devices and versions of operating systems. Beta allows you to expand your test group.
Networks vary from place to place so having users distributed around the globe can help identify performance issues.
The Reality Check – What are the Limitations of Beta for Apps
1. Too many Permutations and Combinations to Test
According to Open Signal’s 2014 Report on Android Fragmentation there were 18,796 distinct Android devices. And that’s not counting different versions of operating systems or the whole iOS landscape.
2. Limited Information can be collected using Analytics
Free analytics such as Google Analytics App Tracking for Android and App Tracking for iOS can only provide clues about the user experience. For instance, if a bad crash occurs with the App it may be unable to send any information about the crash.
If a user drops out of a process at a certain point, logging data will show where they dropped out but it does not tell you why. Perhaps it is because the user interface is confusing or not working. Unless a user gives you feedback you can’t know for certain.
3. Startups typically won’t have enough Beta Users
You need a group of users large enough that the vocal few will actually give you feedback. Statistically only a tiny portion of users will care enough to report bugs or give you their opinion.
A small user group won’t be able to test across the array of devices, operating system and networks available so Beta tests will be limited.
This can cause a chicken and egg situation. For instance if there is a bug in a registration process new users may not be able to use the App. Since this will suppress the number of users, you may not get enough users that you are statistically likely to get a report of the bug from the Beta users.
How to Solve the Beta Testing Problem
There have been efforts to solve the Beta testing problem.
- Companies offer testing services but these are usually too expensive for most startups.
- Groups of developers on the internet have formed to test each other’s Apps but this has obvious limitations.
Beta Testing will remain a conundrum, especially for startups. It is because of the innate nature of App development which is an iterative process. An App is not developed and tested in a highly orchestrated fashion where you write the App for a time period, then release it, test it and fix all the bugs. It is far messier than that.
You will wear out even your oldest and best friends who so generously and enthusiastically agree to test your App. De-bugging is not for the faint of heart. Most ordinary folks just get worn out.
One of our developer friends tested a very early Windows version of our MondoPlayer . It crashed his computer so badly he had to re-install his operating system. Oops!! Needless to say, he is rather jaded on the subject of Beta testing for us 🙂
My Co-Founder, Cristian Worthington, says he feels sorry for the well meaning person at a cocktail party who asks him what he does.
He briefly describes the MondoPlayer App and if they show evening a passing interest he gets them to install the App on their phone. He watches over their shoulder as they install it.
They think I am just showing them the App but what I’m actually doing is watching every twitch of their finger to see if they are doing what we we think users should do.
You become a bit of a nut. You literally crave the opportunity to watch people because those opportunities are incredibly valuable.
Why Most Apps are Simple
There is a reason that we are seeing simpler Apps with fewer functions. Many of these Apps are a beautifully reduced. Why? Because, the more complicated your App, the more challenging the testing becomes. There are many more things that can go wrong and you are more likely to be affected by the operating environment on the device and the interplay with other Apps that may compete for memory and resources.
Why Your App Has to Satisfy a Need
The bottom line is that if you are a small startup and you are relying on Beta to do your debugging for you, you are probably deluded.
You are going to have to make an extraordinary effort to make sure your App is good. That’s probably why many Apps stay in Beta for a long time.
We need users to be patient and tolerant so we can continue to develop in a less than perfect situation. If your App provides enough of a benefit to users they will cut you the slack you need. And they will help you to succeed.
Photo Credit: Steve Jurvetson