Does Your Business Have a Mobile App? Here’s How to Keep It From Breaking
Last time we talked about preventing bad customer reviews when you launch that shiny new app for your business.
Today, I want to tell you how to keep your beautiful new app running smoothly on user’s devices for years to come.
There’s one thing — above all others — your software team needs to be doing to keep your app performing at its best: regression testing.
What Is Regression Testing?
The term “regression testing” sounds big and scary and technical, but it’s really not.
Literally, “regression” means “going back” to a previous (usually worse) state of your application. In practical terms, it means your app used to work — but now a feature within the app is broken.
Your app has “regressed” to a state before the broken feature worked in the first place.
Regression Testing: A Tool to Identify and Prevent Regressions
“Regression testing” is the process of regularly testing your application to detect and prevent regressions.
When done well, regression testing will help you identify when something in your application breaks. This allows you to fix the problem and release a new version of the app quickly — hopefully before you start receiving negative reviews in the App Store.
Why Do Regressions Happen?
Regressions happen within apps for many reasons:
Changes to the operating system. Just like you make changes to your application, Apple and Google continuously make changes to their operating system. The “rules and regulations” of what your mobile application can and cannot do are determined at the OS level and can also change over time.
For example, as of iOS 5, app developers could access a user’s address book information without jumping through any hoops. In iOS 6, after some apps were found abusing this access, Apple introduced a new permissions model that required app developers to ask the user for permissions before accessing their private address book information.
If your iOS app was designed for iOS 5 and relied on the address book to work properly, you soon found out that your app didn’t work anymore! A regression had been created in your app when the operating system changed under you.
Changes to the development platform. Between the operating system and your mobile app is a layer called the “development platform.” It includes all the tools mobile developers use to bring your app to the market. Like the OS, the development layer can cause regressions in your app.
The development platform evolves over time. This is usually a good thing — since it gives developers more and better ways to work on your app. But every change also introduces the risk of regressions and bugs in the development platform itself.
Another way the development platform can introduce regressions in your application is by deprecating tools your mobile app currently uses.
Apple and Google are constantly improving their development platforms, introducing new tools and removing others. When tools in the development platform are deprecated, this means that they will stop working at some point in the future and your continued use of them comes at your own risk.
For example, Apple deprecated its long-running iOS networking layer when it released iOS 7. As of this writing, the old networking framework still works, but don’t be surprised if at some point down the line, you start seeing bugs in apps that haven’t been updated in a while. There’s a good chance they’ll be picking up regressions in their application since they’re still relying on the old networking layer.
Changes to third-party dependencies. Regressions can also be caused by open-source, third-party libraries you use in your app.
A regression caused by a change in one of your third-party dependencies is often easier to fix than one caused by changes in the operating system or the development platform. That’s because the code is open source, so you can inspect the error and fix it yourself.
Keep in mind that not all third-party dependencies are open source. Some vendors, like ad networks, require you to use their code libraries, but they don’t give you access to the source code.
If there are any regressions caused by them, you’ll be on your own to fix them.
Human error during development. Finally, the last and most common cause of regressions is human error during development. Human-caused regressions are far more common than regressions caused by the operating system, development platform or third-party dependencies (if you choose them correctly).
If you encounter a regression in your app, the most likely cause is that it was introduced by a developer on your software development team.
Ideally, any regressions introduced by active development should be caught by the QA processes you set up during development. If not, the last line of defense before releasing to your end users is regression testing.
How to Create a Regression Testing Plan
A regression testing plan is the tool you’ll use to identify and resolve regressions before they become a problem for users. Once you’ve created, you’ll lay out everything you’ll need to test your app on a regular basis to identify and prevent regressions.
Start your regression testing plan by making a list of what you need to test — of the features, screens and expected behaviors in your app. Then, add more things to test by looking back at what areas of your app have been known to break in the past.
After you have a defined list of functionality to test, your next step is to decide where to test it. This may seem like a question with an obvious answer: “Test on a smartphone, of course!”
But there are two things to consider about where you test your application:
- What operating system versions do you target? Ideally, you should run your entire regression test plan on every version of the platform you support. For example, if you make your app available on iOS 9 and iOS 10, you should run your entire test plan twice, once per operating system version.
Which hardware devices do you target? Let’s say you have an iPhone application that runs on iOS 9 and 10. As of this writing, these are the all the devices where your application can be installed:
- iPhone 4s
- iPhone 5
- iPhone 5c
- iPhone 5s
- iPhone SE
- iPhone 6
- iPhone 6s
- iPhone 7
- iPhone 6 Plus
- iPhone 6s Plus
- iPhone 7 Plus
- iPod Touch 5th Gen
- iPod Touch 6th Gen
The number of devices you support grows in relation to the number of OS versions you support, as well as the number of form factors you support (e.g. phone versus tablet).
In an ideal world, you would run your entire regression test suite on all the operating system/device pairings you support.
If you end up with too many combinations of operating system and hardware devices (a common problem on Android), then you should pick a representative number of pairings based on your current installed base and common hardware differences.
For example, if you support both iOS 9 and iOS 10 and you see that 95% of your users are using iOS 10, target most of the OS/hardware pairings towards iOS 10.
You should also make sure you hit all screen sizes you support, as this is a common source of UI bugs. The iPhone 4s, iPhone 5, iPhone 6 and iPhone 6s all have different screen sizes. Make sure you account for each in your regression testing plan.
Additionally, if your app uses special hardware features that are available on certain devices but unavailable on older devices, you should also take this into account when deciding on which OS/hardware pairings to test on.
On iOS, some recent examples of special hardware are Touch ID, 3D touch, Apple Pay, and the motion co-processor.
How to Create Regression Tests
Ideally, the QA engineers on your development team will create a series of regression tests prior to releasing the app for download.
A good QA engineer or QA team is responsible for maintaining the regression plan — and for keeping it up to date and ready to use. There are specialized tools to maintain your test cases (i.e. how you document what and how to test) like Testrail or Testopia, but for smaller projects, a simple list or spreadsheet can also work.
If your app is post-launch (it’s already live in App Store), treat regression tests like a new development effort. Assign someone to create the tests, then give them a date to deliver their work to the team.
When you add new features to the app, have your QA team add tests to the regression test suite so they can be used moving forward.
Once you have the tests created, a good rule of thumb is to go through regression testing when you or any software you depend on (e.g. OS, development layer, third-party tools, etc.) are about to release an update.
For example, it’d be a good idea to run your regression tests when:
- Your initial launch went well. Users love your app! Two months later, you decide to add two new features to your mobile app that will touch 10% of the code base. After development is complete — but before public release of the update — run through your entire regression test suite.
- Your mobile app has been out for months. Users are happy and haven’t had any major problems reported. You’re planning no further changes for the foreseeable future, but Apple is about to release a new version of iOS (let’s call it iOS 11) in exactly one month. Before that happens, run through your entire regression test suite on the beta version of iOS 11.
Doing this will ensure you find out any bugs early before your users do.
What Does This Mean for You?
As we said in our previous article about QA best practices, developing a high-quality app requires the right people, process, and culture. In this article, we talked more about the the process of keeping your app running smoothly by catching problems early and often.
Bugs and regressions are two of the major causes of negative App Store reviews, a fact that can be extremely frustrating when the regressions come from changes outside of your control.
Developing a high-quality mobile app is a little like hitting a moving target while standing on a moving platform. Changes to the operating system, development platform, or any other piece of software you depend on can end up causing problems for your end users.
A solid regression plan helps you mitigate this problem.