At Mubaloo, we take great pride in the quality of mobile apps we produce and as Head of QA & Testing, it’s my responsibility to ensure that we deliver our projects to the best standard of quality. Our employee knowledge, teamed with continuous R&D and testing, ensures we are constantly pushing the boundaries of what’s possible through mobile.
One of our key strengths when developing and testing is that as often as we can we stick to physical devices for our integration phases. While we could rely on cloud platforms to test the bulk of our apps, they are not able to test specific requirements and are mostly based on the most basic of test automation frameworks (more on this later).
So, how do we get around problems experienced when testing real world devices? How do we obtain sufficient test coverage in such a competitive environment? How do we dictate and decide which devices and OS versions to support? We’ve opened the Mubaloo test lab to help answer some of these questions and also to show how it is possible to achieve a high level of test coverage and quality even when it’s not feasible to test every single device and OS.
Challenge One to 4,000+
As of January 2013, there were 3,994 different Android devices on the market worldwide. Many of these devices sell for around £100-£150. This is great in terms of accessibility but because many of these devices are low end, they sometimes create issues when it comes to testing. They will often have older generation hardware that often lacks the punch of the likes of the Galaxy S4 or HTC One. Additionally, with continual updates of the Android OS, many of these older devices have been updated several times in the past couple of years and supporting all of them just isn’t cost-effective.
The ‘F’ word
The Android ecosystem is fragile and quite frankly in a state of disrepair. Google and OEMs have made it incredibly difficult for indie developers or companies like us to easily target specific software versions or hardware types on a budget. The Samsung Galaxy SII for instance, was launched on Android 2.3.3. This device has been, and still is, incredibly popular worldwide. As a result, Samsung has supported it for a long time and continues to do so. The device is now in the middle of receiving an update to Android 4.1.2.
Let me put that into perspective for you. Since launching on 2.3.3 in February 2012, the device has received in total, seven Android upgrades (2.3.4, 2.3.5, 2.3.6, 4.0.3, 4.0.4, 4.1.1, 4.1.2) – this does not include firmware versions supplied by Samsung directly. Due to Android’s open environment as well, Google give OEMs the power to modify the core components of the OS. This means that every time a ‘vanilla’ Android update is released, OEMs need to engineer the OS to be compatible with each of their devices. It can often take anything from 6 months upwards for a manufacturer to release an Android update from the time it was initially released by Google. Still, two years on, Android 2.3 has the highest market share for the platform and there have been almost 10 OS versions released since it was released in December 2010.
Android is fragmented, and it is impossible for you, us or anyone else to support every device and every OS version. Even if you could afford to buy EVERY device, running EVERY software version ever, it would take you an infinite amount of time to accommodate development and test efforts.
iOS, however, is a different story and is easier to work with in terms of testing. Due to the fact that Apple own the hardware and the software, they can easily control which future OS updates will be compatible with which devices. This means that the iOS update process is lightning fast when compared to Android.
To put that in numbers; in the first 24 hours of Apple’s iOS 6 being available to download, around 15% of their user base had installed the update. Compared to Google’s Android Jelly Bean update, which saw only 1.5% of users adopt the update in two months.
A different core
Devices. Aren’t. Cheap. Top spec Android phones, the BlackBerry Z10, Nokia Lumia 925 and pretty much anything created by Apple average at around £500-£600 per device. Luckily for us, Apple tends to only update their devices once a year and generally we know that a new version will mean that apps will run faster than before. Sometimes though, Apple likes to throw little curve balls with its software, which can sometimes require additional testing across devices.
A great example of this was the iOS bump from 5.x to 6.0.1 mid-2012. Apple had removed Google from the picture by booting their Maps service from iOS 6 and instead, launched their own mapping system. As a result, any application using the Google Maps API did not work as they had in the earlier 5.x OS version, and they needed to be switched over to a Google specific API or to Apple’s own version of maps.
Testing to a common denominator
At Mubaloo, we are continually monitoring the market and keeping up with the latest mobile trends. We forecast and review on a monthly basis asking questions like:
- Who are the upcoming stars on the mobile market?
- Which OS versions are still really popular popular? (Gingerbread, I’m looking at you!)
- Which OS versions are likely to be updated soon?
- How quickly will we need to adapt to these updates?
We have a unique maintenance process in place that provides us the ability to get hold of software updates as soon as they’re made available, giving us more time to investigate potential issues that come out of a software version bump.
9 times out of 10 we know in advance whether an update will have a negative or undesired effect on any of our apps, but there are the times where we simply need to sit down and test the updated platforms to see first hand. We’ve created a unique environment to allow our development and test teams to get the most out of all mobile devices available without blowing the budget.
Cloud-based app testing platforms are springing up everywhere at the moment. They offer up a huge range of dynamics that allow solo developers or agencies to test their apps across the massive market of mobile and tablet devices. Most are based on the Monkey framework that’s provided with the Android SDK and provide a simple ‘Install, test & report’ service that provides vital feedback about their application. This can range from crash logs, ANR instances to CPU performance and can also provide screen shots from all of the devices.
As mentioned earlier, cloud-based services are a fantastic approach to battling fragmentation for developers who do not have access to multiple device, but they cannot report back to you whether your app is doing the correct thing or not – this is where manual testing should take a priority.
At present we have access over 100 different devices, covering iOS, Android, Blackberry, Symbian and Windows 7 and 8. These support both old and current versions of operating systems. We believe it’s important to support as wide a market share as possible and with the processes we have in place, we are able to easily estimate the amount of testing and developer efforts needed to invest into an update. This makes it easier for us to judge test coverage and prioritise test efforts after the final software version has been released.
Let’s delve a little deeper! Below we’ve created a few visuals based on our own test lab and it seems that these correlate with the market as it stands.
Testing to destruction
Using the devices and insight we have at our disposal, we test our apps to destruction using manual, scripted, exploratory and risk-based testing. Here is a quick glimpse into our testing process:
- Our QA and Testing teams are integrated into each and every part of the project development life cycles.
- Static analysis on scoping and specification documents to identify defects within the documentation and as early on in the development process as possible.
- Determine testing scope and efforts via static analysis.
- Review and provide input into design phases for usability.
- Create and prepare detailed test scripts for formal manual testing.
- Prepare any automation material (defined in test planning phase).
- Execute manual, exploratory and automated tests.
- Defect logging and monitoring via project management platform.
All things considered, no matter how hard you work or research; fragmentation on platforms is here for good and there really isn’t much we can do to battle it – only adapt. In addition to our strict QA process, we test each and every one of our client’s requirements to ensure all of our mobile apps are performing in the best possible way.
By Daniel Box, Head of QA & Testing