Testing Mobile Applications Different from Testing Traditional Applications

By:Anurag Khode

Testing Mobile Applications Different from Testing Traditional Applications

This Article is By Lion Bridge

Just giving here to share his knowledge with us:


Due to the evolution of technology, mobile devices and their networks are becoming more complex and  require a great deal of testing to enhance their security and performance.

Executive Summary:

Pat leaves the house early to meet a friend for breakfast before work, but not early enough. On the way to the coffee shop, Pat sends a quick email message to his friend to say he's running 10 minutes late. When he arrives, his friend mentions that the car dealership next door has broadcast a special deal on the model Pat is looking for to his PDA via a Bluetooth advertising service. The coffee house provides an 802.11 WLAN access, so after breakfast, Pat takes a minute to check webmail before heading next door to check out the car sale.After checking out the car sale, Pat heads off on his office commute, but he has barely driven three blocks when the car's "low fuel" indicator begins to blink. Not being familiar with the area, Pat speaks into the hands free PC-phone to ask, "Where is the cheapest gas station?", and pulls to the side of the road while the device generates a local map with various options. Pat picks a station, gets directions, and is on track again.
Just as Pat drives up to the pump, an alert pops up on the Pocket PC and a synthesized voice provides a reminder that Mom's birthday is next week! While the gas flows, Pat uses a mobile browser to surf an array of merchants near Mom's hometown. After quickly browsing the stores, Pat looks up Mom's online gift registry service profile, chooses the right gift and purchases it using a digital wallet. Having quickly polished off these chores, Pat gets back on the road to work.The number and type of wireless applications envisioned for enterprise and mass market users grows daily. With each of these types of applications, there are a number of factors that must be considered when testing for conformance to functional, user and performance requirements.

Any good tester knows to pay attention to the basics, which can include:

• verifying the baseline functionality and features
• checking the design and proof-of-concept solutions against user requirements early in the development cycle
• testing under tightly controlled conditions to validate executable code against design during later stages of the development lifecycle
• compatibility testing all known, planned variations in the software and hardwareconfigurations where the application will run
• exposing the entire system or application to unexpected events, faults in dependentdatabases, networks or applications, or unpredictable user behavior
• subjecting the software to volume, load and stress conditions to gauge performance at the boundaries of its designed capacity and measure actual limitations of that performance
• determining if the application or system not only meets formal design requirements, but also whether it will be usable and meet the (perhaps undocumented) needs of its users.

Not all of these factors are within the scope of all testing engagements, but a good test plan will at
least address them so as to make all assumptions about test scope visible.When it comes to wireless enabled applications, however, there are factors that require the testing strategy to be modified significantly in order to ensure that hardware and software elements of the solution under test are suitable for its target environment and users.
A strategic approach to testing mobile solutions takes into account a number of characteristics
unique to the mobile paradigm:

• the increased complexity of emerging handheld devices
• the greater sensitivity to security and load related problems in wireless infrastructure
• increased complexities of scale

Increased Complexity in Handheld Devices
Perhaps no other area within the realm of mobile computing has changed quite as fast as the presentation device. The first handheld computing device to achieve any real market presence was the Apple Newton, introduced in 1993 and much beloved - but only by 200,000 people. Since then, this space has evolved to the point where there are literally thousands of different combination of hardware devices, software capabilities and wireless networking features. In fact, the handheld computing platform has now become a far more complex element of the overall mobile services technology chain on its own, than the entire networked-applications architecture of a generation ago.

Specific characteristics of the handheld device that make wireless computing
a challenge include:

• Device and firmware diversity: today's devices have not just firmware, but an operating system, browser and even an applications runtime layer (in the case of Java or BREW enabled phones). Recently, some very expensive and customer impacting quality problems for wireless users have been traced to firmware or operating system bugs in cell phones. Testing for compatibility between these layers, as well as with the wireless network interface, is paramount to deploying a trouble-free mobile service

• Support for multiple interface and rendering standards: device manufacturers are moving away from network specific handsets toward dual- or tri-mode phones capable of supporting multiple networks. And, despite the recently announced M-Services metastandard, applications rendered in HDML, WML, WAP 1.2, WAP2.0 and cHTML and xHTML will continue to require focused testing at the presentation device.
• Support for the embedded base: imagine desktop applications which were forced to be compatible with Windows 3.1, Win95, Win98, Win2K, Windows Me and Windows XP -plus the Macintosh. A similar challenge confronts developers and testers of applications targeted for handsets and PDAs with multiple, older versions of software.
  

• New paradigms in the client applications platform: the client software on a networkconnected PDA or cell phone was once relatively simple; most processing was performed in the network on WAP or custom mobile server platforms then rendered on the client over low speed, text- based interfaces to the device. With the advent of highly functional PDAs and handsets that have rich client execution environments and persistent data store, testing becomes much more complex.
  

• New security concerns: with the ability to download applications and execute code on the device comes the corresponding risk associated with user authentication, enterprise data and transactional security as well as viruses and other malicious code. The newfound power of the mobile desktop increases the need for testing and verification of security mechanisms, both in the devices themselves and in the network infrastructure.

Increased Complexity in Mobile Networks

Paging was once defined as a one-way service for short text messages, and cellular phones provided analog voice services only. Today, paging can be two-way, utilize many different network types (digital cellular, PCS, packet radio, etc) and offer the types of messaging services once reserved for enterprise private networks. Likewise, cellular isn't cellular anymore, with changes to the signaling, data transport and switching techniques that bring added complexity. Over the next five years, every major region of the world will undergo a network evolution that moves it through a series of technology upgrades on the way from second generation (2G) networks to third generation (3G). Along the way, we will see networks which now offer low speed (9.6 Kbps), circuit-switched services using dedicated cellular channels and wireless modems transformed into high speed (up to 2Mbps), packet-switched systems with direct digital interfaces. We will witness modifications of radio base station equipment, implementation of overlay networks and entirely new architectures and communications protocols (including IPv6, mandated by 3G standards), and deployment of signaling, protocol and applications gateways to move data between operator networks and public/private internet-based infrastructures. Changes in handheld devices will be required to accommodate this unsynchronized global shift, and verification of end to end services will be a challenge even for those with the proper access to in-country test facilities. Mobile applications will also need to survive cell or network handover problems which are bound to arise and the inevitable mid-session drop back to slower, second
generation networks lacking the signaling features of 3G. Lastly, the development of piconet technologies such as Bluetooth and Wi-Fi (802.11b) will soon lead to deployment of networks and access points for these unlicensed radio bands and inclusion of their protocol stacks into devices. For wireless applications, verification of interoperability alone will be a challenge. It is easy to see that both functional and performance testing of mobile applications prior to deployment will be a necessity for firms whose customers have come to
expect a seamless user experience, even while mobile.

Security, Performance and Scale
Although enterprise networks and applications have invested both time and money in recent years to beef up their infrastructure so they can handle security and performance challenges, many of the same issues have yet to be overcome in the mobile services world. Until now, the lower processing and datastore functionality of handhelds along with the network operators' unwillingness to open up more control of the wireless infrastructure have kept security concerns in check, for better or worse. However, concern over the robustness of WAP security mechanisms (Wireless Transport Layer Security, or WTLS), for example, has impacted enterprise user adoption. Small devices without the processing power to handle strong security methods such as RSA 1024 and SSL and the risk of loss and theft mean the challenge of deploying and testing for security in mobile devices is inherently different from that of other client devices. The move toward more robust operating systems, like Java, will bring both greater support for enhanced security such as SSL and IPSec and increased security risk of device-based processing and data store. Security concerns will only
increase with the availability of enterprise data extended to the device. Similar concerns relate to the expected increase in scale of mobile applications deployment, which will be much larger in the future than anything seen to date in the cellular data arena. Internet ready handhelds are becoming pervasive worldwide as content providers or enterprises can be reached from many parts of the globe. Therefore, it will be ever more critical for enterprises and mobile service providers alike to thoroughly test to ensure that performance levels both in the mobile network and the applications infrastructure remain acceptable.

Lessons for Testing Mobile Devices

If anyone doubts whether the deployment of next generation mobile services will require the support of knowledgeable test and quality assurance partners, the evidence is there to examine:

• Users of Nokia, Samsung, Sony and Matsushita handsets have encountered software bugs that have significantly impacted the quality of mobile voice and data services.
• An industry analyst covering Nokia reported that in a demonstration of GPRS network equipment, Nokia had to use Motorola handsets in the demo after discovering problems with the software in its own handsets
• 3G compatible handsets now reaching testbeds in the Tokyo area are buggy. According to Japanese daily newspapers, even when signals are reaching the phones, complex i-mode pages and not-so-complex Java applications are as reliable.Of course, bugs are expected during a testing period for new mobile networks and devices. But manufacturers and service providers must find and fix them early and cost effectively if they hope to retain user loyalty and market share.

Testing wireless services must address the following important considerations relative to
handheld presentation devices:

• Testers must pay more attention to usability and form factors than with traditional desktop applications. Smaller screens, slower processors, lack of persistent data store and lower bandwidth datalinks all require different testing methods. Testers must be aware of specific methods developers use to compensate for these factors and develop strategies to test those methods as well.

• Testers need to develop expertise in testing multiple layers of the mobile application's software model. The diversity of device hardware platforms, operating systems, microbrowsers and applications middleware require test experts to be aware of the compatibility issues impacting functionality and performance of mobile applications.
• Testers should prepare to modify test strategies to treat the handheld as a computing system itself, not as a simple client as has been the case with mobile services historically.
  

• Mobile applications require testers to increase the focus on end-to-end testing as interaction between handheld applications and enterprise data stores increases data processing complexity.

Lessons for Testing Enterprise Wireless Infrastructure

It's fair to say that to this point, the challenges of mobile applications testing have been largely in
the realm of device-specific hardware and software and the interfaces of those devices to wireless
networks. However, the greatest challenges are likely to be found in the deployment of next
generation networks and the accompanying services, protocols and wireless applications
middleware that will support them. Mobile subscribers have made it clear that behind coverage
and price, quality of service ranks as the third most important factor influencing customer
satisfaction and adoption.

The lessons here are:

• Testers should develop expertise with the unique network configurations in use.
• Thoroughly understand the network interfaces, protocols, gateways and firewalls in use and the technical issues that put their interoperability at risk. Without the ability to discern root defect causes in multiple levels of the wireless infrastructure, finger-pointing between carriers, service providers and developers abounds, and deployment costs soar.
• Utilize established testbeds within your target deployment area. This is particularly important for complex applications, as it allows testing under known network service quality conditions, testing application survivability during cell or network handovers and dropback to slower, second generation without needed signaling features. The technology and network implementations are simply not yet mature enough to rely solely on lab testing of mobile applications. Testing on a target instance, rather than a prototype or proof of concept, will enable proper testing ofs ecurity schemes (40-bit vs. 128-bit, network authentication & roaming, WTLS/SSL translations), and performance under realistic loads (web, client/server background loads, incremental wireless traffic, variations in message size/type).
  

•  Choose tools carefully. Few tools provide good support for wireless applications and infrastructure. Even where tools do exist, the level of test effort and success can vary widely with the type of gateway, complexity of the application and tool support for scripting, coding standards (WML, HDML, xHTML) and devices.
• Focus on key features of wireless middleware platforms. If your application allows commerce transactions and depends on automated billing, then placing a test emphasis on the real-time wireless billing features of your middleware platform and interfaces to corporate systems is critical. Likewise, solutions that rely on location-based technology should get special test attention to both the device side (e.g., a GPS chip) and network side components. Similar concerns should accompany test strategies for data synchronization (between mobile devices and enterprise datastores), mobile to nonmobile user collaboration, enterprise messaging, and content formatting/transcoding.

Lessons for Testing Mobile Services and Applications

The number and type of wireless applications cited at the beginning of this article indicate the
diversity of usage mobile networks will support in the future. In terms of functional verification of
these applications, testers need to keep in mind:

• Verification of end-to-end services will require access to in-country test facilities.
• Testing conducted through emulators or simulators in lab testbeds cannot accomplish the same level of quality verification of the user experience as field-based testing will.
• Emulators and simulators are useful for validating functionality and compatibility under controlled conditions, particularly during the development cycle and for white-box testing. However, field verification is necessary to ensure proper validation of services in a real world environment. Each unique configuration instance of a wireless application should be field tested as a separate solution (device, browser, data, network, carrier, gateway and enterprise components).
  

• Wireless WAN/PAN interoperability will require extra focused testing. Most wireless applications and services will be delivered through a combination of wide-area wireless networks (licensed spectrum) and personal area networks (PAN), such as Bluetooth and 802.11b, which operate on unlicensed spectrums. Ensuring service and application continuity across these networks is important.
  

• Test mobile applications in the same manner they are designed, but with tailored approaches. Despite attempts to the contrary, applications are still designed for a particular mobile device, and customized for the job function of each user. The same data residing within an inventory database, for example, might be accessible and displayed via separate applications:

o one for a non-mobile inventory clerk showing all inventory information
o one for a mobile sales representative showing just product availability
o one for a locally mobile stock picker showing warehouse location for product to
ship

As OracleMobile CTO Jacob Christfort asserts, "There's no free lunch." He proclaims it folly to
assume that "...you can take complicated business applications and simply transcode them." A
test strategy that takes into account the user experience of each unique mobile application and its
requirements is no less necessary than a design strategy for i-mode services that recognize the
attraction of downloadable cartoon characters and customizable ring tones. The key is to know
your audience, your user base, your application requirements and your infrastructure. Or risk a
repetition of the same mistakes that have stumped developers and integrators in the web and
client server domains.”