The Status of Wireless Survey Solutions:
The Emerging “Power of the Thumb”

Leslie Townsend

Mobile Memoir LLC

Abstract

Due to the increasing penetration of cell phone-only households, survey administration via wireless devices is gaining attention. Regulations currently prohibit autodialing of cell phones, making administration by telephone cumbersome at best. This article focuses primarily upon use of the wireless web for survey administration in conjunction with SMS (Short Message Service). Survey functionality can mimic the experience of a traditional web survey although screen size severely limits information display. Significant issues including higher costs, lack of available panels, unknown incentive structures, and uncertain user acceptance need to be addressed, but applications exist today for survey administration on wireless devices. One of the most promising involves the use of camera phones for ethnography studies and longitudinal diaries. Future enhancements will enable new and unique applications that are not currently addressable by today’s survey technology solutions.

Introduction

The “power of the thumb” refers to the use of a tool that is in the pocket or purse of most adults and many teenagers – the cell phone. An entire generation is being raised to send text messages, surf the web, and conduct transactions with the use of a few keys on their cell phones using their increasingly adept thumbs.

Survey methodologies have shifted rapidly as web-based administration has been more widely adopted. During the past decade, the use of telephone surveying has diminished, due to increased use of caller ID, call blocking, use of answering machines, and unpublished numbers. Many households that have a landline now use it as a multipurpose line – for fax and modem connectivity – often relying upon cell phones as a primary option for voice calls or for backup. These factors make it less likely that a dialed telephone household will respond on the first, or on any, call.

Concurrently the popularity of web surveying has increased as panels have improved in their level of representation. However, it is widely recognized that both methodologies omit key segments of the population. Now, the growing penetration of individuals that rely exclusively or primarily upon cell phones for their communication requirements complicates the inherent limitations of a single-mode survey solution even further.

In anticipation of increased demand for wireless survey administration, our firm has provided a wireless survey solution since early 2003. The views in this document are those of a software provider and deal with what is practical and technologically feasible today, as opposed to that which comprises an optimal survey methodology, a task that is left to others.

The paper is organized as follows: First, there is a review of the relevant literature, which leads to certain research questions. Then, there is the report on the development of the questionnaire, the sample, as well as the measures used in the survey. Data collection and analysis, as well as the results, are also discussed. Finally, there is a general discussion of the research findings, as well as a presentation of the limitations of the study and avenues for future research.

Survey Administration Modes for Cell Phones, PDAs, and Other Wirelessly- Enabled Devices

Current wireless solutions on the market support survey administration in one of four modes:

(1) In a single, real-time, continuous session via WAP and other supported wireless web protocols, supporting the viewing of digital images and other media within the session itself.
(2) In a discontinuous session administered via SMS (Short Message Service). As SMS does not support multimedia within the session, digital images and other media cannot be utilized within the survey itself.
(3) Via a non-real time solution by utilizing downloadable software, and for reasons of practicality generally requiring the use of a device dedicated to survey use.
(4) Via MMS (Multimedia Messaging Service). This mode supports the sending and receiving of digital images and other media within a survey question and/or response. MMS works like SMS, so questions and responses would be sent and received in discontinuous sessions.

The determination of which mode is most appropriate to use for any given survey project must be determined within the context of several factors, including technology adoption rates within the country or countries of use and within the demographic segments under study; the cost constraints both for the survey administrator and the underlying respondent base; the preferences of the respondent base; and the capabilities of the devices that will be used in the study.

In the United States, the use of the wireless web has been preferred over other modes for several reasons. First, it establishes a continuous session with no time delay between questions and receipt of responses (the receipt of a response being required to generate the next question), thereby shortening overall survey response time and encouraging higher completion rates (lower drop-out rates). Second, it is generally perceived as a more cost-effective solution from the respondent’s perspective, since the respondent must pay for each SMS or MMS message sent. (The cost to the respondent is approximately $.10/message per SMS and $.35 per MMS sent.) Third, it is more cost-effective to use the wireless web from an administrative standpoint. When using SMS, the survey administrator must establish an independent SMS gateway or utilize a third-party SMS gateway – which generally entails long-term, contractual commitments. Fourth, digital images and other media can be used within the survey itself, such as for branding or concept display (and contingent upon capacity constraints of the underlying network). Fifth, due to the low adoption of SMS in the United States (when compared with other regions of the world), the technical challenges imposed upon respondents remain virtually identical regardless of the wireless survey mode adopted.

However, SMS is used a great deal (mostly in a non-research environment) for simple polling, where the “survey” generally consists of only one to a few short questions, and is usually restricted to single choice (radio) questions or text-entry questions. In this type of survey, the data is usually stored in the database as an open-ended response, allowing no simple real-time means of retrieval and analysis – whereas the use of a conventional mode of administration – i.e., the web — enables administration of a variety of question types with the ability to perform cross-tabulations and statistical analyses “on the fly.” In Europe and other parts of the world where mobile penetration rates exceed those in the United States and/or SMS usage is high, SMS is usually the preferred mode of wireless survey implementation in spite of its limitations.

There are several barriers to successful implementation of a survey via MMS including high costs for both sending and receiving messages, the lack of a single, uniform standard for multimedia messaging, and the lack of implemented gateways for routing return messages.

Survey Functionality

The predominant use of wireless survey technology within the U.S. market research industry has been the wireless web approach described previously, as opposed to SMS administration. The primary reason for selecting the wireless web over SMS is one of cost primarily and functionality secondarily. By utilizing the wireless web, the vast majority of functions supported in a web or telephone survey are supported. For instance, it is possible to present concepts in a random order, anchoring specific concepts if desired. Conditional branching (making a question appear only for respondents on the basis of answers to one or more previous questions and/or stored database variables); validation (ensuring that responses fall within a range that is considered reasonable); quota administration (terminating respondents once a specific number of respondents of a given profile have completed a survey); displaying images and streaming media; branding the survey; and similar capabilities that are associated with web-based survey administration can all be programmed for wireless administration.

On the other hand, there will always be some functions that cannot easily be rendered on cell phones and other wireless devices. Cell phone browsers do not support embedding active javascript on a web page. In addition, even simple tables cannot be displayed on most cell phones (due to space considerations), and more complex tables cannot be displayed even on PDAs. Thus both technological and practical limitations exist, and often it is difficult for a researcher to understand what is pragmatic to implement without experimentation.

Auto-Dialing: Why not Just Call a Cell Phone?

In addition, wireless survey administration can be conducted by simply dialing a cell phone and administering the phone in voice mode and/or transferring the individual to an IVR program once connected. Random digit dialing (RDD) samples are selected from a base of telephone numbers that exclude cell phone numbers, due to the fact that the Federal Communications Commission (FCC) has specifically banned the use of automated outdialing processes to cell phones and prosecuted in some instances. Likewise, call lists purge cell phone numbers for this same reason. The public is likely to support this practice, as cell phone subscribers essentially pay, in one form or fashion, for their inbound calls. In fact, for some consumers, the relative privacy of the cell phone has become a secondary reason (in addition to mobility) to adopt it as a sole or primary communications device. Thus, to legally conduct a survey among cell phone subscribers, their numbers must be dialed manually, increasing administrative costs and increasing the possibility of error in dialing regimen.

Assuming legal barriers could be overcome, there are other reasons why calling individuals on their cell phones to administer a survey is impractical. These include the following:

Cell phone respondents incur a cost when taking a survey. Unless an acceptable mode of compensation for the call is provided, it is likely that respondents will resent the call, either resulting in a higher disconnect rate or possibly in a greater percentage of erroneous responses.

Cell phone respondents may be in any location when called, not necessarily at home. This means that they are more likely to be interrupted during another activity, again most likely resulting in a higher disconnect rate.

Safety may be an issue if the cell phone respondent is driving, engaged in operating machinery, or distracted from another similar type of activity

ARBITRON conducted a survey to determine the feasibility of calling cell phone-only households, beginning with a pilot in 2002 and continuing in waves into 2005 (Fleeman 2005). Respondents were randomly called from a pool of 7,500 phone numbers (to comply with FCC regulations, numbers were hand-dialed) and administered a standard ARBITRON survey dealing with Radio Ratings. In both projects, ARBITRON found that the performance of the cell phone sample was about the same as those who were contacted via landline phones, and, significantly, that those respondents identifying their households as cell phone-only were more likely to be in two of ARBITRON’s lowest responding categories: men age 18-34, and women age 18-24.

Another means of utilizing the voice mechanism for cell phones would be via Interactive Voice Response (IVR). An SMS message could be sent to cell phones requesting that the subscriber dial into an automated question-and-answer system. While this approach poses no technological challenges, it requires an opt-in list of cell phone subscribers, as the FCC has banned cell phone spam. In addition, there has been no experimentation with this technique, so estimated completion rates, necessary incentive structures, and other aspects of administration are unknown.

There are two slowly developing areas that could positively impact the ability to administer surveys by cell phones. The first is a reverse billing mechanism known as ‘Calling Party Pays’ (CPP) that essentially allows the caller, rather than the subscriber, of a wireless call to pay the charges. Regulations for CPP are not yet in place and, more significantly, neither are billing mechanisms, which would need to be standardized and implemented across a large number of billing vendors. However, if implemented CPP would provide a mechanism for market researchers to pay for their calls, potentially removing one barrier to acceptance of wireless survey administration (either by voice or data), and opening the door to removal of existing regulations prohibiting such calls.

The second area of development that could impact uptake wireless survey administration is mobile advertising. While implementation of any advertising to cell phones has been extremely cautious, several companies are promoting free or reduced price phones, airtime, or other giveaways in exchange for listening to or viewing advertising while a phone rings, or accepting occasional text messages. This type of forum would allow solicitation of cell phone subscribers into opt-in panels or for one-time projects. However, acceptance of wireless advertising has been low and it could reasonably be argued that this base of wireless subscribers is unlikely to be representative of the overall user base.

Panel and Incentive Issues

What motives does the market research industry have to matriculate to wireless devices for survey administration? While there remain many technology challenges within the industry, the primary driver today is the desire to obtain a representative sample for political, social, and scientific studies.

A multimodal survey approach and/or door-to-door methodology has always been required to garner statistical representation among specific demographic segments, such the elderly, rural, and households at either end of the income curve. While there have always been concerns about representation among households without telephones, the sampling dilemma is exacerbated by the rapidly growing number of adults living in households where only a wireless telephone is used. Random-digit-dialing (RDD) surveys do not call wireless phone numbers, in part because so doing incurs a cost on the part of the respondent. The wireless-only household represents a skewed demographic – namely, primarily urban, young, and single. The Centers for Disease Control and Prevention indicated earlier this year that it believes that the number of adults living in wireless-only households has doubled in less than two years, and currently represents approximately 6.1% of U.S. households (Blumberg, Luke, and Cynamon 2005). The prevalence increases for adults under the age of 25 to 14.2%. The CDC found that wireless-only households were skewed in gender, racial, income, and other factors. Undoubtedly, with wireless-only usage often representing a lifestyle choice, the underlying bias will have potential implications for a broad range of survey research findings. On the one hand, determining a method for reaching individuals via wireless devices has become a significant issue for some types of survey research, while on the other, it only complicates the manner in which data is collected in the future because findings from surveys implemented via wireless devices would only bias research findings unless used in conjunction with other modes. Thus, many in the social sciences view wireless survey administration as a means of augmenting existing survey modes, not as a replacement for other modes of research.

Within the U.S. market research industry, panels are typically used as the means of obtaining an opt-in group of individuals for surveying. These individuals are compensated in various ways – through contests, point schemes that accrue toward cash or prizes, or by electronic gift certificates. Such opt-in panels do not exist in the survey research world, although there are several established SMS communities that might represent a type of “list” substitute. However, SMS communities are by their very nature biased and focused upon specific types of content (such as dating, or hip-hop music) and have not opted into the survey-taking process. The entire area of how best to develop, maintain, and provide incentives to wireless survey respondents is virtually unexplored within the United States, although there has been some exploration of how this could be accomplished within Europe.

Applications

In spite of the many challenges to implementation of wireless surveys, areas of promise are emerging. One application utilizes the camera phone for multimode ethnographies. In these applications, respondents may either utilize their own camera phones or they are provided a camera phone for their own use. Ethnographies typically focus upon documenting the lifestyle of the survey participants over a prolonged period of time. Often the camera phone is but one aspect of the study – they may also be interviewed at length, participate in focus groups, and/or be “followed” in some other way, so that the wireless technology is only a part of the overall research methodology. In particular, camera phones have proven motivational to teens for documenting their lifestyle. Easily bored, teens are notoriously poor respondents in longitudinal studies such as ethnographies and diaries but are enthusiastic participants when a camera phone and the novelty of wireless research are in their hands. Could this novelty be sustained over time, and over what time period? The answers to these questions are unknown, but in one-time studies they have shown great promise.

In typical camera phone ethnography, respondents may be recruited by any traditional fashion (phone, web, or mall intercept). Respondents are usually provided verbal and/or written instructions regarding the purpose of the study and what will be expected of them as research participants. Typically they are asked to submit media in the form of digital images, short videos, sound recordings, and/or text messages at certain intervals throughout the days. These might be triggered by events – such as consuming a snack or playing a video game – or they might be regular occurrences. As an optional part of the ethnography, respondents can log into a web site with their camera phones and take a survey – which may not be a questionnaire so much as information to place the media in a desired context (such as indicating where they are or who they were with when they submitted the media). This approach is less intrusive than other modes of ethnographies – such as trailing a respondent with a video camera and tape recorder. Because media and survey responses arrive in real time, researchers are able to track the level of cooperation among respondents to ensure compliance.

In addition, SMS is available as an “invitation” management option (plain text and html web invitations are also supported). In camera phone ethnography, SMS messages can be sent to respondents to inform them of daily tasks and to serve as reminder messages when these tasks are not being completed. Because ethnographies are focused on the collection of qualitative rather than quantitative data, it may be preferable in some studies to customize the messages that are sent to respondents, so that the researcher interacts with the respondent but in a less intrusive way than if tagging that individual.

There are three primary mechanisms for sending media in for reporting purposes. The first way is to use MMS, attaching the media with a text message. While available software supports this approach, other approaches provide significant advantages. MMS costs the sender approximately $.35 per message, and respondents are generally limited in terms of maximum file size.

Since most researchers do not want their respondents timing a video on the basis of the capabilities of the wireless network, the second and currently preferred approach is to use the device’s email sending capabilities. When attaching media to an email message, there is typically no inherent file size limitation (or the file size limitation is much larger). In addition, the incoming media have a higher resolution and are more appropriate for inserting in research findings and presentations. Furthermore, the cost is lower, since email is usually bundled in with a data rate plan and does not incur a per-message fee.

A third means of sending media is to use an upload feature within the survey itself. In the United States, very few phones on the market currently support this feature, although it is expected to become more commonplace over time.

Once captured, media must be organized in a manner in which researchers can make use of them. In the Mobile Memoir system, all media are stamped with a time and date and associated with a set of survey responses (if available). They are sorted by type of media and can be accessed by a double-click of the mouse. Summary statistics show the number of media submitted by each respondent.

Similarly, wireless surveys are effective when administered for longitudinal diaries, particularly for impulse purchases that respondents often forget to log in conventional diaries (paper-based and/or web). The cell phone is always with the subscriber, making it convenient to log in and report a purchase at the time of purchase. Other situational aspects of the purchase are more easily reported in this manner, since these would often be forgotten later (“What shelf was it on?” “Was it on sale?” “Did you use a coupon?”), and the submission of media can be used to augment diary entries. In addition, SMS can be used to automatically remind non-respondents when their reporting activity lapses.

Similarly, mystery shopping represents an established field of research which easily lends itself to use of wireless devices. A cell phone can be used without attracting attention and can be used as a recording instrument in a variety of ways (web forms, sound recordings, digital images and videos).

SMS, as a stand-alone offering, represents a way to invite or remind respondents that they have surveys or other research tasks that require their attention. These surveys may be completed via the web or some other fashion. SMS can cut through the clutter of too many emails which are easily confused with spam, even though the respondent may not utilize their wireless device to complete the research itself

Where Do We Take it From Here?

Research conducted utilizing wireless devices is in a nascent state today and presents many challenges of a technological, regulatory, methodological, and behavioral nature. However, it also presents great potential to solve problems within the research industry. Consider some of the following scenarios for future research, some of which are feasible today but not functionally developed, optimized, or assessed for feasibility.

Using location determination, each survey reports the geographic coordinates of respondents as they take surveys. Just as today’s surveys provide a time/date stamp for responses, a location stamp would also be provided. An automobile manufacturer could request a respondent to complete a diary entry each time they are at a location where they are evaluating a new car purchase. The results could be street-mapped to specific dealerships. Location stamps could provide an entirely new dimension to understanding the distances that individuals are willing to go to travel, the time that they take shopping for items, and possibly the routes that they take to arrive there.

There may also be great potential in wireless scanning solutions. Several research companies provide detailed data from extensive panels of grocery shoppers. Today’s technology generally utilizes at-home scanners. These successfully capture the bulk of grocery purchases made at grocery stores, Wal-Mart and other superstores, but often fail to capture the (frequently impulse-oriented) purchases made at convenience stores, health food stores, vending machines, and other locations. A wireless scanner can be attached to a cell phone through the infrared port or through a serial cable, or smaller handled scanners exist which can store a limited number of bar codes for later downloading. Cost studies, performance comparisons, and respondent completion rates need to be assessed before such technologies can be implemented, but they may be feasible for certain types of products in the short term.

Advances in both the wireless networks themselves and in the devices used will have significant impact upon the feasibility of migrating some research to wireless devices in the future. The implementation of Wi-Fi (wireless LANs or “hot spots”) could mean that there is no longer a cost associated with wireless data over cell phones and PDAs in the future, or that the price model changes to something comparable to today’s internet pricing. Such price changes would greatly enhance the viability of one-time (as opposed to longitudinal) studies. The use of Bluetooth (a very short-range “Personal Area Network”) can be used interactively in cross-marketing and research campaigns by triggering a message to a device that is located within a store or other on-site research location. Bluetooth enables the automated administration of surveys, couponing, and other activities, potentially replacing low-response IVR surveys and in-person interviewers, and greatly decreasing the cost of some types of data collection.

Megapixel camera phones are appearing on the market with DVD quality. So are embedded MP3 players and Macromedia Flash. The implications are that future surveys can become just as media-rich as today’s Internet surveys, providing an extremely compelling survey experience. However, the use of such capabilities is best suited to high-speed networks, such as Wi-Fi or 3XRTT, which are not accessible to most U.S. cellular subscribers today. It should be noted that while today’s networks support web and SMS surveys at very acceptable speeds, media, and in particular videos, are slower.

There are research organizations that specialize in web and application usability studies. These are virtually nonexistent today for wireless web applications. It is feasible on some phones today, and on most new devices on the market to send a java application for downloading, either via email, MMS, or within the survey itself, for a rating or review process. It is also possible to integrate a survey within the web surfing experience, so that the user is interrupted periodically to rate wireless web sites.

Wireless survey solutions are likely to be implemented in a more complex and application-specific fashion than their web counterparts. However, it is already certain that the future of research activity will include the harnessing of “the power of the thumb.”

Reference

Blumberg, Stephen J., Julian V. Luke, and Marcie L. Cynamon (2005), “The Prevalence and Impact of Wireless Substitution: Updated Data from the 2004 National Health Interview Survey,” Centers for Disease Control and Prevention, presented at the Annual conference of the American Association for Public Opinion Research, May 14, 2005.

Fleeman, Anna (2005), “Will They Talk to Us? Survey Performance Rates of a Cell Phone Sample,” ARBITRON, Research Methods Development & Evaluation, presented at the Annual conference of the American Association for Public Opinion Research, May 14, 2005.

About the Authors

Leslie Townsend is President and Founder of Mobile Memoir, LLC which provides software that enables data collection via the web and to wireless devices. She may be reached for comments at [email protected]