Due to the increasing penetration of cell phone-only households, survey administration via wireless devices is gaining attention. Regulations currently prohibit autodialing of cell phones, making administration by telephone cumbersome at best. This article focuses primarily upon use of the wireless web for survey administration in conjunction with SMS (Short Message Service). Survey functionality can mimic the experience of a traditional web survey although screen size severely limits information display. Significant issues including higher costs, lack of available panels, unknown incentive structures, and uncertain user acceptance need to be addressed, but applications exist today for survey administration on wireless devices. One of the most promising involves the use of camera phones for ethnography studies and longitudinal diaries. Future enhancements will enable new and unique applications that are not currently addressable by today’s survey technology solutions.
The “power of the thumb” refers to the use of a tool that is in the pocket or purse of most adults and many teenagers – the cell phone. An entire generation is being raised to send text messages, surf the web, and conduct transactions with the use of a few keys on their cell phones using their increasingly adept thumbs.
Survey methodologies have shifted rapidly as web-based administration has been more widely adopted. During the past decade, the use of telephone surveying has diminished, due to increased use of caller ID, call blocking, use of answering machines, and unpublished numbers. Many households that have a landline now use it as a multipurpose line – for fax and modem connectivity – often relying upon cell phones as a primary option for voice calls or for backup. These factors make it less likely that a dialed telephone household will respond on the first, or on any, call.
Concurrently the popularity of web surveying has increased as panels have improved in their level of representation. However, it is widely recognized that both methodologies omit key segments of the population. Now, the growing penetration of individuals that rely exclusively or primarily upon cell phones for their communication requirements complicates the inherent limitations of a single-mode survey solution even further.
In anticipation of increased demand for wireless survey administration, our firm has provided a wireless survey solution since early 2003. The views in this document are those of a software provider and deal with what is practical and technologically feasible today, as opposed to that which comprises an optimal survey methodology, a task that is left to others.
The paper is organized as follows: First, there is a review of the relevant literature, which leads to certain research questions. Then, there is the report on the development of the questionnaire, the sample, as well as the measures used in the survey. Data collection and analysis, as well as the results, are also discussed. Finally, there is a general discussion of the research findings, as well as a presentation of the limitations of the study and avenues for future research.
Current wireless solutions on the market support survey administration in one of four modes:
(1) In a single, real-time, continuous session via WAP and other supported wireless web protocols, supporting the viewing of digital images and other media within the session itself.
(2) In a discontinuous session administered via SMS (Short Message Service). As SMS does not support multimedia within the session, digital images and other media cannot be utilized within the survey itself.
(3) Via a non-real time solution by utilizing downloadable software, and for reasons of practicality generally requiring the use of a device dedicated to survey use.
(4) Via MMS (Multimedia Messaging Service). This mode supports the sending and receiving of digital images and other media within a survey question and/or response. MMS works like SMS, so questions and responses would be sent and received in discontinuous sessions.
The determination of which mode is most appropriate to use for any given survey project must be determined within the context of several factors, including technology adoption rates within the country or countries of use and within the demographic segments under study; the cost constraints both for the survey administrator and the underlying respondent base; the preferences of the respondent base; and the capabilities of the devices that will be used in the study.
In the United States, the use of the wireless web has been preferred over other modes for several reasons. First, it establishes a continuous session with no time delay between questions and receipt of responses (the receipt of a response being required to generate the next question), thereby shortening overall survey response time and encouraging higher completion rates (lower drop-out rates). Second, it is generally perceived as a more cost-effective solution from the respondent’s perspective, since the respondent must pay for each SMS or MMS message sent. (The cost to the respondent is approximately $.10/message per SMS and $.35 per MMS sent.) Third, it is more cost-effective to use the wireless web from an administrative standpoint. When using SMS, the survey administrator must establish an independent SMS gateway or utilize a third-party SMS gateway – which generally entails long-term, contractual commitments. Fourth, digital images and other media can be used within the survey itself, such as for branding or concept display (and contingent upon capacity constraints of the underlying network). Fifth, due to the low adoption of SMS in the United States (when compared with other regions of the world), the technical challenges imposed upon respondents remain virtually identical regardless of the wireless survey mode adopted.
However,
SMS is used a great deal (mostly in a non-research environment)
for simple polling, where the “survey” generally
consists of only one to a few short questions, and is usually
restricted to single choice (radio) questions or text-entry
questions. In this type of survey, the data is usually stored
in the database as an open-ended response, allowing no simple
real-time means of retrieval and analysis – whereas
the use of a conventional mode of administration – i.e.,
the web — enables administration of a variety of question
types with the ability to perform cross-tabulations and statistical
analyses “on the fly.” In Europe and other parts
of the world where mobile penetration rates exceed those in
the United States and/or SMS usage is high, SMS is usually
the preferred mode of wireless survey implementation in spite
of its limitations.
There are several barriers to successful implementation of
a survey via MMS including high costs for both sending and
receiving messages, the lack of a single, uniform standard
for multimedia messaging, and the lack of implemented gateways
for routing return messages.
The predominant use of wireless survey technology within the U.S. market research industry has been the wireless web approach described previously, as opposed to SMS administration. The primary reason for selecting the wireless web over SMS is one of cost primarily and functionality secondarily. By utilizing the wireless web, the vast majority of functions supported in a web or telephone survey are supported. For instance, it is possible to present concepts in a random order, anchoring specific concepts if desired. Conditional branching (making a question appear only for respondents on the basis of answers to one or more previous questions and/or stored database variables); validation (ensuring that responses fall within a range that is considered reasonable); quota administration (terminating respondents once a specific number of respondents of a given profile have completed a survey); displaying images and streaming media; branding the survey; and similar capabilities that are associated with web-based survey administration can all be programmed for wireless administration.
On the other hand, there will always be some functions that cannot easily be rendered on cell phones and other wireless devices. Cell phone browsers do not support embedding active javascript on a web page. In addition, even simple tables cannot be displayed on most cell phones (due to space considerations), and more complex tables cannot be displayed even on PDAs. Thus both technological and practical limitations exist, and often it is difficult for a researcher to understand what is pragmatic to implement without experimentation.
Auto-Dialing: Why not Just Call a Cell Phone?
In addition, wireless survey administration
can be conducted by simply dialing a cell phone and administering
the phone in voice mode and/or transferring the individual
to an IVR program once connected. Random digit dialing (RDD)
samples are selected from a base of telephone numbers that
exclude cell phone numbers, due to the fact that the Federal
Communications Commission (FCC) has specifically banned the
use of automated outdialing processes to cell phones and prosecuted
in some instances. Likewise, call lists purge cell phone numbers
for this same reason. The public is likely to support this
practice, as cell phone subscribers essentially pay, in one
form or fashion, for their inbound calls. In fact, for some
consumers, the relative privacy of the cell phone has become
a secondary reason (in addition to mobility) to adopt it as
a sole or primary communications device. Thus, to legally
conduct a survey among cell phone subscribers, their numbers
must be dialed manually, increasing administrative costs and
increasing the possibility of error in dialing regimen.
Assuming
legal barriers could be overcome, there are other reasons
why calling individuals on their cell phones to administer
a survey is impractical. These include the following:
Cell phone respondents incur a cost when taking a survey.
Unless an acceptable mode of compensation for the call is
provided, it is likely that respondents will resent the call,
either resulting in a higher disconnect rate or possibly in
a greater percentage of erroneous responses.
Cell phone respondents may be in any location when called,
not necessarily at home. This means that they are more likely
to be interrupted during another activity, again most likely
resulting in a higher disconnect rate.
Safety may be an issue if the cell phone respondent is driving,
engaged in operating machinery, or distracted from another
similar type of activity
ARBITRON conducted a survey to determine the feasibility of calling cell phone-only households, beginning with a pilot in 2002 and continuing in waves into 2005 (Fleeman 2005). Respondents were randomly called from a pool of 7,500 phone numbers (to comply with FCC regulations, numbers were hand-dialed) and administered a standard ARBITRON survey dealing with Radio Ratings. In both projects, ARBITRON found that the performance of the cell phone sample was about the same as those who were contacted via landline phones, and, significantly, that those respondents identifying their households as cell phone-only were more likely to be in two of ARBITRON’s lowest responding categories: men age 18-34, and women age 18-24.
Another means of utilizing the voice mechanism
for cell phones would be via Interactive Voice Response (IVR).
An SMS message could be sent to cell phones requesting that the
subscriber dial into an automated question-and-answer system.
While this approach poses no technological challenges, it requires
an opt-in list of cell phone subscribers, as the FCC has banned
cell phone spam. In addition, there has been no experimentation
with this technique, so estimated completion rates, necessary
incentive structures, and other aspects of administration are
unknown.
There are two slowly developing areas that could positively impact
the ability to administer surveys by cell phones. The first is
a reverse billing mechanism known as ‘Calling Party Pays’
(CPP) that essentially allows the caller, rather than the subscriber,
of a wireless call to pay the charges. Regulations for CPP are
not yet in place and, more significantly, neither are billing
mechanisms, which would need to be standardized and implemented
across a large number of billing vendors. However, if implemented
CPP would provide a mechanism for market researchers to pay for
their calls, potentially removing one barrier to acceptance of
wireless survey administration (either by voice or data), and
opening the door to removal of existing regulations prohibiting
such calls.
The second area of development that could impact uptake wireless
survey administration is mobile advertising. While implementation
of any advertising to cell phones has been extremely cautious,
several companies are promoting free or reduced price phones,
airtime, or other giveaways in exchange for listening to or viewing
advertising while a phone rings, or accepting occasional text
messages. This type of forum would allow solicitation of cell
phone subscribers into opt-in panels or for one-time projects.
However, acceptance of wireless advertising has been low and it
could reasonably be argued that this base of wireless subscribers
is unlikely to be representative of the overall user base.
What motives does the market research
industry have to matriculate to wireless devices for survey administration?
While there remain many technology challenges within the industry,
the primary driver today is the desire to obtain a representative
sample for political, social, and scientific studies.
A multimodal survey approach and/or door-to-door methodology has
always been required to garner statistical representation among
specific demographic segments, such the elderly, rural, and households
at either end of the income curve. While there have always been
concerns about representation among households without telephones,
the sampling dilemma is exacerbated by the rapidly growing number
of adults living in households where only a wireless telephone
is used. Random-digit-dialing (RDD) surveys do not call wireless
phone numbers, in part because so doing incurs a cost on the part
of the respondent. The wireless-only household represents a skewed
demographic – namely, primarily urban, young, and single.
The Centers for Disease Control and Prevention indicated earlier
this year that it believes that the number of adults living in
wireless-only households has doubled in less than two years, and
currently represents approximately 6.1% of U.S. households (Blumberg,
Luke, and Cynamon 2005). The prevalence increases for adults under
the age of 25 to 14.2%. The CDC found that wireless-only households
were skewed in gender, racial, income, and other factors. Undoubtedly,
with wireless-only usage often representing a lifestyle choice,
the underlying bias will have potential implications for a broad
range of survey research findings. On the one hand, determining
a method for reaching individuals via wireless devices has become
a significant issue for some types of survey research, while on
the other, it only complicates the manner in which data is collected
in the future because findings from surveys implemented via wireless
devices would only bias research findings unless used in conjunction
with other modes. Thus, many in the social sciences view wireless
survey administration as a means of augmenting existing survey
modes, not as a replacement for other modes of research.
Within the U.S. market research industry, panels are typically
used as the means of obtaining an opt-in group of individuals
for surveying. These individuals are compensated in various ways
– through contests, point schemes that accrue toward cash
or prizes, or by electronic gift certificates. Such opt-in panels
do not exist in the survey research world, although there are
several established SMS communities that might represent a type
of “list” substitute. However, SMS communities are
by their very nature biased and focused upon specific types of
content (such as dating, or hip-hop music) and have not opted
into the survey-taking process. The entire area of how best to
develop, maintain, and provide incentives to wireless survey respondents
is virtually unexplored within the United States, although there
has been some exploration of how this could be accomplished within
Europe.
In spite of the many challenges to implementation
of wireless surveys, areas of promise are emerging. One application
utilizes the camera phone for multimode ethnographies. In these
applications, respondents may either utilize their own camera
phones or they are provided a camera phone for their own use.
Ethnographies typically focus upon documenting the lifestyle of
the survey participants over a prolonged period of time. Often
the camera phone is but one aspect of the study – they may
also be interviewed at length, participate in focus groups, and/or
be “followed” in some other way, so that the wireless
technology is only a part of the overall research methodology.
In particular, camera phones have proven motivational to teens
for documenting their lifestyle. Easily bored, teens are notoriously
poor respondents in longitudinal studies such as ethnographies
and diaries but are enthusiastic participants when a camera phone
and the novelty of wireless research are in their hands. Could
this novelty be sustained over time, and over what time period?
The answers to these questions are unknown, but in one-time studies
they have shown great promise.
In typical camera phone ethnography, respondents may be recruited
by any traditional fashion (phone, web, or mall intercept). Respondents
are usually provided verbal and/or written instructions regarding
the purpose of the study and what will be expected of them as
research participants. Typically they are asked to submit media
in the form of digital images, short videos, sound recordings,
and/or text messages at certain intervals throughout the days.
These might be triggered by events – such as consuming a
snack or playing a video game – or they might be regular
occurrences. As an optional part of the ethnography, respondents
can log into a web site with their camera phones and take a survey
– which may not be a questionnaire so much as information
to place the media in a desired context (such as indicating where
they are or who they were with when they submitted the media).
This approach is less intrusive than other modes of ethnographies
– such as trailing a respondent with a video camera and
tape recorder. Because media and survey responses arrive in real
time, researchers are able to track the level of cooperation among
respondents to ensure compliance.
In addition, SMS is available as an “invitation” management
option (plain text and html web invitations are also supported).
In camera phone ethnography, SMS messages can be sent to respondents
to inform them of daily tasks and to serve as reminder messages
when these tasks are not being completed. Because ethnographies
are focused on the collection of qualitative rather than quantitative
data, it may be preferable in some studies to customize the messages
that are sent to respondents, so that the researcher interacts
with the respondent but in a less intrusive way than if tagging
that individual.
There are three primary mechanisms for sending media in for reporting
purposes. The first way is to use MMS, attaching the media with
a text message. While available software supports this approach,
other approaches provide significant advantages. MMS costs the
sender approximately $.35 per message, and respondents are generally
limited in terms of maximum file size.
Since most researchers do not want their respondents timing a
video on the basis of the capabilities of the wireless network,
the second and currently preferred approach is to use the device’s
email sending capabilities. When attaching media to an email message,
there is typically no inherent file size limitation (or the file
size limitation is much larger). In addition, the incoming media
have a higher resolution and are more appropriate for inserting
in research findings and presentations. Furthermore, the cost
is lower, since email is usually bundled in with a data rate plan
and does not incur a per-message fee.
A third means of sending media is to use an upload feature within
the survey itself. In the United States, very few phones on the
market currently support this feature, although it is expected
to become more commonplace over time.
Once captured, media must be organized in a manner in which researchers
can make use of them. In the Mobile Memoir system, all media are
stamped with a time and date and associated with a set of survey
responses (if available). They are sorted by type of media and
can be accessed by a double-click of the mouse. Summary statistics
show the number of media submitted by each respondent.
Similarly, wireless surveys are effective when administered for
longitudinal diaries, particularly for impulse purchases that
respondents often forget to log in conventional diaries (paper-based
and/or web). The cell phone is always with the subscriber, making
it convenient to log in and report a purchase at the time of purchase.
Other situational aspects of the purchase are more easily reported
in this manner, since these would often be forgotten later (“What
shelf was it on?” “Was it on sale?” “Did
you use a coupon?”), and the submission of media can be
used to augment diary entries. In addition, SMS can be used to
automatically remind non-respondents when their reporting activity
lapses.
Similarly, mystery shopping represents an established field of
research which easily lends itself to use of wireless devices.
A cell phone can be used without attracting attention and can
be used as a recording instrument in a variety of ways (web forms,
sound recordings, digital images and videos).
SMS, as a stand-alone offering, represents a way to invite or
remind respondents that they have surveys or other research tasks
that require their attention. These surveys may be completed via
the web or some other fashion. SMS can cut through the clutter
of too many emails which are easily confused with spam, even though
the respondent may not utilize their wireless device to complete
the research itself
Research conducted utilizing wireless devices is in a nascent state today and presents many challenges of a technological, regulatory, methodological, and behavioral nature. However, it also presents great potential to solve problems within the research industry. Consider some of the following scenarios for future research, some of which are feasible today but not functionally developed, optimized, or assessed for feasibility.
Using location determination, each survey reports the geographic coordinates of respondents as they take surveys. Just as today’s surveys provide a time/date stamp for responses, a location stamp would also be provided. An automobile manufacturer could request a respondent to complete a diary entry each time they are at a location where they are evaluating a new car purchase. The results could be street-mapped to specific dealerships. Location stamps could provide an entirely new dimension to understanding the distances that individuals are willing to go to travel, the time that they take shopping for items, and possibly the routes that they take to arrive there.
There may also be great potential in wireless scanning solutions. Several research companies provide detailed data from extensive panels of grocery shoppers. Today’s technology generally utilizes at-home scanners. These successfully capture the bulk of grocery purchases made at grocery stores, Wal-Mart and other superstores, but often fail to capture the (frequently impulse-oriented) purchases made at convenience stores, health food stores, vending machines, and other locations. A wireless scanner can be attached to a cell phone through the infrared port or through a serial cable, or smaller handled scanners exist which can store a limited number of bar codes for later downloading. Cost studies, performance comparisons, and respondent completion rates need to be assessed before such technologies can be implemented, but they may be feasible for certain types of products in the short term.
Advances in both the wireless networks themselves and in the devices used will have significant impact upon the feasibility of migrating some research to wireless devices in the future. The implementation of Wi-Fi (wireless LANs or “hot spots”) could mean that there is no longer a cost associated with wireless data over cell phones and PDAs in the future, or that the price model changes to something comparable to today’s internet pricing. Such price changes would greatly enhance the viability of one-time (as opposed to longitudinal) studies. The use of Bluetooth (a very short-range “Personal Area Network”) can be used interactively in cross-marketing and research campaigns by triggering a message to a device that is located within a store or other on-site research location. Bluetooth enables the automated administration of surveys, couponing, and other activities, potentially replacing low-response IVR surveys and in-person interviewers, and greatly decreasing the cost of some types of data collection.
Megapixel camera phones are appearing on the market with DVD quality. So are embedded MP3 players and Macromedia Flash. The implications are that future surveys can become just as media-rich as today’s Internet surveys, providing an extremely compelling survey experience. However, the use of such capabilities is best suited to high-speed networks, such as Wi-Fi or 3XRTT, which are not accessible to most U.S. cellular subscribers today. It should be noted that while today’s networks support web and SMS surveys at very acceptable speeds, media, and in particular videos, are slower.
There are research organizations that specialize in web and application usability studies. These are virtually nonexistent today for wireless web applications. It is feasible on some phones today, and on most new devices on the market to send a java application for downloading, either via email, MMS, or within the survey itself, for a rating or review process. It is also possible to integrate a survey within the web surfing experience, so that the user is interrupted periodically to rate wireless web sites.
Wireless survey solutions are likely to be implemented in a more complex and application-specific fashion than their web counterparts. However, it is already certain that the future of research activity will include the harnessing of “the power of the thumb.”
Blumberg, Stephen J., Julian V. Luke, and Marcie L. Cynamon (2005), “The Prevalence and Impact of Wireless Substitution: Updated Data from the 2004 National Health Interview Survey,” Centers for Disease Control and Prevention, presented at the Annual conference of the American Association for Public Opinion Research, May 14, 2005.
Fleeman, Anna (2005), “Will They Talk to Us? Survey Performance Rates of a Cell Phone Sample,” ARBITRON, Research Methods Development & Evaluation, presented at the Annual conference of the American Association for Public Opinion Research, May 14, 2005.
Leslie Townsend is President and Founder of Mobile Memoir, LLC which provides software that enables data collection via the web and to wireless devices. She may be reached for comments at [email protected].