Measuring the quality of the
on-line service experience: inferred versus direct disconfirmation modelling
Martin
O’Neill Ph.D
Adrian Palmer
Interest in the measurement of on-line quality is understandably high and measuring the quality of the on-line service experience is now an integral part of most IT managers’ responsibilities. The challenge facing them, however is to identify and implement the most appropriate measurement tools for their operation. This has led to the development of a multitude of measurement techniques aimed at assessing the quality of the on-line service experience. This is well evidenced in both the practitioner and academic literature, and to the uninitiated is a veritable minefield of terminology. In highlighting the importance of service quality to the IT sector, this paper seeks to compare and critically examine two of the more prominent evaluation techniques: inferred and direct disconfirmation models and their usefulness in evaluating the on-line service experience.
Key
words
(Service Quality, Evaluation,
Disconfirmation Models, Internet)
Introduction
In an increasingly competitive business environment, the issues of quality service and service excellence are becoming increasingly more important. This is especially true of the IT sector, where an ever increasing range and number of on-line service suppliers has forced companies to invest in the delivery of higher levels of service quality as a competitive strategy aimed at differentiating their product offering. . Inherent in approaches to continual quality improvement is the need to continually monitor organisational performance so that energies can be better directed at consistently satisfying and keeping up with customer needs. This is especially true for on-line service suppliers, where other than the company’s homepage, there are few other tangible indicators of the service to be received. This has led to the development of a plethora of measurement tools and techniques aimed at assessing on-line service quality . This paper focuses on the quality of users’ online experience, which is defined here as some measure of the performance of the online medium. The online experience may facilitate other offline service delivery, but this research conceptually seeks to separate such offline delivery from activities that occur while an individual is interacting with a company through the medium of the Internet.
At the forefront of such developments has been the use of disconfirmation models, which seek to define service quality as the difference between user expectations of the service to be received and their perceptions of the service actually received. This paper compares two such models (inferred and direct disconfirmation models) for measuring quality of the online experience . There has been debate on the reliability and practical value of both techniques and this paper seeks to add to this debate. In the search for a reliable and valid measure of quality of the online experience , this paper addresses the psychometric and diagnostic performance of each technique. It reports the findings from an exploratory study of user perceptions of quality of the online experience as it relates to the use of on-line library services at a university . The results reveal those dimensions of significance to on-line consumers and demonstrates the usefulness and relative simplicity of both techniques for evaluating the service quality construct in on-line environments.
Research
issues
The development of IT and corresponding increase in
Internet use has offered new opportunities for service firms to develop their
service offerings (Bitner, Brown and Meuter, 2000), and gives
customers the means to access the
services of a manufacturer or service firm far more quickly and easily
(Groonross, 2001, p.172). In a society where time is often viewed as being
every bit as valuable as money, it is not surprising that there should be a
huge take up of the on-line service delivery medium (Ernst and Young, 1999). As
with other more traditional service delivery channels however, companies doing
business on-line are finding that
consumers are every bit as vocal in their demands for a quality service
experience (Robledo, 2001). Consumers, who have gained expectations of quality
in offline environments are
unrepenting in their demands for its consistent delivery, regardless of medium.
The challenge for teams of software designers, marketers and experts in human-computer
interaction (among others) is to create
a business model that not only increases productivity and enhances bottom line
performance, but one that also seeks to add value for customers, employees and
suppliers alike (Lovelock, 2001, p.543). As Lovelock putsit “rushing to adopt
new Web-based strategies without thinking through the implications for
employees and customers, and the overall operating system can be a recipe for
pain”.
Above all, this requires the creation of what has
been termed ‘compelling on-line experiences’ for customers (Bezos, 1999). This
is seen as central to the customer’s perception of service quality, the
creation of a value proposition and the sustainability of long-term customer
relationships. Bezos made this point
when he highlighted the link between on-line service provision and its
contribution to on-line recommendation intention, which according to
Cognitiative (1999) is the most important driver of customer traffic to
commercial websites. When one considers the alternative, the expression of
customer dissatisfaction through negative web-site advertising, the costs of
not getting things right first time are great (Zellner, 1999, p.86). This view is shared by Novak, Hoffman and Yung (2000,
p.22) who state that “on-line executives note that creating a compelling
on-line experience for cyber customers is critical to creating competitive
advantage on the Internet”.
The Internet, reduces the time and effort required of customers to seek out new
suppliers and enter into new service relationships. In other words, new product
and service alternatives are only a few keystrokes away, and the consumer can
gain greater control over their relationship
with a seller. . This view is also shared by Hamel and Sampler (1998, p.82) who
in referring to the advertising challenge of distributing services through
electronic channels suggest that unlike traditional forms of advertising where
the consumer is a passive receiver of a message, the user of the web is
different. “They want the choice to click or not, to view or not, and anything
more than the gentlest form of persuasion from an advertiser is likely to be
construed as an intrusion”. Both authors go on to suggest that “the Web is not
about push; it’s about suck: On-line consumers can take out of cyberspace
whatever interests them and leave behind whatever doesn’t”. , The winners in the on-line search for
market space will be those who consistently provide compelling, user-friendly
and highly responsive on-line service experiences.
Yet, while research suggests clear and unequivocal
links between the provision of technology based experiences and competitive
differentiation (Van Hoof, Collins, Combrink and Verbeeten, 1995; Bezos, 1999),
specific research on the identification of attributes that work together to
create effective on-line service experiences is lacking. Indeed, Novak et al
(2000) suggest that while marketers have gained considerable understanding of
what brings people to their site, they still have little knowledge about what creates an effective
on-line experience.
One attempt to address this issue, has been that
proposed by Barnes and Vidgen (2001) and their WebQUAL assessment tool. This
was born out of the earlier work of Deans and McKinney (1997) who were amongst
the first researchers to examine the objectives, strategies and techniques used
by commercial websites in their pursuit of market share. WebQUAL, as the term
suggests was designed to assess Web site quality as well as audit the way in
which businesses use Internet technologies. Best described as a weighted
performance measure, this tool was developed as a result of a two-stage
research process. The first and more qualitative stage was designed to elicit
both consumer and Web designer perceptions of Web site quality. In turn, these
were to form the basis of the WebQUAL instrument, which was subsequently tested
and refined over various rounds of data collection including Internet bookstores
and Internet auction sites.
Pitt, Watson and Kavan (1995) in their study of
“Information System (IS) Effectiveness”, proposed the use of a modified
SERVQUAL. The researchers were concerned that most of the then existing
measures of IS effectiveness were flawed as they focused totally on the more
product related aspects of the IS Function, with little, if any reference to IS
service quality. This, they suggested,
had the potential to lead to a mismeasure of IS effectiveness. The team went on
to address the suitability of the SERVQUAL instrument as a possible measure of
IS service quality and after various tests for validity and reliability, concluded
that SERVQUAL was an appropriate instrument for researchers seeking to measure
IS service quality. This view was subsequently challenged however, in a debate
that mirrors that which has developed within the more generic marketing
literature surrounding the use of SERVQUAL as a reliable and valid measure of
the service quality construct (Van Dyke, Kappleman and Prybutok, 1997). The
authors point to a number of conceptual and empirical problems surrounding the
use of SERVQUAL in an IS context and suggest that caution should be exercised
in the interpretation of IS-SERVQUAL scores.
While these tools have broken new ground in the area
of assessing consumer perceptions of Web quality it has been suggested that
their use may be questionable. It has been suggested for example, that most of
the dominant service quality research may not even be applicable to e-business
environments. This is a view expressed by Cox and Dale (2001, p.121) who put
this down to two key differences between traditional and
e-service environments, namely, the lack of personal interaction and the web’s ability
to customise its service offerings. The authors go on to explore the issue of
whether the theories and concepts developed in the service quality literature
can be applied equally to the e-business medium and highlight the need for
further research on this very issue.
Building on previous research and the gaps in
knowledge that have been identified, the research propositions for this paper can
be stated as:
P1: Service quality in online environments can be measured
using the SERVQUAL methodology, achieving standards of reliability and validity
comparable to its application in offline environments.
P2: Service quality in online environments can be measured
using the SERVPERF methodology, achieving standards of reliability and validity
comparable to its application in offline environments.
Measuring
service quality
Today’s service manager faces much choice when it comes to measuring
customer perceptions of service quality, with a full range of measurement
techniques on offer. The difficulty is that many of these techniques are too
costly, too complicated or totally inappropriate for what is being measured.
The most critical challenge facing managers, therefore, is to identify and
implement the most appropriate methods for measuring the quality of the service
experience (Ford and Bach, 1997).
In the main, organisations employ a mix of qualitative and quantitative methods, choosing to collect feedback through a combination of both observation and/or communication techniques. Qualitative methods include interviews, focus groups, customer role-play and observation research, and whilst highly subjective, they provide an interesting insight into the mindset of individual customers. Quantitative techniques, on the other hand, claim to be more objective and measurable in nature. . Quantitative surveys can be administered either face to face (as in the case of exit surveys), indirectly (by telephone) or simply left for the customer to fill out in their own time (as in the case of room surveys and customer comment cards). Similarly, surveys can be employed on a regular basis, as in the case of comment cards or on a less regular basis, as in the case of ad hoc research to investigate specific issues associated with service delivery. Of importance here is the validity, reliability and practicability of the particular survey instrument.
The confirmation-disconfirmation paradigm has
been extensively incorporated into surveys of service quality. This seeks to
explore the relationship between a customer’s pre-purchase expectations and
their perceptions of service performance. As consumers evaluate the levels of
the service’s performance, they typically cannot help but compare that
performance to what they expected. In turn, these expectations provide a
baseline for the assessment of a customer’s level of satisfaction. These models
contend that service
quality can be conceptualised as the difference between what a consumer expects
to receive and his or her perceptions of actual delivery. They suggest that product and service performance exceeding
some form of standard leads to satisfaction while performance falling below
this standard results in dissatisfaction (Wilkie, 1990; Wells and Prensky,
1996; Oliver, 1997). According to Mowen (1995) this expectancy disconfirmation
approach helps explain consumer perceptions of service quality as well as
consumer satisfaction judgements.
Over the years researchers have adopted both
inferred and direct disconfirmation techniques, where the inferred approach
seeks to estimate the size of any gap between the customer’s expectations and
the actual performance received. Expectations and perceptions are measured
separately producing a relative measure of how well the service has performed
relative to what the consumer expected. Direct disconfirmation measures, on the
other hand, provide an absolute measure of performance. It is a measure of how
the service has performed on the basis of the customer’s absolute level of
satisfaction or dissatisfaction with the service encounter.
Pre-eminent amongst these studies has been the work of Parasuraman, Zeithaml and Berry (1985) and the development of their SERVQUAL instrument. Their research has concentrated on the belief that service quality is measurable but only in the eyes of the consumer. They take the view that service is deemed to be of high quality when customers’ expectations are confirmed by subsequent service delivery. They postulate that, as services are less tangible than goods, the dimensions on which customers form expectations may also be different. Initial qualitative research has led to the identification of five dimensions (tangibles, reliability, responsiveness, assurance and empathy – more commonly referred to as RATER) on which customers evaluate service quality. If expectancies are disconfirmed on any of these dimensions, satisfaction gaps result, and the customer is likely to record a poor rating of the quality of service provided. Over the years these researchers have developed their initial qualitative studies into the more comprehensive statistical tool known as SERVQUAL which is now widely used to measure service quality throughout the services sector.
SERVQUAL has been extensively researched to validate its psychometric
properties and has been applied in a wide variety of sectors (Lewis
1987; Lee & Hing 1995; Ryan & Cliff 1997; Lam, Wong and Yeung 1997). It takes the form
of a two part 22-item questionnaire (Appendix. 1), which seeks to estimate
customer’s pre-consumption expectations of service as well as post-consumption
perceptions of actual service received. Customers are asked to self-complete
each section of the survey on the basis of a seven point Likert scale which extends
from 1 (strongly disagree) to 7 (strongly agree). Measures of service quality
can be derived by subtracting the expectation scores from perception scores,
which can also be weighted to take account of the relative importance of each
quality dimension. In turn these importance scores allow managers to focus
attention where it is likely to have most impact or where it is most needed.
The scores across all the questionnaires are summed and averaged to find a
score for each question. The results of the questions within each dimension are
then averaged to obtain a score for each dimension which can then be used to
highlight how well an organisation is performing in light of customer
expectations.
The benefits derived from this approach are clear and may be summarised
as follows:
·
SERVQUAL gives management a clear indication of how the company is
performing in the customer’s eyes both individually and en mass.
·
It helps prioritise customer needs, wants and expectations by
identifying what is most important in the customer’s eyes. As stated this
information can be gleaned from the weighting of individual dimensions.
·
It allows the organisation to set an expected standard of performance
that can then be communicated to all staff and patrons.
·
It can also identify the existence of any gaps between customers and
providers and thereby helps focus improvement efforts by directing
organisational energies at closing these gaps.
While the SERVQUAL technique has attracted a lot of attention for its conceptualisation of quality measurement issues, it has also attracted considerable criticism. Some researchers have debated whether the dimensions of SERVQUAL are consistent across industries; others have suggested better wording for some of the scale items (Babakus and Boller, 1992). In addition, researchers have asked whether the calculated difference scores (the difference between expectations and evaluation) are appropriate from a measurement and theoretical perspective (Brown, Churchill and Peters, 1993). From a measurement perspective, there are three psychometric problems associated with the use of difference scores: reliability, discriminant validity and variance restriction problems. A study by Brown et al., (1993) found evidence that a number of psychometric problems arise with the use of SERVQUAL; they recommend, instead, use of non-difference score measures which display better discriminant and nomological validity. As mentioned above, other researchers have suggested better wording for some of the scale items (Bolton and Drew 1991). Customers find it hard to differentiate between many of the scale items, particularly when “negative forms of questions are used” (Hope and Muhlemann, 1997, p.288). For example, consider the following: ‘the company strives to get it right first time’ and ‘the company gets it right first time’.
There has also been debate surrounding the
practicalities of administering the instrument, principally about whether it is
practical to ask consumers about their expectations of a service immediately
before consumption and their perceptions of performance immediately after the
service encounter. Customers may become tiresome or distressed as a result of
being asked to complete both surveys. Some analyses have therefore used
combined single scales to measure gaps (Carman, 1990; Babakus and Boller 1992).
It has been suggested that expectations may not exist or be clear enough in
respondents' minds to act as a benchmark against which perceptions are assessed
(Andersson, 1993; Iacobucci., Grayson, and Omstrom,1994).
Furthermore, it is argued that expectations are only formed as a result of
previous service encounters, that is, perceptions feed directly into
expectations (Kahneman and Miller 1986; Teas, 1993). Consequently, customers
have a tendency to circle ‘strongly agree’ or ‘very important’ for all aspects.
For all of these reasons, many researchers now believe that a more direct approach to the measurement of service quality is needed. It is felt that performance-only based measures of service quality may be an improved means of measuring the service quality construct (Churchill and Surprenaut, 1982; Bolton and Drew, 1991; Cronin and Taylor, 1992; Van Dyke., Kappleman, and Prybutok, 1997). This has led to the development and application of a more direct form of measurement technique such as SERVPERF. This technique made use of the original SERVQUAL scale items and also requires the customer to rate a provider’s performance, on a Likert scale extending from strongly disagree to strongly agree. Unlike SERVQUAL, however, it does not seek to estimate difference scores and addressing post-consumption perceptions only. The instrument requires the consumer to rate only the performance of a particular service encounter. Further, it eliminates the need to measure expectations on the grounds that customer expectations change when the consumer experience a service. The inclusion of an expectations measure, therefore, reduces the content and discriminant validity of such measures (Cronin & Taylor, 1992; McAlexander, Kaldenberg, and Koenig, 1994). The model argues against the use of expectations because an accurate expectations measure can only be obtained prior to the service encounter. As such, SERVPERF is an absolute rating of customer attitudes towards service quality. A further study by Boulding, Kalra, Staelin, and Zeithaml, (1993), of which Zeithaml herself was a member concluded that service quality should only be measured via a perceptions-statements instrument.
Studies conducted using this performance-based measure found that
SERVPERF explained more of the variance in an overall measure of service
quality than did SERVQUAL. Cronin and Taylor (1994, p.127) acknowledge that it
is possible for researchers to infer consumers’ disconfirmation through
arithmetic means (the P – E gap) but that “consumer perceptions, not
calculations, govern behaviour”. This approach also overcomes some of the
problems raised regarding SERVQUAL, namely: raising expectations,
administration of the two parts of the questionnaire and the statistical
properties of difference scores (Hope and Muhlemann, 1997). Taking a single
measure of service performance is seen to circumvent all of these issues. It is
felt however, that from an operational point of view, ‘much useful information
is lost’ when performance only measures are taken.
It is to this debate that attention now turns by way of comparing the
psychometric and performance of both techniques for
evaluating the quality of the on-line experience.
Methodology
The study made use of both the inferred
and direct disconfirmation methodologies for measuring perceptions of service
quality of an on-line library service in a public sector university in Western Australia. The inferred
approach took the form of the original SERVQUAL (P-E) model consisting of two
sections: one section to measure user expectations of the on-line service
offered and a corresponding section to measure user perceptions of the actual
service received. Expectations were measured retrospectively and – in common
with much application of the SERVQUAL methodology – the limitations of
measuring expectations which have been conditioned by subsequent experience
must be noted. The more direct approach (SERVPERF) on the other hand, was
designed to evaluate user perceptions only, of the actual service received. The
scales developed took the form of an 18-item self-completion questionnaire,
which visitors were asked to complete immediately prior to departing the library.
For each item respondents were asked to rate their perceptions of the
attributes listed on a five point Likert scale anchored at (1) strongly
disagree and (5) strongly agree. In addition, respondents were also asked to
rate their level of expectation attributed to each quality dimension on a
similar scale anchored from (1) I had low expectations through to (5) I had
high expectations.
Scale items were based on the 22 items of
the original SERVQUAL instrument, but were modified to take account of the
particular service setting. This scale has been widely replicated in both
inferred disconfirmation and performance-only based measures (Churchill and
Surprenaut, 1982; Bolton and Drew, 1991; Cronin and Taylor, 1992) and the
factor structure found to be appropriate to a wide range of consumer services,
of which university services are typical (Lewis 1987; Babakus and Boller 1992 ;
Saleh and Ryan 1992; Ryan & Cliff 1997; Lam, Wong and Yeung 1997). However,
it was felt that not all scale items would be directly relevant to measuring
service quality in an online environment. In order to refine the original scales
so that they achieved a high degree of validity, a series of focus groups were
organised. Three focus groups were organised, consisting of 5 to 7 students. who had used the online library service. A moderator of the group initially sought
unprompted discussion of the factors that contributed to quality of online
services, initially in general, and then specifically in the context of library
services. Respondents were then presented with a draft list of revised SERVQUAL
scale items and asked to comment on their relevance. In respect of each item,
respondents were asked to develop alternative forms of the scale which they
considered to be more useful. In the case of four scale items, no meaningful alternative
formulation of the item was forthcoming. In the absence of any theoretical
justification for retaining these four items, they were deleted from the scale
items to be used in the survey instrument. An iterative approach was applied to
the second and third focus groups, in which discussion was additionally invited
on the refined scale items derived from the previous groups. All discussions
were recorded and analysis was subsequently agreed with an independent assessor
who had not been involved in the focus groups. A list of the 18 refined
scale items is shown in Table ##. . This ‘customisation’ is in keeping with
similar survey adaptations, for example, Allen and Davis (1991), Babakus and
Boller (1992) and Carman (1990).
The sample comprised students from a university in Western Australia. A total of 400 surveys were distributed to students on two campuses over a two-month period (February and March 2001). The sampling frame comprised all students visiting the two libraries during a one-week period at each library. Students were approached randomly by a surveyor as they left the library building and asked if they would complete the questionnaire. Willing volunteers were given a questionnaire for self-completion, to be returned to a collection box in the library. Of those issued a total of two hundred and sixty-nine useable surveys were returned, representing a return rate of just over 67 percent. No incentive was offered. This is a high response rate, which might be explained by students’ high level of involvement in online library services, evidenced by their propensity to make suggestions directly to the library and through class representatives about methods to improve the service. It was felt that the use of a student sample comprising individuals of a broadly similar age and educational background would reduce the confounding effects of differences in exposure to Internet media. All students in the sample frame had been trained in the use of online library services and had been required to use the services as part of assignments.
Research Setting
The research setting relates to the on-line library
service offered by a university in Western Australia. In the search for
competitive advantage, universities have recently been forced to reconsider the
nature of both their traditional product and service delivery methods. Not
surprisingly, technological developments have had a remarkable impact on how
educational services are now produced, delivered and consumed. One such
approach has been the development of on-line learning environments that are
directed more toward independent and distance learning. The advent of the cyber
classroom has revolutionised higher education delivery and presented a unique
opportunity to add value to the more traditional educational product, whilst at
the same time satisfying organisational demands for improved resource
productivity.
The library service offered through the web allows the user to access the library catalogues, journal articles through the use of what is referred as “in-Door Gateway”. It should be pointed out that the university library has reciprocal sharing agreements with the four other universities in WA. The site allows students and faculties the convenience of all library services without leaving their office: reserve materials, have books and articles delivered at a cost, as well as the ability to extend dates on already checked out books. Its many offerings allow students to access a magnitude of information off campus.
Results and Analysis
The results of the study are presented in three sections. Section one
provides a brief description on the demographic characteristics of the sample.
Section two reports on the psychometric performance of both instruments and
section three reports on the diagnostic performance and practical value of each
technique.
Looking firstly at the demographic characteristics
of the sample, respondents included 111 (41%) males and 144 (53.5%) females,
with 14 missing entries accounting for the remaining 5.2 percent. Approximately
64 percent of the sample were in the age range 17-23, with the remaining 35
percent of respondents being classed as non-traditional, mature aged students.
Approximately 55 percent of respondents classed themselves as Australian
nationals, with the remaining 45 percent of respondents classing themselves as
international students. In terms of year of study 24.9 percent of respondents
could be classed as year one students, 38.7 percent as year two, 19 percent as
year three and 17 percent of students could be classed as other encompassing
postgraduates. Just over 71 percent of students stated that they used the
Internet between one and ten hours per week.
Take in Table I
Looking firstly at the original SERVQUAL (P-E)
instrument, table 1 illustrates that the five-component structure proposed by
Parasuraman, Zeithaml and Berry (1988) for their SERVQUAL scale was not
confirmed, revealing a different factor
structure. Additional analysis reveals factor loadings along three dimensions with coefficient alpha
scores ranging from 0.57, 0.70 to 0.92,. with
two out of the three components exceeding the required coefficient alpha of
0.70 (Nunnally, 1978). The cross loading of two of the quality attributes
assessed (V5 and V6) across extracted components one and three may in part
explain the low reliability coefficient for extracted component three (Table
I). (## MARTIN – WERE THESE DELETED FROM FURTHER ANALYSIS?
Further analysis reveals that extracted component one is reflective of a combination of the responsiveness, assurance and empathy dimensions from the original SERVQUAL instrument. Viewed in the context of the on-line experience this seems to relate to the contact issue (CONTACT) and the much fabled ‘moment of truth’ (Norman, 1984). More specifically, this factor seems to revolve around library support staff and their willingness and ability to answer student queries relating to the on-line service. Component two is reflective of the aesthetic appeal of the actual web site (TANGIBLES) and the library environment within which it is located. Component three seems to relate to the mediums ability to consistently locate specific and relevant articles and books (RELIABILITY).
These results are somewhat reflective of
an earlier study by Getty and Thompson (1994) who uncovered a similar
three-factor structure. As with the present study, the authors found that
tangibles and reliability were readily identifiable as individual components.
Similarly, the authors also identified a third dimension that seemed to
represent a composite of SERVQUAL’s responsiveness, empathy and assurance
dimensions. This dimension they also labelled as contact as it was deemed to be
reflective of those elements of the service encounter that related specifically
to points of encounter between the delivery mechanism and customers.
Construct validity was addressed in terms of both convergence and the research instrument’s ability to discriminate between the underlying dimensionality of the service quality construct. Convergence was investigated by calculating the mean difference scores for each of the 18 scale items and correlating (Pearson’s product moment correlation) these with the mean score from an overall single item measure of quality which was also included in the instrument. A correlation of 0.646 was found which was significant at the 1% level. Discriminant validity was assessed by ## MARTIN
The SERVPERF measure of on-line service
quality was exposed to the same
rigorous factor analysis. As with the inferred measure, Table II again
illustrates that the five-component structure proposed for SERVQUAL did not
emerge. Additional analysis revealed
strong factor loadings along four dimensions with coefficient alpha scores
ranging from 0.68 to 0.86. Once again, this is indicative of a high degree of
reliability with three out of four components exceeding the required
coefficient alpha of 0.70.
Take in Table II
Further analysis revealed that extracted component
one is reflective of a combination of the reliability, assurance and empathy
dimensions from the original SERVQUAL instrument. As with the preceding
SERVQUAL analysis, this factor also seems to relate to the contact issue
(CONTACT) and is once again indicative of the actual encounter that the on-line
user has with both the search engine itself and supporting library staff.
Component two relates directly to the ease of flow of information and waiting
time (responsiveness) associated
with the on-line service. Component
three seems to relate to the mediums ability to consistently locate specific
and relevant articles and books (RELIABILITY) and component four is reflective
of the aesthetic appeal of the actual web site (TANGIBLES) and the library
environment within which it is located.
The SERVPERF instrument also performed well in terms of both reliability and validity. It recorded an overall reliability score of 0.90, once again exceeding the usual recommendation of alpha = 0.70. Construct validity was also addressed in terms of both convergence and the research instruments ability to discriminate between the underlying dimensionality of the service quality construct. As with the SERVQUAL scale, convergence was investigated by calculating the average perception score for each of the 18 scale items and correlating (Pearson’s product moment correlation) this with the mean score from the overall single item measure of quality included in the scale. A correlation of 0.683 was found, which was again significant at the 1% level.
Overall, both instruments performed well in terms of reliability and
validity, . Both demonstrated high internal consistency, with high overall
reliability scores on the majority of dimensions across all scales. They also
met the basic criterion for content validity and on the face of it were well
suited to the evaluation of the on-line service quality construct.
Correlational data also suggests strong convergent validity between both techniques and the single item measure
of service quality. Regardless of whether the appropriate factor structure for
perceived on-line quality should properly exhibit a three or four factor
structure, the conclusion, based upon
this data, is that the factor structure identified by both instruments does not
confirm the original SERVQUAL factor structure.
This provides further evidence of the complexity of the service quality
construct and the fact that it cannot be defined in any one way for all service
encounters, not least the on-line service experience.
From a practitioner’s
viewpoint, much of the debate surrounding the psychometric performance of such
tools is of little concern. In reality operators are more interested in the
practicalities of such tools, their ability to provide timely and relevant
customer feedback and to assist with quality improvement and decision-making.
While the debate continues as to the one best method of evaluating customer
perceived service quality, academics run the risk of forgetting about the
original purpose for which such techniques were designed, ie their value as
diagnostic quality improvement tools. The real power of disconfirmation modelling
and in particular the SERVQUAL instrument lies in its ability to objectively
identify fail points and to target quality improvement efforts throughout the
organisation.
While SERVPERF overcomes many of the problems raised
regarding SERVQUAL, namely: raising expectations, administration of the two
parts of the questionnaire and the statistical properties of difference scores,
it is felt that from an operational point of view, much useful information is
lost when performance only measures are taken (Hope and Muhlemann, 1987;
Gronroos, 2001). This view is also taken up by Pitt, Watson and Kavan (1997)
who state that while perceptions-only measurement of service quality might have
marginally better predictive and convergent validity, this comes at
considerable expense to managerial diagnostics. Through its ability to pinpoint
and quantify the extent of actual gaps between pre-consumption consumer
expectations and their post-consumption perceptions, the SERVQUAL technique is
able to serve up a more meaningful discourse of operationally relevant quality
performance data.
This can be clearly demonstrated using data from the
direct disconfirmation (SERVPERF) factor analysis (Table II). Mean expectation
and perception values were calculated for each of the extracted dimensions
(CONTACT, RESPONSE, TANGIBLES, and RELIABILITY). Table III presents
corresponding perception and expectation scores for each of the extracted
dimensions, as well as the P-E difference scores, which are representative of
the corresponding SERVQUAL gap scores.
Take in Table III
Using the inferred SERVQUAL method of evaluation, it can be seen that RELIABILITY is the dimension of the on-line service encounter furthest from meeting customer expectations with a gap score of –0.51. Using a paired samples t-test, related P-E mean scores were found to be significantly different at the level of 1% (t=-7.33; p<0.001). This was followed by the CONTACT dimension (-0.50; t=-6.42; p<0.001), RESPONSIVENESS (-0.41; t=-7.29; p<0.001) and TANGIBLES (-0.14; t=-4.86; p<0.001). From an operational standpoint this information highlights the need for quality improvement efforts in all areas, but most specifically in relation to the RELIABILITY and CONTACT aspects of the on-line service offering.
If a performance only (SERVPERF) measure were taken
however, the rank ordering of each of the dimensions listed, as well as the
corresponding quality improvement efforts for each would change. RESPONSIVENESS
(m=3.28) would top the list in terms of the need for quality improvement,
followed by RELIABILITY (m=3.38), TANGIBLES (m=3.52) and CONTACT (m=3.62). This
is at odds with the information presented by the gap analysis and as a
consequence the prioritisation of any organisational improvement effort may be
wrong and even misdirected. Whilst there can be no doubt that the service under
investigation requires targeted quality improvement in each of the areas
listed, it is essential from a user point of view that any such improvement
effort be prioritized in line with on-line consumer needs and wants. By
considering both user expectations and perceptions, the SERVQUAL technique
permits the on-line operator to do just that. Regardless of which method is
used however, the preceding analysis should make it clear that from a purely
operational perspective, both techniques are of value in assessing the quality
of the on-line service experience.
Discussion and Implications
This study has a number of implications for both
academics and practitioners in relation to the evaluation of on-line service quality . Heightened
competition in the e-commerce domain continues to force the need for a reliable
and user-friendly service quality measurement methodology. Whilst the more
direct disconfirmation technique may prove more useful in this regard from a
psychometric viewpoint, it is felt that the inferred SERVQUAL approach offers
the on-line operator more valuable
information as an input to sustained
quality improvement.
·
The study
demonstrates that both techniques can
identify how on-line operators are performing and help target quality improvement efforts which have the
greatest impact on users’ online experience. Regardless of the approach taken,
practitioners must be prepared to influence and manage both consumer
expectations and perceptions, relative to the on-line service experience.
Formed as a result of word of mouth, previous experience or some other external
company communication, expectations are crucial in helping on-line users
evaluate the quality of the service they have received. It is essential
therefore, that on-line operators undertake extensive research in order to identify those factors
deemed most important by consumers in their evaluations of the on-line service
experience.
·
Coupled
with this is the need to continually monitor and incorporate changing
user-expectations and perceptions into the on-line delivery system. Operators should be mindful that system
requirements might have to change over time due to a change in the relative
importance of particular attributes to customers. In other words, customers’ needs may remain the same, but the
relative priority of the needs may change in line with changing expectations. This
could, for instance, occur as a result of the level of experience a consumer
has with a service provider over time. This may inform decision making on a
range of fronts including product development, system design, market
segmentation, targeting and positioning attributes, communication and
promotions strategy, physical evidence and actual service delivery.
·
It seems logical, therefore, that practitioners
should develop and operationalise some mechanism by which they can track
consumer expectations and perceptions over time. Whilst the use of more
quantitative measures of service quality is strongly encouraged for such
purposes, the approach that could actually be undertaken might be either
quantitative or qualitative in nature and may comprise any of the dominant
measurement techniques employed within the services sector today. Ideally this
should involve conducting ongoing longitudinal research with new and existing
users through a portfolio of research approaches of which the more detailed
disconfirmation approaches have a significant role to play.
·
Whilst
beyond the remit of this paper, there are operational implications
that must be considered with respect to the use of such techniques. All
elements of the on-line delivery system should be designed around its ability
to meet and surpass customer expectations for each of the attributes assessed.
That is, the service facility, product, operations and customer service
functions must be customer focused.
This requires that all customer feedback be treated as the valuable asset it is,
heart felt feeling regarding the service experienced and acted upon
accordingly. This should allow the organisation to extract specifications for the design of the on-line
delivery system from the actual end user as opposed to the service provider.
Such specifications can then be used to design the more technical aspects of
the service.
Limitations
of this study relate to the isolated evaluation of one on-line service, at one
academic institution. There is need to
conduct further extensive empirical testing of such models in a range of
different service settings and with a range of different on-line customers in
order to continue to establish the reliability and validity of both techniques
as a general framework for the assessment of on-line quality . The results are further limited by the
choice of students as the subject body. It is quite possible that as a group
they might have responded differently to the service encounter than would a
broader cross-section of the population if exposed to a similar service
encounter.
relationship between service
quality and three outcome variables.
Journal of
Marketing
Education, Vol 12: 47-55.
effects of service quality tested
on a case within the restaurant industry, in
Kunst, P. and Lemmink, J.
(Eds.). Quality management in service.
The
Netherlands, van Goraum:
41-48.
Babakus, E., and Boller, G. (1992). An
Empirical Assessment of the SERVQUAL
Scale.
Journal of Business Research, 24
(May): 253-268.
Barnes, S .J., and Vidgen, R.T. (2001).
"An Integrative Approach to the Assessment
of
E-Commerce Quality," Centre for Information Management Working
Paper (ref. CIM2001/01), University of Bath.
Bergeron,
B.P. (2001). The Eternal E-customer. New
York: McGraw-Hill.
Bezos,
J. (1999). Setting the standard with Jeff Bezos. Internet Summit 99, July 20th,
Dana Point, CA.
Bitner, M.J., Brown, S.W. and Meuter, M.L. (2000). Technology Infusion
in Service
Encounters.
Journal of the Academy of Marketing
Science, 28 (1):138-149
Bolton, R., and Drew, J.H.
(1991). A Multistage Model of Customers' Assessments of
Service Quality and Value. Journal of Consumer Research, Vol 17 No 4: 375-
84.
process
model of service quality: from expectations to behavioural
intentions.
Journal of Marketing Research, Vol 30
No 1: 7-27.
Measurement of Service Quality. Journal of Retailing, 69(Spring): 127-
139
Carman, J.M. (1990):
Consumer perceptions of service quality: an assessment of the
SERVQUAL dimensions, Journal
of Retailing, Vol 66 (1): 33-55.
Churchill, G.A., and Suprenant, C. (1982). An
investigation into the determinants of
customer satisfaction. Journal of Marketing Research, Vol 19:
491-504.
Cognitiative. (1999). E-commerce and the
evolution of retail shopping behaviour.
Pulse of the Customer, Vol 1 No 2 (http://www.cognitiative.com/).
Cox, J and Dale, B.G. (2001). Service quality
and e-commerce: an exploratory
analysis.
Managing Service Quality, Vol11 No 2:
121-131.
Cronbach, L.J. (1951). Coefficient alpha and the interanl structure of
tests.
Psychometrica,
16
(October): 297-334.
Cronin, J.J., and Taylor, S. A. (1992)
Measuring Service Quality: A Re-examination
and
Extension. Journal of Marketing, 56 (July): 55-68.
Cronin, J.J. and Taylor, S.A. (1994).
SERVPERF versus SERVQUAL:
reconciling
performance-based and perceptions-minus-expectations
measurement
of service quality. Journal of Marketing,
Vol 58 (1): 125-131.
Deans, K.R. and McKinney, S. (1997). A
Presence on the Internet: the New Zealand
Perspective.
Australia and New Zealand Marketing
Academy Conference
(ANZMAC), December 1-3, Monash
University, Australia.
DeVellis, R. (1991). Scale development theory
and applications. Newbury Park:
SAGE
Publications.
Ernst & Young. (1999). 1999 E-Commerce Survey. Ernst & Young.
Ford, R.C and Bach, S.A.
(1997). Measuring Hotel Service Quality: Tools for Gaining
the Competitive Edge, Florida
International University Hospitality Review,
Vol 15 No 1 (Spring):
83-95
Getty, J.M., and Thompson, K.N. (1993).
Assessing customers’ perceptions of
lodging
quality: LODGQUAL – A measure of quality for the lodging industry.
Unpublished working paper, University of North Texas.
Getty, J.M and Thompson, K.N. (1994). The
relationship between quality,
satisfaction
and recommending behaviour in lodging decisions, Journal of
Hospitality and Leisure Marketing,
Vol
2 No 3: 3-22.
Gronroos, C. (2001). Service Management and Marketing. New York: John Wiley
and
Sons: 172
Hamel, G. and Sampler, J. (1998). The
e-Corporation. Fortune, December 7th:
82-89
Hope, C. & Muhlemann, A.
(1997). Service Operations Management, London:
Prentice Hall
Iacobucci, D. Grayson, K.A.
& Omstrom, O.L. (1994). The calculus of service
quality and customer satisfaction: theoretical and
empirical
differentiation and integration. In T.A. Swartz, D.E.
Bowen & S.W.
Brown (Eds.),
Advances in Services marketing and Management, 3rd Ed, 1-
68, Greenwich CT:
JAI Press
Lam, T., Wong, A., and
Yeung, S. (1997). Measuring Service Quality in Clubs: an
Application of the SERVQUAL Instrument. Australian Journal of Hospitality
Management. 4
(1): 7-14.
Lee, Y.L. & Hing, N. (1995). Measuring
quality in restaurant operations: an
application
of the SERVQUAL instrument, International
Journal of
Hospitality Management, Vol 14, No 3-4:
293-310
Lewis, R.C. (1987). The
Measurement of Gaps in the Quality of Hotel Services.
International Journal
of Hospitality Management, 6 (2): 83-88.
Lovelock, C.
(2001). Services Marketing:
People, Technology and Strategy. 4rd Ed
Sydney:
Prentice Hall.
measurement, Journal
of Health Care Marketing, Vol 14 No 3 (Fall): 34-
39.
Mowen, J.C. (1995). Consumer behaviour. 4th Edition, New Jersey, USA: Prentice
Hall.
Normann, R. (1984). Service management, John Wiley and Sons, New York.
Novak, T.P., Hoffman, D.L. and Yung, Y.F.
(2000). Measuring the Customer
Experience
in Online Environments: A Structural Modeling Approach.
Marketing Science, Vol19 No 1 (Winter):
22-42.
Nunnally, J.C. (1978). Psychometric theory, New York, NY: McGraw-Hill Book
Company.
Oliver, RL (1997). Satisfaction:
a behavioural perspective on the consumer, London,
McGraw-Hill.
Parasuraman, A., Zeithaml, V.A., and Berry,
L.L. (1985) A Conceptual Model of
Service
Quality and its Implications for Future Research, Journal of
Marketing, 49 (Fall):
41-50.
Parasuraman, A., Zeithaml, V.A., and Berry,
L.L. (1988). SERVQUAL: A Multiple-
item
scale for measuring consumer perceptions of service quality. Journal of
Retailing, 64 (Spring):
12-40.
Parasuraman A., Zeithaml V.,
and Berry, L. (1994). Reassessment of expectations as
a comparison standard in measuring service quality:
implications for future
research. Journal of Marketing, Vol 59 (January): 111-124.
Services
using a new “P-C-P” attributes model. International
Journal of
Quality and Reliability Management, Vol 18 No 9:
900-916.
Pitt, L.F., Watson, R.T, and Kavan, C.B.
(1995). Service Quality: A Measure of
Information
Systems Effectiveness. MIS Quarterly, Vol 19 No 2.
Pitt, L.F., Watson, R.T. and Kavan, C.B.
(1997). Measuring Information Systems
Service
Quality: Concerns for a Complete Canvas. MIS
Quarterly, Vol21
No
2.
expectations.
Managing Service Quality, Vol 11 No
1: 22-31.
Ryan C. and Cliff A. (1997).
Do travel agencies measure up to customer
expectations?
An empirical investigation of travel agencies’ service quality as
measured by SERVQUAL, Journal
of Travel and Tourism Marketing, Vol
6
No 2: 1-28.
Saleh, F., and Ryan, C. (1991). Analysing service quality in the
hospitality industry
using the SERVQUAL model, The Service Industries Journal, Vol 11
(July):
324-343.
Tabachnik, B.G., and Fidell, L.S. (1989). Using multivariate statistics, (2nd Ed.). New
York: Harper and Row.
service
quality, Journal of Professional Services
Marketing, 8 (2): 33-53.
Van Dyke, T. P., Kappleman, L.A, and
Prybutok, V.R. (1997). Measuring
Information
Systems Service Quality: Concerns on the Use of the
SERVQUAL
Questionnaire. MIS Quarterly, Vol 21 No 2.
Van Hoof, H., Collins, R., Combrink, E. and
Verbeeten, M. (1995). Technology
Needs
and Perceptions: An Assessment of the US Lodging Industry. Cornell
Hotel and Restaurant Administration
Quarterly, Vol36 No 5: 64-69.
Wells, W., and Prensky, D.
(1996). Consumer Behaviour, John
Wiley
Wilkie, W.L. (1990). Consumer
Behaviour, 2nd edn, John Wiley and Sons.
Zellner, W. (1999). A Site for Soreheads. Business Week, April 12: 86.