Development and Validation of Four Social Scales for the UX Evaluation of Interactive Products

1. Introduction

User experience (UX) is a multidimensional construct that encompasses all aspects of user interaction with products, services, and systemsFootnote¹ (Robert & Lesage, Citation2011). It includes pragmatic dimensions, that are associated with products usability and functionality, as well as hedonic dimensions that are associated with users’ psychological well-being, like pleasure and stimulation, that together give a comprehensive picture of user-product interaction (Hassenzahl, Citation2003). As technology advances, new environments and ways of interactions such as virtual reality are introduced, resulting in new UX dimensions that should be considered in UX design and evaluation (Mortazavi et al., Citation2021). Despite the fact that people use more and more products that connect them to others, the social aspects of UX have not been adequately investigated in UX studies (Hinderks et al., Citation2019; Mortazavi et al., Citation2021). Yet, they have major impacts at a personal level for communication and entertainment purposes, as well as at a business level to improve technology adoption and productivity (Li, Citation2015). An evident example of such products is social media platforms that are being used by 4.7 billion people (Petrosyan, Citation2023). Social media serves purposes beyond basic individual communication, as businesses design marketing strategies that leverage the potential of customers on these platforms to promote and share information about the launch of new products (Zarnadze, Citation2020). Moreover, social features can contribute to improving people’s wellbeing and preventing isolation. The case of wearable social interaction aid for children with autism is a prominent example of assistive technologies that improve social interaction (Washington et al., Citation2016). By using wearable glasses, children with autism are provided with real-time social cues during their interactions with others, which would have otherwise been challenging for them to discern. This assistance aids them in sustaining engagement within social interactions with others (Washington et al., Citation2016). Similarly, a social glasses device and a tactile wristband were used to replicate eye contact during a blind-sighted conversation which can help blind people by giving them more visual information and enhance their communication quality (Qiu et al., Citation2020). Even products that were not primarily built for social activities are now incorporating them into their core features. For instance, thanks to the influence of social media, social sharing and social competing features are added to fitness trackers to encourage users to exercise more and share their achievements with their community (Zhu et al., Citation2017). Communicating with others, expressing oneself through the possession or use of a product, collaborating with others on a task, and creating an experience with others (i.e., co-experience) are all manifestations of social aspects.

The impact of social aspects on UX has been studied from two perspectives. The first characterizes it as an element of the context of use. Arhippainen and Tahti (Arhippainen & Tähti, Citation2003) identified social and cultural factors as part of the context in user-product interaction that influence UX. They mentioned time pressure, pressure to succeed or fail, and explicit and implicit requirements as examples of social factors (Arhippainen & Tähti, Citation2003). Lallemand and Koenig (Lallemand & Koenig, Citation2020) took a step further and developed the UX context scale (UXCS), a tool that can be used in conjunction with UX evaluation tools to portray the contexts within which users interact with a product. Other people’s presence, interactions with them, and how users feel in that environment are items measured in UXCS under social context (Lallemand & Koenig, Citation2020). However, it does not evaluate the social aspects of the UX with a product.

The second perspective considers social aspect as a UX dimension that should be measured on its own. In order to ensure a common understanding, the terms used in this paper are classified as follows: at the highest level, a “dimension” represents a significant factor explaining the UX of a product (Robert, Citation2014), “sub-dimensions” are the constituent elements of each dimension and “items” are the constituent elements of a sub-dimension. Identification is an example of a UX sub-dimension of the social dimension present in the AttrakDiff questionnaire (Hassenzahl et al., Citation2003). Identification is the degree to which a user relates to a product and expresses oneself through its possession (Hassenzahl, Citation2003). It is described as either self-focused or relationship-focused. The former is concerned with the users’ self-perception as a result of possessing or using a product, whereas the latter reflects their social identity (Yoon et al., Citation2020). Identification has roots in the influence and popularity needs of individuals, i.e., the need to be liked, respected, and regarded as influential to others (Hassenzahl et al., Citation2010), which is in line with Jordan’s notion of socio-pleasure (Jordan, Citation2000). In this study, we followed the second perspective, which considers social aspects as an integral part of UX evaluation tools, treating them as a UX dimension in need of more investigation. Knowing how socially attractive a product is, and how its social features influence the overall UX are of growing importance and deserve more attention from the research community. As presented in the following section, despite the wealth of available UX evaluation tools, there remains a gap in having a single tool that comprehensively measures different aspects of the social dimension without being bounded to a specific field of use. This study aims to bridge this gap by developing and validating social scales that can be used in conjunction with other UX evaluation tools.

2. Literature review

Reviewing the UX evaluation tools shows that social dimension is evaluated under different terms. Table 1 shows subjective UX evaluation tools and the social sub-dimensions they cover. General evaluation tools such as AttrakDiff are applicable to a variety of products, while others, such as the game experience questionnaire (GEQ), are developed for a particular field of use. However, neither covers all sub-dimensions of the social dimension.

In addition to AttrakDiff, meCUE (Minge et al., Citation2017) is another generally applicable UX evaluation tool that includes a social sub-dimension. It is analogous to AttrakDiff in that it focuses on self-expression and how a product can communicate identity to others under the UX sub-dimension of “status.” A more comprehensive view on social dimension of UX is presented by Park et al (Park et al., Citation2013). Although they did not develop a UX evaluation tool they defined sociability with three sub-dimensions—social emotion, social value, and friendship—for the UX of mobile phones and services. Sociability was defined as the “degree to which a product satisfies the user’s desire of being sociable.” The sub-dimensions respectively evaluate product’s ability to enable feeling and sharing emotions socially; its ability to support user values such as social issues and problems; and its ability to enable making relationship with other people. In another study, Ryu and Kim (Ryu & Kim, Citation2019) evaluated sociability with only one item in their questionnaire (without covering the three sub-dimensions) to find whether or not a medical information system satisfied the user social needs. One can notice the limited attention to different social aspects in general UX evaluation tools. AttrakDiff and meCUE only evaluate identification, and the User Experience Questionnaire (UEQ) (Laugwitz et al., Citation2008)—the most commonly used subjective evaluation tool—does not cover any social aspect (Mortazavi et al., Citation2021). The same goes for UEQ+ (Schrepp & Thomaschewski, Citation2019), a modular framework developed based on UEQ, including 20 UX dimensions that can be used for the evaluation of different products, yet none relates to social aspect.

Considering the impact of culture on UX, the Chinese UX questionnaire was developed based on AttrakDiff (Liu et al., Citation2013). Its authors proposed “conformity” as the new UX dimension that addressed the eastern culture of giving importance to other’s opinions. Conformity deals with the prevalence of a product in the market and how widely it is used by people. It is to some extent close to social acceptance, another sub-dimension that has been rarely used in UX evaluation (Weiss & Willkomm, Citation2013). Social acceptance is “how a user feels when interacting with a system in relation to the social situation, e.g., how uncomfortable or embarrassed they feel with respect to other people or one’s own norm” (Weiss & Willkomm, Citation2013). The negative impact of lack of social acceptability affects not just the UX of a product, but also the user’s self and social image, with the risk of stigmatization and misjudgment by others (Koelle et al., Citation2020). Making an intimate relationship with others when watching smart TVs (Jang & Yi, Citation2019), sharing information via a mobile app (Sun & May, Citation2013), collaborating with other students to co-write a poem (Kantosalo & Riihiaho, Citation2019), and using an interactive public display system for improving social interaction (Kang et al., Citation2022) are all examples of where different social sub-dimensions of UX need to be addressed.

Social interaction in the form of communicating with others, collaborating with teammates, or expressing oneself through game characters has been widely used in the game industry. Multiplayer online games, gaming conventions, and game streaming services are all examples of social interaction media for players and spectators. As a result, UX evaluation tools developed specifically for games include social dimension, such as GEQ (IJsselsteijn et al., Citation2013) and Game User Experience Satisfaction Scale (GUESS) (Phan et al., Citation2016). GEQ features a separate questionnaire for the social presence that covers players’ emphatic responses, negative feelings towards others, and behavioral involvement in games (IJsselsteijn et al., Citation2013). Another field where social dimension has gained attention is education. Interactions within and between learners and instructors, subjective norm, and self-image are social sub-dimensions evaluated in the FASER LX tool that evaluates learners experience in an e-learning system (Safsouf et al., Citation2019). Other researchers used serious games for learning purposes and developed tools like EGameFlow (Fu et al., Citation2009) and Model for the Evaluation of Educational Games (MEEGA+) (Petri et al., Citation2016) with social sub-dimensions. Reviewing the items of these tools highlights the case-specific nature of their development. Moreover, there is no suggestion of the possibility of a modular use of these tools except for GEQ social presence module. As a result, it is unclear whether these tools can be applied to other contexts.

Analyzing the UX evaluation tools showed that the social dimension of UX concerns three categories: product, user, and group (society). The product-related items investigate whether a product can provide social interactions. For example in EGameFlow, the item “the game supports social interaction between players” highlights the product’s capability of making interaction. The user-related items evaluate what users can do with the social features of a product. For instance, Smart TV UX questionnaire (Jang & Yi, Citation2019) uses the item “I can form an intimate relationship with others by using the smart TV” and GEQ uses “I empathized with the other(s)” as actions that can be done by users with social features of products. The group-related items investigate how users/owners of a product are perceived by other people and social norms. The item “I was not worried about other people’s judgement” of the flow sub-dimension of UX in IVE questionnaire (Tcha-Tokey et al., Citation2016) and the item “Rarely used – widely used” of the Chinese UX questionnaire are examples of this category. Regardless of the naming in different tools, the social sub-dimensions provided in Table 1 includes at least one of the three categories of items. However, none of the UX tools cover all the social sub-dimensions.

Clearly, the UX community needs a new tool to evaluate the social dimension of UX when interacting with a product that is not bounded to a specific application field (e.g., games or education) and ensure a comprehensive coverage. A promising way of achieving this goal is to develop social scales and include them into the UEQ + framework. Modularity of the UEQ + enables it to be used in different contexts, allowing the evaluator the flexibility to tailor the tool and evaluate the set of UX dimensions most relevant to the study (Schrepp & Thomaschewski, Citation2019). Researchers can construct new UX dimensions and add them to the UEQ + framework. For instance, response behavior, response quality, and comprehensibility are the new UX dimensions developed for voice assistants (Klein et al., Citation2020). Similarly, haptic and acoustic dimensions were developed for the UX evaluation of household devices (Boos & Brau, Citation2017). In UEQ+, each dimension consists of four items measuring the dimension and a single item that determines the importance of that dimension for the UX evaluation of the product. The evaluation of an item is done on a 7-point Likert scale with two semantic differential anchor points. The consistency of UEQ + simplifies the UX evaluation compared to using a combination of UX evaluation tools each with different rating scales and possible overlapping dimensions (Schrepp & Thomaschewski, Citation2019). It also provides an overall UX rating for products that can be used for comparisons. All these advantages make UEQ + a suitable candidate for hosting UX social dimension that can be used with other UX dimensions for different products.

In this study, we designed, developed, and validated social scales that can be used with other UX dimensions of the UEQ + framework. To this end, we followed the dimension development process used in Schrepp and Thomaschewski (Citation2019) for other UX dimensions of the UEQ + framework, focusing on social dimension and present the validated questionnaire.

3. Study framework

This study followed three phases: identification, integration, and validation (Figure 1). The identification phase consisted in extracting social sub-dimensions and corresponding items from the literature and from the analysis of the UX evaluation tools, and in selecting a sample of items for each social sub-dimension by UX experts. These items were used in the integration phase to develop social scales by performing Exploratory Factor Analysis (EFA) on the responses of 229 participants who rated their UX with interactive products having social characteristics. The validation phase included another survey study with 450 participants who rated their UX with specific products from three different product categories with various levels of social dimension. The results were used to calculate the reliability and validity of the social scales. This study was approved by Polytechnique Montreal’s Research Ethics Board (CER-2122-47-D) and participants read and signed an informed consent form before taking part in the study.

Figure 1. Three phases of the study.

Display full size

This paper is organized into eight sections. Section 4 presents the identification phase, followed by Section 5 including the procedure, methodology, and results of the integration phase. Then, Section 6 presents the validation phase and its main results with the validated social scales. Section 7 puts our results into perspective by comparing them with those of other social scales and presents limitations of our work. Section 8 concludes the paper with the main take-aways and some propositions for future research work.

4. Phase 1 identification

In a previous study, we performed a systematic literature review and an analysis of the UX evaluation tools and their dimensions (Mortazavi et al., Citation2021). As a result, we found that social dimension can be translated into three main sub-dimensions: identification, sociability, and social acceptance. Identification is the degree to which a user relates to a product and expresses oneself through its use or possession (Hassenzahl, Citation2003). Sociability is the degree to which a product enables communication with others to meet user’s social need (Park et al., Citation2013). Social acceptance is “how a user feels when interacting with a system in relation to the social situation, e.g., how uncomfortable or embarrassed they feel with respect to other people or one’s own norm” (Weiss & Willkomm, Citation2013).

An online workshop with seven UX experts took place in December 2021 on the collaborative whiteboard platform Miro. They reviewed the three social sub-dimensions and 24 items, discussed the possibility of merging them, and proposed 12 new items in a 90-minute session. Results of the workshop were reassessed by four of these experts and the number of items was reduced from 36 to 27. Similar-meaning items were removed, and some modifications were made to the wording of the semantic differential poles. Overall, the 27 items were grouped under three social sub-dimensions: identification, sociability, and social acceptance.

5. Phase 2 integration

6. Phase 3 validation

The goal of this phase was three-fold. First, to investigate the relationship between the social sub-dimensions using confirmatory factor analysis (CFA) and to test the fitness of this new data to the structural model we proposed. Second, to validate the scales by comparing the scores with those of other common evaluation tools. Third, to evaluate the social aspects of three product categories with the new social scales.

6.3. Results

6.3.1. CFA

The results of the three CFAs of the first model showed a high correlation (0.92) between social stimulation and social interaction for the social network product category (Figure 2-left). According to EFA results, these two dimensions could be represented by a higher-level factor of sociability. Thus, using sociability as a second-order factor, we kept both social interaction and social stimulation in the second model (Figure 2-right). The three latent variables had correlation coefficients above 0.3 but below 0.9, indicating that they were reasonably related, but not multicollinear. Although below the threshold, the correlation coefficient of 0.77 was observed between sociability and social acceptance.

Figure 2. CFA for the first model with four factors (left) and the second model with three factors (right) for social network category.

Display full size

Table 9 shows the goodness of fit measures calculated for the three CFAs on the second model after the modifications. The thresholds shown in the table are from (Hu & Bentler, Citation1999; Wang & Wang, Citation2020). Results confirm the model fit on the data for the three product categories.

Table 9. Goodness of fit measures for the 3 CFAs of the second model.

Download CSVDisplay Table

Table 10 shows factor loadings greater than 0.5 for all items of each factor, which is better than the results of the EFAs (Table 6). The average variance extract (AVE) of all factors exceeded the cutoff value of 0.5, confirming the convergent validity of the second model (Hair et al., Citation2014). The discriminant validity of the three factors of identification, social acceptance, and sociability was confirmed because maximum shared variance (MSV) was greater than AVE for each factor, and the squared root of AVE (bold in Table 11) was greater than the correlation of the factors (Gefen & Straub, Citation2005).

Table 10. Factor loadings of the CFAs for the 3 product categories.

Download CSVDisplay Table

Table 11. Model validity measures of the CFAs for the 3 product categories (AVE squared root in bold).

Download CSVDisplay Table

Table 12 shows the final four social scales with the corresponding items used for the UX evaluation of the three product categories.

Table 12. Developed social scales and their items.

Download CSVDisplay Table

6.3.2. Comparisons

Table 13 shows the correlations between the four social scales developed in this study and the four scales of the AttrakDiff questionnaire for the three product categories. The highest correlations were observed with identification and attractiveness scales, particularly for the social network product category. All the statistically significant correlations are marked with a star in the table.

Table 13. Correlations between the social scales and those of AttrakDiff (* significant with p < 0.05).

Download CSVDisplay Table

Table 14 shows the correlations between the four social scales developed in this study and the six scales of UEQ+. The four social scales were not redundant given that all the correlation coefficients were below 0.8. The highest correlations were observed with the Value and Attractiveness scales of the UEQ+. However, the social network product category displayed strong correlations with the pragmatic dimensions of Usefulness, Efficiency and Intuitive usage.

Table 14. Correlations between the social scales and the UEQ + scales.

Download CSVDisplay Table

Table 15 shows the average importance rating of each social scale. The highest ratings for all social scales went to social network and the lowest to online banking product category, as expected.

Table 15. Average importance rating of social scales for the three product categories (from 1: completely irrelevant to 7: very important).

Download CSVDisplay Table

6.3.3. MANOVA

Before conducting MANOVA to test the social scales ratings differences between product categories, participants who responded to more than one product category were removed yielding three independent samples of 106, 98, and 96 participants for social network, online shopping, and online banking product categories, respectively. Following Mahanalobis distance calculations, multivariate outliers were eliminated such that further analyses were performed on three independent samples of 95 participants.

Results of Shapiro–Wilk test of normality was significant (p < 0.001), showing that the data was not normality distributed. However, the sample size was greater than 30 for each independent variable (i.e., product category), which is robust to the violation of normality or equality of variance (Pallant, Citation2020). The general pattern of scatterplot showed a linear relationship for each pair of dependent variables (i.e., four social scales). Table 16 shows that there was no multicollinearity between dependent variables because none of them had a correlation coefficient larger than 0.9 with another one. Moreover, being greater than 0.2, they were all at least moderately correlated. Box’s test of equality of covariance matrices was significant (p < 0.001) meaning that the observed covariance matrices of the dependent variables were not equal for all categories. Assumptions testing showed no violations for conducting MANOVA except for normality of data and homogeneity of variance-covariance matrices. As our sample size exceeds 30, violation of normality is not a concern. Moreover, the results of Pillai’s trace were used for multivariate test because it is more robust when there is violation of assumptions (Tabachnick & Fidell, Citation2019).

Table 16. Correlation between the dependent variables.

Download CSVDisplay Table

The results of one-way MANOVA showed significant difference between the three product categories on the combined social scales, F (8, 560) = 20.68, p < 0.001, Pillai’s trace = 0.456, partial eta squared = 0.228. More investigation on each dependent variable showed statistical significance for social interaction and social stimulation using a Bonferroni adjusted alpha level of 0.0125 (highlighted in Table 17). Bonferroni adjustment is used to reduce the risk of a Type 1 error when several separate analyses are performed. Table 17 shows the multiple comparisons between product categories for each social scale.

Table 17. Significance of mean differences of dependent variables for three product categories.

Download CSVDisplay Table

7. Discussion

This study covered the design, development, and validation of social scales for the UX evaluation of interactive products. UEQ + scales used EFA to identify the four most representative items. However, scale development in this study was different in that we also performed CFA on three different samples to provide more validity measures for the social scales. Results showed good internal consistency (Table 7), convergent validity, and discriminant validity (Tables 10 and 11) for the social scales. Although we only have a good model fit for the second model, the CFA confirmed our structural model of social dimension with the three main sub-dimensions of identification, sociability, and social acceptance.

Identification, social stimulation, social interaction, and social acceptance are the four social scales that were developed in this study (Table 12). The identification scale is similar to that of AttrakDiff and status dimension of meCUE. It deals with users’ personal and social image. Users meet their influence and popularity need by expressing their identity, influence, and power to others through the use or possession of a product. The social interaction and social stimulation scales are complementary. The social stimulation scale represents the social incentives provided by a product to enable users to be socially active. A socially engaging, encouraging, empowering, and inclusive product enables users to achieve their social interaction goals in the form of communication, collaboration, and sharing of emotions with others. Finally, the social acceptance scale entails the feeling of being accepted by others. It can evoke self-conscious emotions like pride that are fulfilled when one is recognized and approved by others.

The social scales were subject to validation studies to find their relationship with other UX dimensions. Identification, the only social dimension of the AttrakDiff questionnaire, was expected to have strong correlations with the four social scales developed in this study. Our results confirmed this expectation by finding correlation coefficients between 0.45 and 0.69 with identification, except for social interaction (r = 0.34) and social stimulation (r = 0.24) for the online banking product category (Table 13). In addition to identification, we observed high correlations between the social scales and the attractiveness dimension of AttrakDiff. Hassenzahl (Hassenzahl, Citation2004) found a strong correlation between identification and beauty. He stated that beauty is social because it communicates identity and can be shared with and approved by others. Considering that beauty is an item in the attractiveness dimension of AttrakDiff, we can justify finding high correlations of social scales with attractiveness.

In addition to AttrakDiff, we measured the correlations of the four social scales with six UX dimensions of UEQ+. In the AttrakDiff, UEQ, and UEQ + questionnaires, the attractiveness dimension is used for the overall assessment of a product. Boos and Brau (Boos & Brau, Citation2017) calculated the correlations of the acoustic and haptic dimensions with the attractiveness dimension of UEQ + to demonstrate that their new UX dimensions measured the same construct (i.e., UX). They concluded that the strong correlations with attractiveness validated the two new UX dimensions for the UEQ + modular framework. Following the same approach, we found correlation coefficients ranging from 0.42 to 0.67 between the four social scales and the attractiveness dimension of UEQ+ (Table 14). However, there were three exceptions: social interaction (r = 0.34) and social stimulation (r = 0.24) for online banking, and social stimulation (r = 0.30) for online shopping product category. We observed higher correlations between the social scales and the value dimension of UEQ+. This suggests that social scales are closer in nature to the hedonic dimensions of UX, such as value, than the pragmatic dimensions of efficiency or intuitive usage. However, high correlations with pragmatic dimensions were found in the social network product categories. Other studies have presented the influence of social features on both hedonic and pragmatic dimensions. For instance, (Guo & Li, Citation2021) found that interactivity, recommendations, and feedback as social features of social commerce platforms have an impact on both utilitarian and hedonic values that ultimately influence consumers’ repurchasing intention. The influence of perceived co-experience on enjoyment in social live streaming services (Bründl et al., Citation2017) and the impact of social image on the use of hedonic systems like games (Lin & Bhattacherjee, Citation2010) are additional examples of social aspects that affect user experience.

Validation of the social scales was tested by evaluating the UX of three product categories. These categories were selected to represent different levels of social aspects. The average importance rating of each scale in the UX evaluation of each product category followed our expectations, with social network having the highest importance ratings and online banking having the lowest across the four social scales (Table 15). Validation with three product categories addressed Lallemand and Koenig’s (Lallemand & Koenig, Citation2017) concerns on standardized scales development performing only a single validation study.

The MANOVA results showed a significant difference between the three product categories for the combination of social scales. Further analysis of each pair of categories revealed significant differences for social interaction and social stimulation (Table 17). The greatest difference was found in the social interaction scale between the social network and online banking product categories. It was to be expected, given that the former is designed as a communication platform, whereas the latter is exclusively task-related and rarely incorporates social interaction aspects. The mean differences for the two scales of identification and social acceptance were not significant for any product category when using a Bonferroni adjusted alpha level of 0.0125. The fact that all of the evaluated products were software applications could have had an impact on the results. It can be expected that the identification and social acceptance dimensions of software products do differ from those of physical interactive products that attract more attention from users and spectators in a social situation, such as virtual reality glasses, or even non-interactive products such as jewelry or designer handbags. However, at the alpha level of 0.05, significant differences could be observed between social network and online banking, which are at the opposite extremes regarding the social features.

Overall, the four social scales developed in this study can be used in conjunction with other dimensions of the UEQ + modular framework.

The first limitation of this study is that the selected products were all interactive products. Evaluating other products that can develop social UX such as designer clothes, watches, physical board games, and museum pieces could be considered in future studies. The second limitation of this study is that we relied on the users’ reported experience with the selected products. Using laboratory studies and doing specific tasks with better control over the hardware and context could provide more reliable results. However, this would have been possible only at a significantly increased cost with a likely much smaller sample.

8. Conclusion

In this study, we developed and validated four social scales for the UX evaluation of interactive products: identification, social stimulation, social interaction, and social acceptance. These scales are each measured with four items, following the same format as the other scales of the UEQ + modular framework. They can be used in conjunction with the other dimensions of the framework for the UX evaluation of product with social aspects. Therefore, they have the benefits of UEQ+, such as being freely available, rapid to use, and providing a global score for easier comparisons. We used the results of an online workshop with UX experts to prepare the candidate items for each social scale. Then we did two rounds of data collection, the first used for scale development and the second for calculating reliability and the validity of scales. The scales were then validated through correlation analysis with the dimensions of the AttrakDiff questionnaire and the six UX dimensions of the UEQ + framework. Results showed a high correlation with the identification and attractiveness dimensions of these two UX evaluation tools. Finally, using the social scales developed in this study we successfully discriminated among three product categories with different levels of social features. This study highlighted the importance of paying more attention to social dimensions. We recommend that future studies evaluate physical interactive products with social dimensions to better validate the social scales.