Using Twitter for Demographic and Social Science Research: Tools for Data Collection and Processing

Abstract

Introduction

Social media, such as Twitter and Facebook, provide exciting opportunities that can “open up a new era” of social science research (Golder and Macy 2012). These new communication platforms afford the ability to examine social data on a variety of topics, on a massive scale, and over short periods of time. As such, they have sparked the interest of many researchers seeking to better understanding social relationships and behavior (Brickman Bhutta 2012; Golder and Macy 2011; Heaivilin et al. 2011; Lowe et al. 2012; Moreno et al. 2012; Valkenburg, Peter, and Schouten 2006). Although some researchers have begun to use data from social media spaces such as Twitter to document changing moods and other sentiments on the aggregate level (Diakopoulos and Shamma 2010; Golder and Macy 2011; Naaman, Becker, and Gravano 2011; Reips and Garaizar 2011; Yardi and Boyd 2010), the potential of such data for demographic research has yet to be fully realized.

The ability to aggregate vast amounts of digital traces of human behavior through social media platforms represents a new data collection paradigm for social science research (Cambria, Wang, and White 2014; Lazer et al. 2009). In some ways, data produced online parallel more well-researched methods of data collection such as surveys or structured observations, but they also contain unique properties and new features. Surveys, for example, ask respondents to recall behaviors or sentiments retrospectively with limited scope, whereas social media afford the opportunity to observe personal expression and human interaction in real time and on a large scale. With appropriate infrastructure, social scientists can analyze social media data and begin presenting results within a matter of months (or sooner), rather than the years typically required for survey-based data collection.

Social media data also share some characteristics of observational or ethnographic work. Specifically, social media data allow researchers to collect reports of behaviors that are unsolicited and unprompted by the researcher. One might even argue that these data provide a better reflection of day-to-day social experiences than ethnographic work. Indeed, Twitter activity has been described as persons “want[ing] to know what the people around them are thinking and doing and feeling, even when co-presence isn't viable” and “shar[ing] their state of mind and status so that others who care about them feel connected” (Boyd 2009). Unlike previous observational work, however, activity on social media platforms can be captured and stored. This is in contrast to face-to-face interactions, which once passed cannot be reconstructed. A researcher conducting off-line observational work is left with only his or her perceptions (and notes) of what transpired. Interaction on social media, however, is preserved and can be reviewed multiple times and passed to other interested researchers.

Despite these exciting advantages, social scientists often see social media data as inaccessible for social science research and solely relevant to computer scientists and others in related fields. Golder and Macy (2012:7) lament, “most of the social and behavioral science using online data is coming from computer and information scientists who do not always have the training required to ask the right questions, or to recognize unfounded assumptions and socially unjust ramifications.” This perception of inaccessibility is perpetuated by the fact that social scientists often do not gain the computational skills necessary to collect and analyze online data, nor do access points in online platforms—Twitter application programming interface (API) queries, for example—lend themselves to the purposive sampling of users rather than content.

Beyond technical skills, using social media data—in particular Twitter data—for social science research is challenging due to the fact that there are numerous differences between Twitter data and surveys collected using traditional sampling techniques. When using traditional surveys, for example, researchers have comparatively few respondents but have a great deal of control over what information respondents provide. Under these conditions, respondents provide information of interest to the researchers, but the limited sample size may not produce enough variability to study less commonly observed phenomena in their entirety (e.g., self-reports of suicide attempts, eating disorders, or HIV-positive status). Data from Twitter, in many ways, are just the opposite. They are completely unsolicited but offer unprecedented volume and variability.

In addition to challenges associated with specific qualities of the data, it is difficult to gather demographic data from text-based blogs and microblogs such as Twitter. Indeed, in many online platforms, users give few or no explicit, self-reported characteristics. Demographic information is at the heart of many, if not most, social science analyses. It is often vital that researchers are able to utilize information on race, age, and gender to examine differential patterns in attitudes and behaviors. Removing the actual and perceived barriers that prevent social scientists from fully utilizing social media data expands research opportunities for social scientists and increases the potential for interdisciplinary research between computer scientists or statisticians and social and behavioral scientists, thus increasing the potential of studying complex social problems.

Focusing on the above-mentioned challenges, this article will describe an accurate and reliable data processing infrastructure that facilitates access to demographic information encoded as images within Twitter data. Furthermore, it seeks to encourage social scientists to consider Twitter as a valuable source of demographic information to answer relevant social science questions. To help illustrate this process, we examine a specific behavior—reporting an intention to not vote in the 2012 presidential election. This case study demonstrates the validity of the proposed methods as a toolkit that can be modified and applied to different questions and contexts.

In the following sections, we outline our approach to processing data from Twitter for demographic research and offer verification of the accuracy and reliability of these methods. We begin with an introduction to Twitter and a discussion of the resources and challenges associated with using Twitter data for social science research. Next, we propose a data-processing infrastructure that allows us to gather demographic information not directly reported within Twitter users’ profiles. At the heart of the strategy is a framework for using Amazon's Mechanical Turk (AMT) to efficiently code large volumes of users’ profile images, which we validate based on reliability of the AMT workers (Turkers), and comparisons between the evaluations of the Turkers and those of expert, trained coders. We illustrate the possible use of these methods by examining the racial composition of a small sample of Twitter users who state an intention to not vote in the 2012 presidential election. We conclude by addressing the benefits and challenges associated with this method of data collection, as well as the potential for future research that uses demographic data obtained from Twitter. It is our intention to present a flexible framework that can be used and adapted by researchers with varying experience regarding the collection and management of online data. The platforms we feature to collect and analyze data are not the only tools available to researchers, and while we outline our processes, we seek to highlight other potential pathways for analysis.

It is important to note that while social media data are often categorized as big data, these methods intend to focus on the research capacities of social media data that are not directly tied to the scale of the data. Although we seek to propose methods that are semiscalable and can be applied to data sets that would be time consuming for small teams of individual coders to assess, our primary goal is to leverage user metadata in a way that helps make Twitter data more accessible to the field of social science and demographic research. We believe that social media pose a number of benefits to social science research aside from their scale and we seek to highlight these capacities within this study.

Using Twitter for Political Analysis

One popular application of Twitter data is political analysis and election forecasting. Prompted by promising analyses of political opinion trends within the blogosphere and other social media outlets, some scholars have explored whether Twitter provides a useful outlet for examining political preferences as they develop. There are a number of interesting applications of Twitter data in this burgeoning body of research. Tumasjan and colleagues (2010), for example, collected 100,000 Twitter messages prior to the 2009 German federal election, analyzed these tweets for mentions of party affiliation and positive or negative sentiment, and were able to effectively conclude that the opinion trends reflected in their data paralleled the results of the election. Politicians themselves have noted the popularity of Twitter as a tool for political exchange and many now use this platform as a means of reaching out to potential constituents, though the precise relationship between electoral vulnerability and adoption is open to further exploration (Lassen and Brown 2011). Some researchers, such as Conover and colleagues (2011), have found ways to better understand the dynamics of political discussion within Twitter by mapping interaction and tweet sharing along partisan lines. While these studies carry limitations—such as the ability to gather information only from those who volunteer their opinion or the representation of politically active individuals on Twitter—they nonetheless do highlight the potential uses of Twitter data within this analytical arena.

Although many researchers have found ways to examine political trends using Twitter data, these studies unanimously lack thorough consideration of the demographic trends underlying such phenomena. Adding dimension would greatly benefit researchers looking to predict political outcomes or track changing tides in political opinion or participation among individuals of particular groups, because it would allow for a more nuanced view of behavior. This trend is not unique to political analysis, and the same can be said of other research that makes use of Twitter data. The addition of demographic data to social media data sets could greatly expand the explanatory and predictive power of analyses that use these data. In the following sections, we propose a method for gathering demographic data in conjunction with information on collective voting intentions as a means of expanding upon existing applications of social media data within social science research.

Case Study: Using Twitter to Examine Intention to Not Vote

The objective of this study is to establish a systematic approach to gathering demographic information about Twitter users, thus overcoming an important limitation of this rich data source. The first step we take is to choose a focal population. This analysis will focus specifically on extracting the demographic characteristics of individuals who express an intention to not vote in the 2012 presidential election. Subsequent steps involve (1) collecting relevant user data from Twitter, (2) organizing and cleaning the resulting data, and (3) extracting demographic information about the users by utilizing profile content. Following these steps, we assess the reliability and accuracy of the demographic information obtained.

The analysis presented here stands out from similar work in two important ways. First, it focuses not on what is said (i.e., the content of the tweets) but to whom these opinions belong (i.e., the Twitter user who report an intention to not vote). The methods presented here utilize the personal content contained within individuals’ Twitter profiles to systematically add a layer of previously unavailable demographic information. Doing so will allow social scientists to not only track trends and opinions using Twitter but also to examine the demographic characteristics of opinion-based networks and to better predict behaviors and attitudes based on social connections.

Collecting Data From Twitter

Cleaning Twitter Data

In addition to designing queries to help filter out potentially irrelevant tweets to effectively capture the demographics of intended non-voters, we use a two-pronged approach to clean our data. The first step requires one team member to read through and designate whether the tweets displayed indicate someone's intention to not vote. Second, we clean irrelevant tweets using a key word list assembled using a combination of data-driven techniques, including familiarity with the content of the data through casual reading of the tweets, and the discovery of potentially irrelevant and frequently occurring key words through the use of automated text analysis tools.

We initially used the online text visualization service Wordle (www.wordle.net) to view frequently occurring words, but found this to be fairly unhelpful (Figure 1). We then used the tm R package (Feinerer and Hornik 2014) to organize the text of the tweets into a single corpus, stem, normalize and remove stop words from the text, and organize unique terms into a document-term matrix (treating each tweet as a single document). We then used this document-term matrix to view words that appear with designated frequency as a means of detecting potentially irrelevant terms. Reducing our word list to terms that appear at least five times, for instance, allows us to detect terms that seemed not to relate to the 2012 U.S. presidential election, such as “IrvineElection,” or “#healthconscious” which referred to a specific California ballot proposition intended to regulate the labeling of genetically modified organism (GMO) crops. Similar to Wordle, this method did not yield a significant amount of insight and help with data cleaning. However, these methods may prove helpful as a means of autodetecting large portions of off-topic tweets within other corpora and should not be discounted based solely on their application to this analysis. The final key word list used to automatically clean the data is shown in Table 2. These key words were assembled primarily through familiarity with the data content developed by reading through portions of the tweets.

Open in a separate window

Figure 1

Wordle Cloud. This figure illustrates the use of Wordle for preliminary text analysis. Larger terms signify that the words occur more frequently within the document. All words are normalized and stop words are removed from the corpus prior to visualization.

Table 2

Key Words Used to Remove Irrelevant Tweets.

Term	Justification
Obama/Romney	“I won't vote for Romney/Obama” does not indicate intention to not vote at all.
Him/her	Often preceded by “'for”—refers to specific candidate, possibly outside of presidential race.
Color/black	Appears often within tweets refuting intention to vote for a particular candidate because of his skin color.
Irvine	Found within hashtag associated with specific University of California—Irvine campaign.
Health	Found within hashtag associated with GMO labeling bill.
Third	Appears in references to intentions to vote for third party candidates.
Voice	May reference television show “The Voice”.
Republican/democrat	May reference decision to not vote for one of two parties.
School	May reference homecoming elections.
Woman	No female presidential candidates; may be irrelevant. May also refer to possible future Hillary Clinton candidacy.

Open in a separate window

Note: GMO = genetically modified organism.

Ultimately, it was decided that due to the nuance of the topic used for this study, hand cleaning provides the most effective means of removing irrelevant tweets. As text analysis methods advance, however, researchers may be able to predict with high accuracy the presence of irrelevant and relevant tweets using machine learning techniques. It is important to note that the text cleaning process is especially important for this project, given that it samples Twitter users who report the intention to engage in a specific activity related to a particular event. A user who does not plan to vote in his or her high school class elections can easily fall into a collection of tweets intended to reflect users who do not plan to vote in the 2012 presidential election. Researchers who plan to gather data using a single-word query that reflects conversation surrounding a particular topic or event (e.g., gathering all tweets that reflect users’ opinion of Mitt Romney using the hashtag #Romney) may collect very few irrelevant tweets and may not need to clean the tweets at all. It is also important to note that because our unit of analysis for this project is the user rather than the tweet, some users had multiple tweets in the original data collected via the Twitter API, prior to subsampling. If the user's tweet included in the subsample indicated a desire to not vote, it was assumed that the user did in fact intend to not vote.

Application: Examining Demographic Characteristics of Intended Non-voters

Given our confidence in the reliability of the Turkers, we then consider the results of the Turkers’ evaluations of Twitter user profile pictures. Tables A1–A5 in Online Appendix A display the demographic characteristics of the intended nonvoters sampled for this study as determined by the Turkers. These tables are broken down to display evaluations for data that were (a) hand coded to remove irrelevant tweets and (b) cleaned based on the presence of irrelevant key words for both the master's required and the master's not required batches. Note that for a Twitter user to be categorized in a particular way for any given question two or more Turkers had to agree upon that categorization. In this section, we consider only the objective characteristics coded by the Turkers—not attractiveness and grooming. Note that while we assessed the reliability and accuracy of the Turker assessment using the full 1000 user subsample, assessing the demographics of intended nonvoters requires us to filter out irrelevant tweets either by hand (full filter) or using keywords (partial filter).

According to these results, intended nonvoters on Twitter tend to be male (45.51–48.49 percent) adults between the ages of 19 and 35 (65.68–69.23 percent). The majority of intended nonvoting Twitter users are reportedly white (41.97–43.89 percent) followed in descending frequency by black intended nonvoters (26.95–29.10 percent), Hispanic intended nonvoters (5.01–7.02 percent), and Asian intended nonvoters (1.84–2.80 percent).

Although the intention of this article is not to provide empirical evidence regarding the demographic characteristics of individuals who state an intention to not vote in the 2012 presidential election, we attempt to contextualize the results by comparing them to data collected for a 2012 Pew Center report on nonvoters (Kohut et al. 2012; see Table 8). This comparison is not intended to suggest that, given the current state of statistical modeling in this area, Twitter should be used to estimate population proportions or the sizes of certain populations. Instead, we currently find that the most compelling uses of Twitter data lie in studying real-time dynamics of social interactions, as we discuss below.

As expected, the data presented in Tables 5–7 do not exactly parallel the data on national nonvoters gathered by Pew. There are a number of reasons why we would not expect these values to line up directly. For one, the demographic composition of the Twitter population is different than that of the U.S. population. In addition to this, discrepancies between the demographic data on nonvoters gathered in this study may be attributable to social desirability effects in previous surveys. According to Belli et al. (1999), many researchers believe that traditional surveys have a tendency to underrepresent the total number of nonvoters in a given election, as nonvoters often refuse to disclose this information for reasons of self-presentation. Given this systematic bias in survey design, it is possible that collecting information on deviant behaviors through Twitter—a space characterized by high levels of self-disclosure—provides more information on the individuals who engage in these behaviors than traditional surveys (Marwick and Boyd 2010). Future research may determine whether this is the case. Finally, the Twitter population of interest in this article is composed of individuals whose stated intention is to not vote, and research suggests that there may not be a strong relationship between voting intentions and voting outcomes (Rogers and Aida 2011).

We do see some parallels between our data and the nonvoters examined by Pew. Both the Pew and Twitter estimates suggest that nonvoters tend to be white, male, and young (near or below the age of 35). It is clear that the Twitter estimates far exceed the Pew estimates in regard to the proportion of nonvoters who are black and young. However, these specific inconsistencies are likely attributable to two factors. For one, the population composition of Twitter does not align with the national population. As mentioned previously, Twitter is dominated by younger (ages 18–29), non-Hispanic black users (Duggan and Smith 2014). Overall, when discrepancies between the data collected from Twitter on those who state an intention to not vote and existing data on the nonvoting population are considered, these estimates support the face validity of our intended-nonvoter demographic estimates.

Discussion

It is becoming widely acknowledged that “social media offers us the opportunity for the first time to both observe human behavior and interaction in real time and on a global scale” (Golder and Macy 2012:7). Currently, the majority of researchers who are taking advantage of social media data for social science research are not social scientists but rather computer scientists and others in related fields. Perhaps one reason for this trend is the fact that key pieces of information for sociological—and specifically demographic—research, such as age, race, and gender, are currently difficult to extract from social media sites such as Twitter. Adding demographic information to Twitter data increases the breadth of social science research to which these data may be applied. Doing so opens the possibility for engaging in research that examines not only collective attitudes and opinions but also the composition of the groups driving these trends. This information could also be incorporated into network data and used as a means of examining the structure of groups that display particular behaviors or opinions, as well as how this structure changes over time.

In this article, we present an approach to extracting, processing, and analyzing the data from Twitter that can be modified to suit many topics of research. We believe that social media data, such as Twitter, present an opportunity to develop a fundamentally different approach to social science research. As with all new data collection, Twitter has certain limitations to overcome. Although many capacities of Twitter data parallels that of existing data collection, it does not replicate the results of these methods and poses new challenges. Our goal, however, is to suggest reliable and accurate methods for overcoming these challenges in order to make Twitter an accessible resource for a larger number of social scientists. While there are limits to the scalability of these methods, we believe that the use of the Mechanical Turk allows researchers to process larger amounts of data in a timelier and potentially more cost-effective way than they would using hired coders. Furthermore, we believe this analysis framework can be adapted in ways that allow social science researchers to make strong contributions in the growing field of big social data analysis.

Our preliminary analysis indicates that it is possible to collect demographic information on Twitter users using a combination of available, easy-to-use technologies. Gathering raw data from Twitter using the website's API is simple and quick. Although a small sample of 1,000 users was used in this exploratory study, the data collection platform used in this study managed to collect a fairly large sample of individuals who publicly report an intention to note vote in the 2012 presidential election (N = 12,898 unique users). Obtaining evaluations of the Twitter user's demographic characteristics—including sex, age, and race—using the AMT proved efficient and effective. Similar to previous research (Behrend et al. 2011; Bhurmester, Kwang, and Gosling 2011; Mason and Suri 2012), our results indicate that Turkers are a reliable resource for coding information. Although the data cleaning techniques suggested in this study require further exploration and may not apply to queries of all types, the results of these data collection efforts are nonetheless promising. Though the resultant demographic breakdown of intended nonvoters presented in this article do not parallel existing, national data on nonvoters exactly, they nonetheless yield results that make sense given the known biases of the Twitter environment. We are confident that the data yielded through these methods could be used to develop more complex models for social analysis.

Conclusion

Twitter is arguably the largest observational study of human behavior to date. Not only is this source of data large and easily accessible by social scientists, we contend that there is a tremendous opportunity for sociologists to use Twitter data for social science research. However, we recognize that currently a barrier exists regarding the use of these data for demographic research. The purpose of this article is to suggest a reliable and accurate means of gathering demographic data from Twitter—including age, race, and gender—as a means of overcoming this challenge. Supplementing textual data from Twitter with this additional information could open up brand new opportunities for social research and could allow demographers to model and predict behaviors and attitudes on a large scale and/or among difficult to reach populations. Restated, the potential for Twitter in social science research is yet to fully be articulated. However, we believe there are exciting opportunities to use this research to investigate social problems and other phenomena, but as a research community we cannot fully explore these opportunities without (a) widespread access and familiarity with the data by social scientists and (b) reliable information about users demographic characteristics. These are tremendous challenges, but overcoming them is worthwhile if doing so allows social science to play a role in utilizing one of the largest sources of social information available.

Supplementary Material

Acknowledgments

Biography

Footnotes

References