Sentiment Strength Detection in Short Informal Text - School of ...

Sentiment Strength Detection in Short Informal Text[1]
Mike Thelwall, Kevan Buckley, Georgios Paltoglou, Di Cai
Statistical Cybermetrics Research Group, School of Computing and
Information Technology, University of Wolverhampton, Wulfruna Street,
Wolverhampton WV1 1SB, UK.
E-mail: m.thelwall@wlv.ac.uk, K.A.Buckley@wlv.ac.uk, G.Paltoglou@wlv.ac.uk,
caid@wlv.ac.uk
Tel: +44 1902 321470 Fax: +44 1902 321478
Arvid Kappas
School of Humanities and Social Sciences, Jacobs University Bremen, Campus
Ring 1,
28759 Bremen, Germany
E-mail: a.kappas@jacobs-university.de
Tel: +49 421 200-3441 A huge number of informal messages are posted every day in social network
sites, blogs and discussion forums. Emotions seem to be frequently
important in these texts for expressing friendship, showing social support
or as part of online arguments. Algorithms to identify sentiment and
sentiment strength are needed to help understand the role of emotion in
this informal communication and also to identify inappropriate or anomalous
affective utterances, potentially associated with threatening behaviour to
the self or others. Nevertheless, existing sentiment detection algorithms
tend to be commercially-oriented, designed to identify opinions about
products rather than user behaviours. This article partly fills this gap
with a new algorithm, SentiStrength, to extract sentiment strength from
informal English text, using new methods to exploit the de-facto grammars
and spelling styles of cyberspace. Applied to MySpace comments and with a
lookup table of term sentiment strengths optimised by machine learning,
SentiStrength is able to predict positive emotion with 60.6% accuracy and
negative emotion with 72.8% accuracy, both based upon strength scales of 1-
5. The former, but not the latter, is better than baseline and a wide range
of general machine learning approaches. Introduction Most opinion mining algorithms attempt to identify the polarity of
sentiment in text: positive, negative or neutral. Whilst for many
applications this is sufficient, texts often contain a mix of positive and
negative sentiment and for some applications it is necessary to detect both
simultaneously and also to detect the strength of sentiment expressed. For
instance, programs to monitor sentiment in online communication, perhaps
designed to identify and intervene when inappropriate emotions are used or
to identify at-risk users (e.g., Huang, Goh, & Liew, 2007), would need to
be sensitive to the strength of sentiment expressed and whether
participants were appropriately balancing positive and negative sentiment.
In addition, basic research to understand the role of emotion in online
communication (e.g., Derks, Fischer, & Bos, 2008; e.g., Hancock, Gee,
Ciaccio, & Lin, 2008; Nardi, 2005) would also benefit from fine-grained
sentiment detection, as would the growing body of psychology and other
social science research into the role of sentiment in various types of
discussion or general discourse (Balahur, Kozareva, & Montoyo, 2009;
Pennebaker, Mehl, & Niederhoffer, 2003; Short & Palmer, 2008).
A complicating factor for online sentiment detection is that there are
many electronic communications media in which text based communication in
English seems to frequently ignore the rules of grammar and spelling.
Perhaps most famous is mobile phone text language with its abbreviations,
emoticons and truncated sentences (Grinter & Eldridge, 2003; Thurlow, 2003)
but similar styles are evident in many other forms of computer mediated
communication, including chatrooms, bulletin boards and social network
sites (Baron, 2003; Crystal, 2006). Widely recognised innovations include
emoticons like :-) that are reasonably effective in conveying emotion
(Derks, Bos, & von Grumbkow, 2008; Fullwood & Martino, 2007) and word
abbreviations like m8 (mate) and u (you) (Thurlow, 2003). Although
sometimes seen as poor language use, these are a natural response to the
technological affordances and social factors associated with a system
(Baron, 2003; Walther & Parks, 2002). These variations cause problems
because typical linguistic sentiment analysis programs start with part of
speech tagging (e.g., Brill, 1992), which is reliant upon standard spelling
and grammar, and/or apply rules that assume at least correct spelling, if
not correct grammar. Spelling correction can be useful in this context, but
this is based upon the assumption that spelling deviations are likely to be
accidental mistakes (Kukich, 1992; Pollock & Zamora, 1984) and so current
algorithms are unlikely to work well with deliberately non-standard
spellings. Nevertheless, there is a range of common abbreviations and new
words that a linguistic algorithm could, in principle, detect. Non-
linguistic machine learning algorithms typically predict sentiment based
upon occurrences of individual words, word pairs and word triples in
documents. These may also perform poorly on informal text because of
spelling problems and creativity in sentiment expression, even if a large
training corpus is available (see below).
The social network site MySpace, the source of the data used in the
current study, is known for its young members, its musical orientation and
its informal communication patterns (boyd, 2008; boyd, 2008). Probably as a
result of these factors 95% of English public comments exchanged between
friends contain at least one abbreviation from standard English (Thelwall,
2009). Common features include emoticons, texting-style abbreviations and
the use of repeated letters or punctuation for emphasis (e.g., a loooong
time, Hi!!!). Comments are typically short (mean 18.7 words, median 13
words, 68 characters) (Thelwall, 2009) but positive emotion is common
(Thelwall, Wilkinson, & Uppal, 2010).
This article proposes a new algorithm, SentiStrength, which employs
several novel methods to simultaneously extract positive and negative
sentiment strength from short informal electronic text. SentiStrength uses
a dictionary of sentiment words with associated strength measures and
exploits a range of recognised non-standard spellings and other common
textual methods of expressing sentiment. SentiStrength was developed
through an initial set of 2,600 human-classified MySpace comments, and
evaluated on a further random sample of 1,041 MySpace comments. Note that
in some articles, but not in emotion psychology, the term sentiment refers
to affect split into positive, negative and neutral whereas the term
emotion refers to more differentiated affect (e.g., happy, sad,
frightened). In contrast, the two terms are used as synonyms here, with
their meaning effectively defined by the coder instructions described
below. The main novel contributions of this paper are: a machine learning
approach to optimise sentiment term weightings; methods for extracting
sentiment from repeated letter non-standard spelling in informal text; and
a related spelling correction method. In addition, the paper introduces a
dual 5-point system for positive and negative sentiment, a corpus of 1,041
MySpace comments for this system, and a new overall sentiment strength
detection system that combines novel and existing methods. Background and Related Work This literature review section discussed related opinion mining/sentiment
analysis research as well as some relevant contributions from emotion
psychology. Opinion mining Opinion mining, also known as sentiment analysis, is the extraction of
positive or negative opinions from (unstructured) text (Pang & Lee, 2008).
The many applications of opinion mining include detecting movie popularity
from multiple online reviews and diagnosing which parts of a vehicle are
liked or disliked by owners through their comments in a dedicated site or
forum. There are also applications unrelated to marketing, such as
differentiating between emotional and informative social media content
(Denecke & Nejdl, 2009).
Opinion mining typically occurs in two or three stages, although more
may be needed for some tasks (e.g., Balahur et al., 2010). First, the input
text is split into sections, such as sentences, and each section tested to
see if it contains any sentiment: if it is subjective or objective (Pang &
Lee, 2004). Second, the subjective sentences are analysed to detect their
sentiment polarity. Finally, the object about which the opinion is
expressed may be extracted (e.g., Gamon, Aue, Corston-Oliver, & Ringger,
2005). Opinion mining normally deals with only positive and negative
sentiment rather than discrete emotions (e.g., happiness, surprise), does
not detect sentiment strength (but sometimes uses the strength of
association of words with positive or negative sentiment, e.g., Kaji &
Kitsuregawa, 2007), and does not simultaneously identify both positive and
negative emotions. Nevertheless, such opinion mining research can aid the
simultaneous assessment of positive and negative sentiment strength both
because of its general insights into sentiment analysis and also because
most techniques could, in theory, be repurposed for this new task. For
example, phrase analysis techniques could be applied to identify both
positive and negative sentiment even within individual sentences (Choi &
Cardie, 2008; Wilson, 2008; Wilson, Wiebe, & Hoffman, 2009).
Opinion mining algorithms often use machine learning to identify
general features associated with positive and negative sentiment, where