Opinion Mining - An Brief Introduction

This page is a brief introduction to opinion mining. It contains some background information for doing researches in opinion mining.

Supervisors:

• Prof. Irwin King
• Prof. Jimmy Lee

Students:

• Kam Tong CHAN
• Wei WEI

Subjective views on a certain topic

Views can be:

• Pros / Cons
• Suggestions for improvement
• Comparisons

Topics:

• Any objects (usually noun)
• e.g.
• Products: Car, Music Player, Camera, etc.
• People: Barack Obama, Hillary Clinton, etc.
• Organization: Bank, Government, School, etc.
• Others…

Informally: Extract the opinions given in a piece of text.

Or, more formally: A recent discipline that studies the extraction of opinions using Information Retrieval (IR), Artificial Intelligence (AI), Natural Language Processing (NLP) techniques.

• People who wants to buy a camera
  • Look for comments and reviews
• People who just bought a camera
  • Comment on it
  • Write down the usage experience
• Camera Manufacturer
  • Get feedback from customer
  • Improve their products
  • Adjust Marketing Strategies

Big business, right?

Web 2.0 nowadays provides a great medium for people to share what they want to share. This provides a great source of unstructured information (especially opinions) that may be usually (makes a lot of money?)

Identify the segments of text that contain opinions.

e.g. Opinions are in boldface

I have just entered into dslr world with 400d, before I used slr cameras.

400d is extremly well made, precise and overall feeling is vey good.

Decide the sentiment orientation of a given piece of opinion.

• Polarity
  • Positive (e.g. This camera is great!)
  • Negative (e.g. The battery life is too short.)
  • Neutral

• Polarity Scale?
  • (Most Negative) -10 … -5 … 0 (Neutral) … 5 … 10 (Most Positive)

e.g. The picture quality is good. (A positive opinion) e.g. The battery life is short. (A negative opinion)

A problem proposed by Kam Tong CHAN. The problem is related to natural language processing:

Given a text with target features and opinions extracted, decide which opinions comment on which features.

It is known to be a difficult problem in natural language processing. Let's take a look at the following example (Originated from http://en.wikipedia.org/wiki/Natural_language_processing)

Consider the phrase “pretty little girls' school”,

• Does the school look little?
• Do the girls look little?
• Do the girls look pretty?
• Does the school look pretty?

Reference:

[2009, inproceedings]

Kam Tong Chan and Irwin King, "Let's Tango -- Finding the Right Couple for Feature-Opinion Association in Sentiment Analysis," in Proc. PAKDD 2009: Advances in Knowledge Discovery and Data Mining, 13th Pacific-Asia Conference, Bangkok, Thailand, 2009.

Which one (or Who) is being commented?

e.g. He is a kind person.

Who is “he”?

e.g. The camera is great!

Which camera model are you talking about?

Given a review text, identify who made the comment.

Achieving this will allow us to build a Question-Answering System.

e.g. Who support Obama to be the next U.S. president?

Given a set of documents (crawled the web / all the reviews from a particular forum / survey results , etc.), summarize the opinion expressed with respect to the target object.

e.g. For Camera

• Picture Quality (+ve: 290, -ve 73)
• Ease of use (+ve: 57, -ve: 10)
• etc.

Detect whether opinions that are written by spammers.

1. Someone may write something to promote its own image / products
2. Someone may write something to hurt their enemies

A lexicon that contains the sentiment orientation of each term. It may be a domain specific one or a general one.

• is there a way to generate it automatically from a large corpus?

Ontology is a structural description of concepts. It defines the terminologies and hierarchical relationships of a domain.

• Who ontologies can be incorporated in opinion mining? e.g.:
  • Opinion Summarization
  • Processing Comparative Statements

• Is there a way to generate them automatically?

• Which ontology elements are essential for opinion mining? In other words, what should the ontology for opinion mining looks like?

• Can an opinion summarization system works as efficient as a search engine so that all the opinions on the web are crawled and user are able to search for any opinions?

• WordNet, SentiWordNet
• Thesaurus
• Python
  • NLTK (Natural Language Processing Toolkits)
  • Numpy, Scipy
  • Matplotlib
• Text Processing Tools
  • Sentence Splitters
  • POS (Part-of-speech) Taggers
  • Stemmers
• Crawler

• Sentiment Classification bibliography

http://liinwww.ira.uka.de/bibliography/Misc/Sentiment.html

• ACL Anthology - A Digital Archive of Research Papers in Computational Linguistics

http://acl.ldc.upenn.edu/

• Movie Review Data

http://www.cs.cornell.edu/people/pabo/movie%2Dreview%2Ddata/

• Customer Review Data

http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html

• MPQA Opinion Corpus

http://www.cs.pitt.edu/mpqa/databaserelease/

• SentiWordNet

http://sentiwordnet.isti.cnr.it/

• NLTK - Natural Language Processing Toolkits for Python

http://nltk.sourceforge.net/

• WordNet

http://wordnet.princeton.edu/

• The Sentiment & Affect Yahoo! Group

http://groups.yahoo.com/group/SentimentAI

• GI - General Inquirer

http://www.webuse.umd.edu:9090/ http://www.webuse.umd.edu:9090/tags/

• LDC Catalog

http://www.ldc.upenn.edu/Catalog/

• Opinmind

http://opinmind.com/

• Data Mining Resources

http://www.kdnuggets.com/index.html

• SIGIR - ACM SIGIR Special Interest Group on Information Retrieval

http://www.sigir.org

• CIKM - Conference on Information and Knowledge Management

http://www.cikm.org

• IDEAL - International Conference on Intelligent Data Engineering and Automated Learning

http://www.ideal2008.org/

• SIGKDD - ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

http://www.sigkdd.org

• AAAI - Association for the Advancement of Artificial Intelligence

http://www.aaai.org

• WWW - International World Wide Web Conferences

http://www.iw3c2.org/

• TREC - Text REtrieval Conference

http://trec.nist.gov/

• ACL-IJCNLP - A Joint Conference of the Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing

http://www.acl-ijcnlp-2009.org/

• WSDM - ACM International Conference on Web Search and Data Mining

http://wsdm2009.org/

• SIGDAT / EMNLP - Conference on Empirical Methods in Natural Language Processing

http://www.cs.jhu.edu/~yarowsky/sigdat.html

• WI - ACM International Conference on Web Intelligence

http://wi-consortium.org/

• SIGWEB

http://www.sigweb.org/about/

• [2009, inproceedings]

  Kam Tong Chan and Irwin King, "Let's Tango -- Finding the Right Couple for Feature-Opinion Association in Sentiment Analysis," in Proc. PAKDD 2009: Advances in Knowledge Discovery and Data Mining, 13th Pacific-Asia Conference, Bangkok, Thailand, 2009.

  [2003, inproceedings | pdf]

  Rakesh Agrawal, Sridhar Rajagopalan, Ramakrishnan Srikant, and Yirong Xu, "Mining Newsgroups Using Networks Arising From Social Behavior," in Proc. Proceedings of the 12 WWW Conference, Budapest, HU, 2003, pp. 529-535.

  [2006, inproceedings | www | pdf]

  Alina Andreevskaia and Sabine Bergler, "Mining WordNet For a Fuzzy Sentiment: Sentiment Tag Extraction From WordNet Glosses," in Proc. Proceedings EACL-06, the 11rd Conference of the European Chapter of the Association for Computational Linguistics, Trento, IT, 2006, pp. 209-216.

  [2006, inproceedings | www | pdf]

  Alina Andreevskaia and Sabine Bergler, "Semantic Tag Extraction from WordNet Glosses," in Proc. Proceedings of LREC-06, the 5th Conference on Language Resources and Evaluation, Genova, IT, 2006.

  [2006, inproceedings]

  Alina Andreevskaia and Sabine Bergler, "Sentiment Tagging of Adjectives at the Meaning Level.," in Proc. Canadian Conference on AI, 2006, pp. 336-346.

  [2006, inproceedings | www | pdf]

  Giuseppe Attardi and Maria Simi, "Extracting Dependency Relations for Opinion Mining," in Proc. Proceedings of Workshop on Distributed Agent-based Retrieval Tools, Cagliari, {IT}, 2006.

  [2006, inproceedings | www | pdf]

  Giuseppe Attardi and Maria Simi, "Blog Mining Through Opinionated Words," in Proc. Proceedings of TREC 2006, the Fifteenth Text Retrieval Conference, Gaithersburg , {US}, 2006.

  [2005, inproceedings | www | pdf]

  Anthony Aue and Michael Gamon, "Customizing Sentiment Classifiers to New Domains: a Case Study," in Proc. Submitted to RANLP-05, the International Conference on Recent Advances in Natural Language Processing, Borovets, BG, 2005.

  [2005, inproceedings | www | pdf]

  Xue Bai, Rema Padman, and Edoardo Airoldi, "On Learning Parsimonious Models for Extracting Consumer Opinions," in Proc. Proceedings of HICSS-05,the 38th Annual Hawaii International Conference on System Sciences, Washington, DC, USA, 2005, p. 75.

  [2004, inproceedings | www | pdf]

  Faye Baron and Graeme Hirst, "Collocations as Cues to Semantic Orientation," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, US, 2004.

  [2004, inproceedings | www | pdf]

  Marco Baroni and Stefano Vegnaduzzo, "Identifying Subjective Adjectives through Web-based Mutual Information," in Proc. Proceedings of KONVENS-04, Vienna, AU, 2004, pp. 17-24.

  [2003, inproceedings | www | pdf]

  Philip Beineke, Trevor Hastie, Christopher Manning, and Shivakumar Vaithyanathan, "An exploration of sentiment summarization," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, US, 2003.

  [2004, inproceedings | www | pdf]

  Philip Beineke, Trevor Hastie, and Shivakumar Vaithyanathan, "The Sentimental Factor: Improving Review Classification Via Human-Provided Information.," in Proc. Proceedings of ACL-04, the 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, ES, 2004, pp. 263-270.

  [2004, inproceedings | www | pdf]

  Steven Bethard, Hong Yu, Ashley Thornton, Vasileios Hatzivassiloglou, and Dan Jurafsky, "Automatic Extraction of Opinion Propositions and their Holders," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, US, 2004.

  [2007, inproceedings | www | pdf]

  John Blitzer, Mark Dredze, and Fernando Pereira, "Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification," in Proc. Proceedings of ACL-07, the 45th Annual Meeting of the Association of Computational Linguistics, Prague, CZ, 2007, pp. 440-447.

  [2004, inproceedings | www | pdf]

  Eric Breck and Claire Cardie, "Playing the Telephone Game: Determining the Hierarchical Structure of Perspective and Speech Expressions," in Proc. Proceedings of COLING-04, the 20th International Conference on Computational Linguistics, Geneva, CH, 2004, pp. 120-126.

  [2003, inproceedings | www | pdf]

  Claire Cardie, Wiebe Janyce, Wilson Theresa, and Diane Litman, "Combining Low-Level and Summary Representations of Opinions for Multi-Perspective Question Answering," in Proc. Proceedings of the AAAI Spring Symposium on New Directions in Question Answering, 2003, pp. 20-27.

  [2006, inproceedings | www | pdf]

  G. Carenini, R. Ng, and A. Pauls, "Interactive Multimedia Summaries of Evaluative Text," in Proc. Proceedings IUI-06, the 10th International Conference on Intelligent User Interfaces, Sydney, AU, 2006, pp. 124-131.

  [2007, inbook]

  S. Cerini, V. Compagnoni, A. Demontis, M. Formentelli, and G. Gandini, "Language resources and linguistic theory: Typology, second language acquisition, English linguistics.," , Sansò, A., Ed., Milano, IT: Franco Angeli Editore, 2007.

  [2006, inproceedings | www]

  Carmine Cesarano, A. Picariello, Reforgiato D. Recupero, A. Sagoff, V.S.Subrahmanian, and B. Dorr, "Opinion Analysis in Document Databases," in Proc. Proceedings of AAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2004, inproceedings | www | pdf]

  Nathanael Chambers, Joel Tetreault, and James Allen, "Approaches for Automatically Tagging Affect," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, US, 2004.

  [2005, inproceedings | www | pdf]

  Pimwadee Chaovalit and Lina Zhou, "Movie Review Mining: a Comparison between Supervised and Unsupervised Classification Approaches," in Proc. Proceedings of HICSS-05, the 38th Hawaii International Conference on System Sciences, 2005, p. 112.

  [2006, inproceedings | www]

  Paula Chesley, Bruce Vincent, Li Xu, and Rohini Srihari, "Using Verbs and Adjectives to Automatically Classify Blog Sentiment," in Proc. Proceedings of AAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2005, inproceedings | www | pdf]

  Yejin Choi, Claire Cardie, Ellen Riloff, and Siddharth Patwardhan, "Identifying Sources of Opinions with Conditional Random Fields and Extraction Patterns," in Proc. Proceedings of HLT-EMNLP-05, the Human Language Technology Conference/Conference on Empirical Methods in Natural Language Processing, Vancouver, CA, 2005, pp. 355-362.

  [2006, inproceedings | www]

  Hang Cui, Vibhu Mittal, and Mayur Datar, "Comparative Experiments on Sentiment Classification for Online Product Reviews," in Proc. Proceedings of AAAI-06, the 21st National Conference on Artificial Intelligence, Boston, US, 2006.

  [2001, inproceedings | www | pdf]

  Sanjiv R. Das and Mike Y. Chen, "Yahoo! for {Amazon}: {S}entiment Parsing from Small Talk on the {Web}," in Proc. Proceedings of EFA 2001, European Finance Association Annual Conference, Barcelona, ES, 2001.

  [2003, inproceedings | www | pdf]

  Kushal Dave, Steve Lawrence, and David M. Pennock, "Mining the peanut gallery: {O}pinion extraction and semantic classification of product reviews," in Proc. Proceedings of WWW-03, 12th International Conference on the {World Wide Web}, Budapest, HU, 2003, pp. 519-528.

  [2007, inproceedings | www | pdf]

  Ann Devitt and Khurshid Ahmad, "Sentiment Polarity Identification in Financial News: A Cohesion-based Approach," in Proc. Proceedings of ACL-07, the 45th Annual Meeting of the Association of Computational Linguistics, Prague, CZ, 2007, pp. 984-991.

  [2002, inbook | www | pdf]

  L. Dini and Mazzini. G., "Data Mining III," , Zanasi, Brebbia, Ebecken, and Melli, Eds., WIT Press, 2002, pp. 299-310.

  [2003, inproceedings | www | pdf]

  Stephen D. Durbin, Neal J. Richter, and Doug Warner, "A system for affective rating of texts," in Proc. Proceedings of OTC-03, 3rd Workshop on Operational Text Classification, Washington, US, 2003.

  [2004, inproceedings | www | pdf]

  Miles Efron, "Cultural Orientation: Classifying Subjective Documents by Cociation Analysis," in Proc. Proceedings of the AAAI Fall Symposium on Style and Meaning in Language, Art, Music, and Design, 2004, pp. 41-48.

  [, misc | www]

  Andrea Esuli, The Sentiment Classification Bibliography..

  [2007, inproceedings | www | pdf]

  Andrea Esuli and Fabrizio Sebastiani, "PageRanking WordNet Synsets: An Application to Opinion Mining," in Proc. Proceedings of ACL-07, the 45th Annual Meeting of the Association of Computational Linguistics, Prague, CZ, 2007, pp. 424-431.

  [2006, inproceedings | www | pdf]

  Andrea Esuli and Fabrizio Sebastiani, "Determining Term Subjectivity and Term Orientation for Opinion Mining," in Proc. Proceedings EACL-06, the 11rd Conference of the European Chapter of the Association for Computational Linguistics, Trento, IT, 2006, pp. 193-200.

  [2006, inproceedings | www | pdf]

  Andrea Esuli and Fabrizio Sebastiani, "SentiWordNet: A Publicly Available Lexical Resource for Opinion Mining," in Proc. Proceedings of LREC-06, the 5th Conference on Language Resources and Evaluation, Genova, IT, 2006.

  [2005, inproceedings | www | pdf]

  Andrea Esuli and Fabrizio Sebastiani, "Determining the semantic orientation of terms through gloss classification," in Proc. Proceedings of CIKM-05, the ACM SIGIR Conference on Information and Knowledge Management, Bremen,DE, 2005, pp. 617-624.

  [2004, inproceedings]

  Zhongchao Fei, Jian Liu, and Gengfeng Wu, "Sentiment Classification Using Phrase Patterns," in Proc. Proceedings of CIT-04, the 4th International Conference on Computer and Information Technology, Wuhan, CN, 2004, pp. 1147-1152.

  [2006, inproceedings | www]

  Boris Galitsky and Boris Kovalerchuk, "Mining the Blogosphere for Contributor?䏭 Sentiments," in Proc. Proceedings of AAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2004, inproceedings | www | pdf]

  Michael Gamon, "Sentiment classification on customer feedback data: noisy data, large feature vectors, and the role of linguistic analysis," in Proc. Proceeding of COLING-04, the 20th International Conference on Computational Linguistics, Geneva, CH, 2004, pp. 841-847.

  [2005, inproceedings | www | pdf]

  Michael Gamon and Anthony Aue, "Automatic identification of sentiment vocabulary: exploiting low association with known sentiment terms," in Proc. Proceedings of the ACL-05 Workshop on Feature Engineering for Machine Learning in Natural Language Processing, Ann Arbor, US, 2005, pp. 57-64.

  [2005, inproceedings | www | pdf]

  Michael Gamon, Anthony Aue, Simon Corston-Oliver, and Eric Ringger, "Pulse: Mining Customer Opinions from Free Text," in Proc. Proceedings of IDA-05, the 6th International Symposium on Intelligent Data Analysis, Madrid, ES, 2005, pp. 121-132.

  [2004, inproceedings]

  Ashutosh Garg, T. S. Jayram, Shivakumar Vaithyanathan, and Huaiyu Zhu, "Generalized Opinion Pooling," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, US, 2004.

  [2004, inproceedings | www | pdf]

  Ashutosh Garg, T. S. Jayram, Shivakumar Vaithyanathan, and Huaiyu Zhu, "Generalized Opinion Pooling," in Proc. Proceedings of the 8th International Symposium on Artificial Intelligence and Mathematics, Ft. Lauderdale, US, 2004.

  [2007, inproceedings | www | pdf]

  Anindya Ghose, Panagiotis Ipeirotis, and Arun Sundararajan, "Opinion Mining using Econometrics: A Case Study on Reputation Systems," in Proc. Proceedings of ACL-07, the 45th Annual Meeting of the Association of Computational Linguistics, Prague, CZ, 2007, pp. 416-423.

  [2006, inproceedings | www | pdf]

  Andrew B. Goldberg and Xiaojin Zhu, "Seeing stars when there aren't many stars: Graph-based semi-supervised learning for sentiment categorization," in Proc. HLT-NAACL 2006 Workshop on Textgraphs: Graph-based Algorithms for Natural Language Processing, New York, NY, 2006.

  [2004, inproceedings]

  Gregory Grefenstette, Yan Qu, David A. Evans, and James G. Shanahan, "Validating the Coverage of Lexical Resources for Affect Analysis," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, US, 2004.

  [2004, inproceedings | www | pdf]

  Gregory Grefenstette, Yan Qu, James G. Shanahan, and David A. Evans, "Coupling Niche Browsers and Affect Analysis for an Opinion Mining Application," in Proc. Proceeding of RIAO-04, Avignon, FR, 2004.

  [1997, inproceedings | www | pdf]

  Vasileios Hatzivassiloglou and Kathleen R. McKeown, "Predicting the semantic orientation of adjectives," in Proc. Proceedings of ACL-97, 35th Annual Meeting of the Association for Computational Linguistics, Madrid, ES, 1997, pp. 174-181.

  [2000, inproceedings | www | pdf]

  Vasileios Hatzivassiloglou and Janyce M. Wiebe, "Effects of adjective orientation and gradability on sentence subjectivity," in Proc. Proceedings of COLING-00, 18th International Conference on Computational Linguistics, Saarbrücken, GE, 2000, pp. 299-305.

  [2006, inproceedings | www]

  Minqing Hu and Bing Liu, "Opinion Feature Extraction Using Class Sequential Rules," in Proc. Proceedings ofAAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2006, inproceedings | www]

  Minqing Hu and Bing Liu, "Opinion Extraction and Summarization on the Web," in Proc. Proceedings of AAAI-06, the 21st National Conference on Artificial Intelligence, Boston, US, 2006.

  [2004, inproceedings | www | pdf]

  Minqing Hu and Bing Liu, "Mining and summarizing customer reviews," in Proc. Proceedings of KDD '04, the ACM SIGKDD international conference on Knowledge discovery and data mining, Seattle, US, 2004, pp. 168-177.

  [2004, inproceedings | www | pdf]

  Minqing Hu and Bing Liu, "Mining Opinion Features in Customer Reviews," in Proc. Proceedings of AAAI-04, the 19th National Conference on Artificial Intellgience, San Jose, US, 2004.

  [2004, inproceedings | www | pdf]

  Matthew Hurst and Kamal Nigam, "Retrieving Topical Sentiments from Online Document Collections," in Proc. Proceedings of SPIE, Document Recognition and Retrieval XI, 2004, pp. 27-34.

  [2004, inproceedings | www | pdf]

  Diana Zaiu Inkpen, Olga Feiguina, and Graeme Hirst, "Generating more-positive or more-negative text," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, US, 2004.

  [2006, inproceedings | www]

  Nitin Jindal and Bing Liu, "Mining Comparative Sentences and Relations," in Proc. Proceedings of AAAI-06, the 21st National Conference on Artificial Intelligence, Boston, US, 2006.

  [2006, inproceedings | www | pdf]

  Nitin Jindal and Bing Liu, "Identifying comparative sentences in text documents," in Proc. Proceedings of SIGIR-06, the 29th annual international ACM SIGIR conference on Research and development in information retrieval, Seattle, US, 2006, pp. 244-251.

  [2002, inproceedings | www | pdf]

  Jaap Kamps and Maarten Marx, "Words with attitude," in Proc. Proceedings of the 1st International Conference on Global WordNet, Mysore, IN, 2002, pp. 332-341.

  [2004, inproceedings | www | pdf]

  Jaap Kamps, Maarten Marx, R. ort. Mokken, and Maarten {de Rijke}, "Using {WordNet} to measure semantic orientation of adjectives," in Proc. Proceedings of LREC-04, 4th International Conference on Language Resources and Evaluation, Lisbon, PT, 2004, pp. 1115-1118.

  [2005, inproceedings | www | pdf]

  Alistair Kennedy and Diana Inkpen, "Sentiment Classification of Movie and Product Reviews Using Contextual Valence Shifters," in Proc. Proceedings of FINEXIN-05, Workshop on the Analysis of Informal and Formal Information Exchange during Negotiations, Ottawa, CA, 2005.

  [2006, inproceedings | www | pdf]

  Soo-Min Kim and Eduard Hovy, "Extracting Opinions, Opinion Holders, and Topics Expressed in Online News Media Text," in Proc. Proceedings of ACL/COLING Workshop on Sentiment and Subjectivity in Text, Sidney, {AUS}, 2006.

  [2005, inproceedings | www | pdf]

  Soo-Min Kim and Eduard Hovy, "Identifying Opinion Holders for Question Answering in Opinion Texts," in Proc. Proceedings of AAAI-05 Workshop on Question Answering in Restricted Domains, Pittsburgh, US, 2005.

  [2005, inproceedings | www | pdf]

  Soo-Min Kim and Eduard Hovy, "Automatic Detection of Opinion Bearing Words and Sentences," in Proc. Companion Volume to the Proceedings of IJCNLP-05, the Second International Joint Conference on Natural Language Processing, Jeju Island, KR, 2005, pp. 61-66.

  [2004, inproceedings | www | pdf]

  Soo-Min Kim and Eduard Hovy, "Determining the Sentiment of Opinions," in Proc. Proceedings COLING-04, the Conference on Computational Linguistics, Geneva, CH, 2004.

  [2006, inproceedings | www]

  Nozomi Kobayashi, Ryu Iida, Kentaro Inui, and Yuji Matsumoto, "Opinion Mining on the Web by Extracting Subject-Aspect-Evaluation Relasions," in Proc. Proceedings ofAAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2004, inproceedings | www | pdf]

  Nozomi Kobayashi, Kentaro Inui, Yuji Matsumoto, Kenji Tateishi, and Toshikazu Fukushima, "Collecting Evaluative Expressions for Opinion Extraction," in Proc. Proceedings of IJCNLP-04, the 1st International Joint Conference on Natural Language Processing, Hainan Island, CN, 2004, pp. 596-605.

  [2006, article | www | pdf]

  Moshe Koppel and Jonathan Schler, "The importance of neutral examples for learning sentiment," Computational Intelligence, vol. 22, iss. 2, pp. 100-116, 2006.

  [2005, inproceedings | www | pdf]

  Moshe Koppel and Jonathan Schler, "The importance of neutral examples for learning sentiment," in Proc. Proceedings of FINEXIN-05, Workshop on the Analysis of Informal and Formal Information Exchange during Negotiations, Ottawa, CA, 2005.

  [2004, inproceedings | www | pdf]

  Moshe Koppel and Itai Shtrimberg, "Good News or Bad News? Let the Market Decide," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Standford, US, 2004.

  [2006, inproceedings | www]

  Lun-Wei Ku, Hsiu-Wei Ho, and Hsin-Hsi Chen, "Novel Relationship Discovery Using Opinions Mined from the Web," in Proc. Proceedings of AAAI-06, the 21st National Conference on Artificial Intelligence, Boston, US, 2006.

  [2006, inproceedings | www]

  Lun-Wei Ku, Yu-Ting Liang, and Hsin-Hsi Chen, "Opinion Extraction, Summarization and Tracking in News and Blog Corpora," in Proc. Proceedings ofAAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2004, inproceedings | www | pdf]

  Taku Kudo and Yuji Matsumoto, "A Boosting Algorithm for Classification of Semi-Structured Text," in Proc. Proceedings of EMNLP-04, 9th Conference on Empirical Methods in Natural Language Processing, Barcelon, ES, 2004.

  [2006, inproceedings | pdf]

  Cane Wing-ki Leung, Stephen Chi-fai Chan, and Fu-lai Chung, "Integrating Collaborative Filtering and Sentiment Analysis: A Rating Inference Approach," in Proc. Proceedings of The ECAI 2006 Workshop on Recommender Systems, Riva del Garda, I, 2006, pp. 62-66.

  [2006, inproceedings | www | pdf]

  Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander Hauptmann, "Which Side are You on? Identifying Perspectives at the Document and Sentence Levels," in Proc. Proceedings of CoNLL-06, the Tenth Conference on Computational Natural Language Learning, New York, US, 2006, pp. 109-116.

  [2005, inproceedings | www | pdf]

  Bing Liu, Minqing Hu, and Junsheng Cheng, "Opinion observer: analyzing and comparing opinions on the Web," in Proc. Proceedings of WWW '05, the 14th international conference on World Wide Web, Chiba, JP, 2005, pp. 342-351.

  [2003, inproceedings | www | pdf]

  Hugo Liu, Henry Lieberman, and Ted Selker, "A Model of Textual Affect Sensing Using Real-World Knowledge," in Proc. Proceedings of IUI-03, the 8th international conference on Intelligent user interfaces, Miami, US, 2003, pp. 125-132.

  [2006, inproceedings | www | pdf]

  Rada Mihalcea Hugo \&. Liu, "A corpus-based approach to finding happiness," in Proc. Proceedings ofAAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, 2006.

  [2005, inproceedings | www | pdf]

  Beatriz Maeireizo, Diane Litman, and Rebecca Hwa, "Co-training for Predicting Emotions with Spoken Dialogue Data," in Proc. Companion Proceedings of ACL-04, the 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, ES, 2005.

  [2005, inproceedings | www | pdf]

  Shotaro Matsumoto, Hiroya Takamura, and Manabu Okumura, "Sentiment Classification Using Word Sub-sequences and Dependency Sub-trees," in Proc. Proceeding of PAKDD'05, the 9th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, Hanoi, VN, 2005, pp. 301-310.

  [2007, inproceedings | www | pdf]

  Ryan McDonald, Kerry Hannan, Tyler Neylon, Mike Wells, and Jeff Reynar, "Structured Models for Fine-to-Coarse Sentiment Analysis," in Proc. Proceedings of ACL-07, the 45th Annual Meeting of the Association of Computational Linguistics, Prague, CZ, 2007, pp. 432-439.

  [2007, inproceedings | www | pdf]

  Ben Medlock and Ted Briscoe, "Weakly Supervised Learning for Hedge Classification in Scientific Literature," in Proc. Proceedings of ACL-07, the 45th Annual Meeting of the Association of Computational Linguistics, Prague, CZ, 2007, pp. 992-999.

  [2006, inproceedings | www]

  Hongcheng Mi and I-Heng Mei, "Searching Sentiments in Blogs," in Proc. Proceedings ofAAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2007, inproceedings | www | pdf]

  Rada Mihalcea, Carmen Banea, and Janyce Wiebe, "Learning Multilingual Subjective Language via Cross-Lingual Projections," in Proc. Proceedings of ACL-07, the 45th Annual Meeting of the Association of Computational Linguistics, Prague, CZ, 2007, pp. 976-983.

  [2005, inproceedings | www]

  Rada Mihalcea and Carlo Strapparava, "Bootstraping for fun: Web-based construction of large data sets for humor recognition," in Proc. Proceedings of FINEXIN 2005, Workshop on the Analysis of Informal and Formal Information Exchange during Negotiations, Ottawa, CA, 2005.

  [2006, inproceedings | www]

  Gilad Mishne and Natalie Glance, "Predicting Movie Sales from Blogger Sentiment," in Proc. Proceedings ofAAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2006, inproceedings | www]

  Gilad Mishne and Maarten de Rijke, "Capturing Global Mood Levels Using Blog Posts," in Proc. Proceedings ofAAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2006, inproceedings | www]

  Gilad Mishne and Maarten de Rijke, "MoodViews: Tools for Blog Mood Analysis," in Proc. Proceedings ofAAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2002, inproceedings | www | pdf]

  Satoshi Morinaga, Kenji Yamanishi, Kenji Tateishi, and Toshikazu Fukushima, "Mining product reputations on the {W}eb," in Proc. Proceedings of KDD-02, 8th ACM International Conference on Knowledge Discovery and Data Mining, Edmonton, CA, 2002, pp. 341-349.

  [2004, inproceedings | www | pdf]

  Jane Morris and Graeme Hirst, "The Subjectivity of Lexical Cohesion in Text," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, US, 2004.

  [2004, inproceedings | www | pdf]

  Matthijs Mulder, Anton Nijholt, Marten den Uyl, and Peter Terpstra, "A Lexical Grammatical Implementation of Affect," in Proc. Proceedings of TSD-04, the 7th International Conference Text, Speech and Dialogue, Brno, CZ, 2004, pp. 171-178.

  [2004, inproceedings | www | pdf]

  Tony Mullen and Nigel Collier, "Sentiment analysis using support vector machines with diverse information sources," in Proc. Proceedings of EMNLP-04, 9th Conference on Empirical Methods in Natural Language Processing, Barcelon, ES, 2004.

  [2006, inproceedings | www | pdf]

  Tony Mullen and Robert Malouf, "A preliminary investigation into sentiment analysis of informal political discourse," in Proc. Proceedings ofAAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2005, inproceedings | www | pdf]

  Jin-Cheon Na, Christopher S. G. Khoo, Syin Chan, and Norraihan Bte Hamzah, "Sentiment-based search in digital libraries," in Proc. Proceedings of JCDL-05, the 5th ACM/IEEE-CS joint conference on Digital libraries, Denver, US, 2005, pp. 143-144.

  [2003, inproceedings | pdf]

  Tetsuya Nasukawa and Jeonghee Yi, "Sentiment Analysis: Capturing Favorability using Natural Language Processing," in Proc. Proceedings of the K-CAP-03, 2nd International Conference on Knowledge Capture, New York, US, 2003, pp. 70-77.

  [2004, inproceedings | www | pdf]

  Kamal Nigam and Matthew Hurst, "Towards a Robust Metric of Opinion," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Standford, US, 2004.

  [1967, book | www]

  Charles E. Osgood, George J. Suci, and Percy H. Tannenbaum, The Measurement of Meaning, of Press, U. I., Ed., University of Illinois Press, 1967.

  [2006, inproceedings | www]

  Sara Owsley, Sanjay Sood, and Kristian J. Hammond, "Domain Specific Affective Classification of Documents," in Proc. Proceedings ofAAAI-CAAW-06, the Spring Symposia on Computational Approaches to Analyzing Weblogs, Stanford, US, 2006.

  [2005, inproceedings | www | pdf]

  Bo Pang and Lillian Lee, "Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales," in Proc. Proceedings of ACL-05, 43nd Meeting of the Association for Computational Linguistics, Ann Arbor, US, 2005, pp. 115-124.

  [2004, inproceedings | www | pdf]

  Bo Pang and Lillian Lee, "A Sentimental Education: {S}entiment Analysis using Subjectivity Summarization based on Minimum Cuts," in Proc. Proceedings of ACL-04, 42nd Meeting of the Association for Computational Linguistics, Barcelona, ES, 2004, pp. 271-278.

  [2002, inproceedings | www | pdf]

  Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan, "Thumbs up? Sentiment Classification using Machine Learning Techniques," in Proc. Proceedings of EMNLP-02, the Conference on Empirical Methods in Natural Language Processing, Philadelphia, US, 2002, pp. 79-86.

  [2002, techreport | www | pdf]

  Paul Piwek, "An Annotated Bibliography of Affective Natural Language Generation," Information Technology Research Institute (ITRI), University of Brighton, ITRI-02-02, 2002.

  [2005, inproceedings | www | pdf]

  Ana-Maria Popescu and Oren Etzioni, "Extracting Product Features and Opinions from Reviews," in Proc. Proceedings of HLT-EMNLP-05, the Human Language Technology Conference/Conference on Empirical Methods in Natural Language Processing, Vancouver, CA, 2005, pp. 339-346.

  [2005, inproceedings | www | pdf]

  Jonathon Read, "Using Emoticons to reduce Dependency in Machine Learning Techniques for Sentiment Classification," in Proc. Proceedings of ACL-05, 43nd Meeting of the Association for Computational Linguistics, Ann Arbor, US, 2005.

  [2006, inproceedings | www | pdf]

  Ellen Riloff, Siddharth Patwardhan, and Janyce Wiebe, "Feature Subsumption for Opinion Analysis," in Proc. Proceedings of EMNLP-06, the Conference on Empirical Methods in Natural Language Processing, Sydney, AUS, 2006, pp. 440-448.

  [2003, inproceedings | www | pdf]

  Ellen Riloff and Janyce Wiebe, "Learning extraction patterns for subjective expressions," in Proc. Proceedings of EMNLP-03, 8th Conference on Empirical Methods in Natural Language Processing, Sapporo, JP, 2003, p. 105.

  [2005, inproceedings | www | pdf]

  Ellen Riloff, Janyce Wiebe, and William Phillips, "Exploiting Subjectivity Classification to Improve Information Extraction," in Proc. Proceedings of AAAI-05 , the 20th National Conference on Artificial Intelligence, Pittsburgh, US, 2005, pp. 1106-1111.

  [2003, inproceedings | www | pdf]

  Ellen Riloff, Janyce Wiebe, and Theresa Wilson, "Learning subjective nouns using extraction pattern bootstrapping," in Proc. Proceedings of CONLL-03, 7th Conference on Natural Language Learning, Edmonton, CA, 2003, pp. 25-32.

  [1966, book | www]

  Philip J. Stone, Dexter C. Dunphy, Marshall S. Smith, and Daniel M. Ogilvie, The General Inquirer: A Computer Approach to Content Analysis, Press, M., Ed., MIT Press, 1966.

  [2004, inproceedings | www | pdf]

  Veselin Stoyanov, Claire Cardie, Diane Litman, and Janyce Wiebe, "Evaluating an Opinion Annotation Scheme Using a New Multi-Perspective Question and Answer Corpus," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, US, 2004.

  [2001, article | www | pdf]

  Pero Subasic and Alison Huettner, "Affect analysis of text using fuzzy semantic typing," IEEE Transaction on Fuzzy Systems, vol. 9, iss. 4, pp. 483-496, 2001.

  [2006, inproceedings | www | pdf]

  Yasuhiro Suzuki, Hiroya Takamura, and Manabu Okumura, "Application of Semi-supervised Learning to Evaluative Expression Classification," in Proc. Proceedings of CICLing-06, the 7th international conference on Computational Linguistics and Intelligent Text Processing, Mexico City, MX, 2006, pp. 502-513.

  [2006, inproceedings | www | pdf]

  M. Taboada, C. Anthony, and K. Voll, "Methods for Creating Semantic Orientation Databases," in Proc. Proceeding of LREC-06, the 5th International Conference on Language Resources and Evaluation, Genova, IT, 2006, pp. 427-432.

  [2004, inproceedings | www | pdf]

  Maite Taboada and Jack Grieve, "Analyzing Appraisal Automatically," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, US, 2004, pp. 158-161.

  [2006, inproceedings | www | pdf]

  Hiroya Takamura, Takashi Inui, and Manabu Okumura, "Latent Variable Models for Semantic Orientations of Phrases," in Proc. Proceedings EACL-06, the 11rd Conference of the European Chapter of the Association for Computational Linguistics, Trento, IT, 2006, pp. 201-208.

  [2005, inproceedings | www | pdf]

  Hiroya Takamura, Takashi Inui, and Manabu Okumura, "Extracting Emotional Polarity of Words using Spin Model," in Proc. Proceedings of ACL-05, 43rd Annual Meeting of the Association for Computational Linguistics, Ann Arbor, US, 2005.

  [2004, inproceedings | www | pdf]

  Hiroya Takamura, Takashi Inui, and Manabu Okumura, "Extracting Emotional Polarity of Words using Spin Model," in Proc. Proceedings of AM2004, the Joint Workshop of Vietnamese Society of AI, SIGKBS-JSAI, ICS-IPSJ and IEICE-SIGAI on Active Mining, Hanoi, VN, 2004.

  [2005, inproceedings | www | pdf]

  Benjamin K. Y. Tsou, Raymond W. M. Yuen, Oi Yee Kwong, Tom B. Y. Lai, and Wei Lung Wong, "Polarity Classification of Celebrity Coverage in the Chinese Press," in Proc. Proceeding of the International Conference on Intelligence Analysis, McLean, US, 2005.

  [2002, inproceedings | www | pdf]

  Peter Turney, "Thumbs Up or Thumbs Down? {S}emantic Orientation Applied to Unsupervised Classification of Reviews," in Proc. Proceedings of ACL-02, 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, US, 2002, pp. 417-424.

  [2003, article | www | pdf]

  Peter D. Turney and Michael L. Littman, "Measuring praise and criticism: {I}nference of semantic orientation from association," ACM Transactions on Information Systems, vol. 21, iss. 4, pp. 315-346, 2003.

  [2004, article | www | pdf]

  Alessandro Valitutti, Carlo Strapparava, and Oliviero Stock, "Developing Affective Lexical Resources," PsychNology Journal, vol. 2, iss. 1, pp. 61-83, 2004.

  [2004, inproceedings]

  Stefano Vegnaduzzo, "Acquisition of Subjective Adjectives with Limited Resources," in Proc. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, US, 2004.

  [2005, inproceedings | www | pdf]

  Casey Whitelaw, Navendu Garg, and Shlomo Argamon, "Using Appraisal Taxonomies for Sentiment Analysis," in Proc. Proceedings of MCLC-05, the 2nd Midwest Computational Linguistic Colloquium, Columbus, US, 2005.

  [2005, inproceedings | www | pdf]

  Casey Whitelaw, Navendu Garg, and Shlomo Argamon, "Using Appraisal Taxonomies for Sentiment Analysis," in Proc. Proceedings of CIKM-05, the ACM SIGIR Conference on Information and Knowledge Management, Bremen, DE, 2005.

  [2000, inproceedings | www | pdf]

  Janyce Wiebe, "Learning Subjective Adjectives from Corpora," in Proc. Proceedings of AAAI-00, 17th Conference of the American Association for Artificial Intelligence, Austin, US, 2000, pp. 735-740.

  [2003, inproceedings | www | pdf]

  Janyce Wiebe, E. Breck, Christopher Buckley, Claire Cardie, P. Davis, B. Fraser, Diane Litman, D. Pierce, Ellen Riloff, Theresa Wilson, D. Day, and Mark Maybury, "Recognizing and Organizing Opinions Expressed in the World Press," in Proc. Proceedings of the 2003 AAAI Spring Symposium on New Directions in Question Answering, 2003.

  [1999, inproceedings | www | pdf]

  Janyce Wiebe, Rebecca F. Bruce, and Thomas P. O'Hara, "Development and use of a gold-standard data set for subjectivity classifications," in Proc. Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, College Park,US, 1999, pp. 246-253.

  [2006, inproceedings | www | pdf]

  Janyce Wiebe and Rada Mihalcea, "Word sense and subjectivity," in Proc. Proceedings of COLING/ACL-06, the 21st Conference on Computational Linguistics / Association for Computational Linguistics, Sydney, AUS, 2006, pp. 1065-1072.

  [2005, inproceedings | www | pdf]

  Janyce Wiebe and Ellen Riloff, "Creating Subjective and Objective Sentence Classifiers from Unannotated Texts," in Proc. Proceeding of CICLing-05, International Conference on Intelligent Text Processing and Computational Linguistics., Mexico City, MX, 2005, pp. 475-486.

  [2002, inproceedings | www | pdf]

  Janyce Wiebe and Theresa Wilson, "Learning to disambiguate potentially subjective expressions," in Proc. Proceedings of the 6th CoNLL, Taipei, TW, 2002, pp. 112-118.

  [2005, article | www | pdf]

  Janyce Wiebe, Theresa Wilson, and Claire Cardie, "Annotating Expressions of Opinions and Emotions in Language," Language Resources and Evaluation, vol. 1, iss. 2, 2005.

  [2001, inproceedings | www | pdf]

  Janyce M. Wiebe, Theresa Wilson, and Matthew Bell, "Identifying Collocations for Recognizing Opinions," in Proc. Proceedings of the ACL/EACL Workshop on Collocation, Toulouse, FR, 2001.

  [2004, article | www | pdf]

  Janyce M.and Wilson Theresa Wiebe, Rebecca F. Bruce, Matthew Bell, and Melanie Martin, "Learning Subjective Language," Computational linguistics, vol. 30, iss. 3, pp. 277-308, 2004.

  [2005, inproceedings | www | pdf]

  Theresa Wilson and Janyce Wiebe, "Annotating Attributions and Private States," in Proc. Proceedings of the ACL Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, Ann Arbor, US, 2005, pp. 53-60.

  [2005, inproceedings | www | pdf]

  Theresa Wilson, Janyce Wiebe, and Paul Hoffmann, "Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis," in Proc. Proceedings of Human Language Technologies Conference/Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005), Vancouver, CA, 2005.

  [2006, article | www | pdf]

  Theresa Wilson, Janyce Wiebe, and Rebecca Hwa, "Recognizing Strong And Weak Opinion Clauses," Computational Intelligence, vol. 22, iss. 2, pp. 73-99, 2006.

  [2004, inproceedings | www | pdf]

  Theresa Wilson, Janyce Wiebe, and Rebecca Hwa, "Just how mad are you? {F}inding strong and weak opinion clauses," in Proc. Proceedings of AAAI-04, 21st Conference of the American Association for Artificial Intelligence, San Jose, US, 2004, pp. 761-769.

  [2003, inproceedings | www | pdf]

  Jeonghee Yi, Tetsuya Nasukawa, Razvan Bunescu, and Wayne Niblack, "Sentiment Analyzer: Extracting Sentiments about a Given Topic using Natural Language Processing Techniques," in Proc. Proceeding of ICDM-03, the 3ird IEEE International Conference on Data Mining, Melbourne, US, 2003, pp. 427-434.

  [2005, inproceedings]

  Jeonghee Yi and Wayne Niblack, "Sentiment Mining in WebFountain.," in Proc. Proceedings ICDE-05, the 21st International Conference on Data Engineering, Tokyo, JP, 2005, pp. 1073-1083.

  [2004, inproceedings | www | pdf]

  W. Yih, P. Chang, and W. Kim, "Mining Online Deal Forums for Hot Deals," in Proc. Proceedings of WI-04, the IEEE/WIC/ACM International Conference on Web Intelligence, 2004, pp. 384-390.

  [2003, inproceedings | www | pdf]

  Hong Yu and Vasileios Hatzivassiloglou, "Towards Answering Opinion Questions: {S}eparating Facts from Opinions and Identifying the Polarity of Opinion Sentences," in Proc. Proceedings of EMNLP-03, 8th Conference on Empirical Methods in Natural Language Processing, Sapporo, JP, 2003, pp. 129-136.

  [2005, book]

  Computing attitude and affect in text, Shanahan, J., Qu, Y., and Wiebe, J., Eds., Springer, 2005.