IJBTT

Mining Software Quality from Software Reviews: Research Trends and Open Issues

	International Journal of Computer Trends and Technology (IJCTT)
	© 2016 by IJCTT Journal
	Volume-31 Number-2
	Year of Publication : 2016
	Authors : Issa Atoum, Ahmed Otoom
	DOI :  10.14445/22312803/IJCTT-V31P114

Issa Atoum, Ahmed Otoom "Mining Software Quality from Software Reviews: Research Trends and Open Issues". International Journal of Computer Trends and Technology (IJCTT) V31(2):74-83, January 2016. ISSN:2231-2803. www.ijcttjournal.org. Published by Seventh Sense Research Group.

Abstract -
Software review text fragments have considerably valuable information about users? experience. It includes a huge set of properties including the software quality. Opinion mining or sentiment analysis is concerned with analyzing textual user judgments. The application of sentiment analysis on software reviews can find a quantitative value that represents software quality. Although many software quality methods are proposed they are considered difficult to customize and many of them are limited. This article investigates the application of opinion mining as an approach to extract software quality properties. We found that the major issues of software reviews mining using sentiment analysis are due to software lifecycle and the diverse users and teams.

References
[1] A. Ghose and P. G. Ipeirotis, “Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics,” IEEE Trans. Knowl. Data Eng., vol. 23, no. 10, pp. 1498–1512, 2011.
[2] B. Liu and L. Zhang, “A Survey of Opinion Mining and Sentiment Analysis,” in Mining Text Data, C. Aggarwal, Charu C. and Zhai, Ed. Springer US, 2012, pp. 415–463.
[3] B. Pang and L. Lee, “Opinion mining and sentiment analysis,” Found. trends Inf. Retr., vol. 2, no. 1–2, pp. 1–135, 2008.
[4] B. Liu, “Sentiment Analysis and Opinion Mining,” Synth. Lect. Hum. Lang. Technol., vol. 5, no. 1, pp. 1–167, May 2012.
[5] D. A. Garvin, “What does product quality really mean,” Sloan Manage. Rev., vol. 26, no. 1, pp. 25–43, 1984.
[6] I. Atoum and C. H. Bong, “A Framework to Predict Software „Quality in Use? from Software Reviews,” in Proceedings of the First International Conference on Advanced Data and Information Engineering (DaEng-2013), vol. 285, J. Herawan, Tutut and Deris, Mustafa Mat and Abawajy, Ed. Kuala Lumpur: Springer Singapore, 2014, pp. 429–436.
[7] W. Leopairote, A. Surarerks, and N. Prompoon, “Evaluating software quality in use using user reviews mining,” in 10th International Joint Conference on Computer Science and Software Engineering (JCSSE), 2013, pp. 257–262.
[8] S. M. Basheer and S. Farook, “Movie Review Classification and Feature based Summarization of Movie Reviews,” Int. J. Comput. Trends Technol., vol. 4, 2013.
[9] Y. M. Mileva, V. Dallmeier, M. Burger, and A. Zeller, “Mining trends of library usage,” Proc. Jt. Int. Annu. ERCIM Work. Princ. Softw. Evol. Softw. Evol. Work. - IWPSE-Evol ?09, p. 57, 2009.
[10] B. Liu, “Sentiment analysis and subjectivity,” Handb. Nat. Lang. Process., vol. 2, p. 568, 2010.
[11] ISO/IEC, “ISO/IEC 25010: 2011, Systems and software engineering--Systems and software Quality Requirements and Evaluation (SQuaRE)--System and software quality models,” International Organization for Standardization, Geneva, Switzerland, 2011.
[12] H. J. La and S. D. Kim, “A model of quality-in-use for service-based mobile ecosystem,” in 2013 1st International Workshop on the Engineering of Mobile-Enabled Systems (MOBS), 2013, pp. 13–18.
[13] I. Atoum and C. H. Bong, “Measuring Software Quality in Use: State-of-the-Art and Research Challenges,” ASQ.Software Qual. Prof., vol. 17, no. 2, pp. 4–15, 2015.
[14] I. Atoum, C. H. Bong, and N. Kulathuramaiyer, “Building a Pilot Software Quality-in-Use Benchmark Dataset,” in 9th International Conference on IT in Asia, 2015.
[15] I. Atoum, C. H. Bong, and N. Kulathuramaiyer, “Towards Resolving Software Quality-in-Use Measurement Challenges,” J. Emerg. Trends Comput. Inf. Sci., vol. 5, no. 11, pp. 877–885, 2014.
[16] W. Leopairote, A. Surarerks, and N. Prompoon, “Software quality in use characteristic mining from customer reviews,” in 2012 Second International Conference on Digital Information and Communication Technology and it?s Applications (DICTAP), 2012, pp. 434–439.
[17] D. M. Blei, “Probabilistic topic models,” Commun. ACM, vol. 55, no. 4, pp. 77–84, Apr. 2012.
[18] I. Titov and R. McDonald, “Modeling online reviews with multi-grain topic models,” in Proceedings of the 17th international conference on World Wide Web, 2008, pp. 111– 120.
[19] M. Kantardzic, Data mining: concepts, models, methods, and algorithms. Wiley-IEEE Press, 2011.
[20] X. Fei, G. Ping, and M. R. Lyu, “A Novel Method for Early Software Quality Prediction Based on Support Vector Machine,” 16th IEEE Int. Symp. Softw. Reliab. Eng., no. Issre, pp. 213–222, 2005.
[21] A. Duric and F. Song, “Feature selection for sentiment analysis based on content and syntax models,” Decis. Support Syst., vol. 53, no. 4, pp. 704–711, Nov. 2012.
[22] K. Nigam, A. K. Mccallum, S. Thrun, and T. Mitchell, “Text Classification from Labeled and Unlabeled Documents using EM,” Mach. Learn., vol. 39, no. 2, pp. 103–134, 2000.
[23] M. Wen and Y. Wu, “Mining the Sentiment expectation of nouns using Bootstrapping method,” in Proceedings of the 5th international Joint conference on natural Language Processing (iJcnLP-2010), 2011, pp. 1423–1427.
[24] Z. Zhai, B. Liu, J. Wang, H. Xu, and P. Jia, “Product Feature Grouping for Opinion Mining,” Intell. Syst. IEEE, vol. 27, no. 4, pp. 37–44, 2012.
[25] S. T. W. Wendy, B. C. How, and I. Atoum, “Using Latent Semantic Analysis to Identify Quality in Use ( QU ) Indicators from User Reviews,” in The International Conference on Artificial Intelligence and Pattern Recognition (AIPR2014), 2014, pp. 143–151.
[26] S. Deerwester and S. Dumais, “Indexing by latent semantic analysis,” J. Am. Soc. Inf. Sci., vol. 41, no. 6, pp. 391–407, Sep. 1990.
[27] T. K. Landauer, P. W. Foltz, and D. Laham, “An introduction to latent semantic analysis,” Discourse Process., vol. 25, no. 2–3, pp. 259–284, 1998.
[28] T. Hofmann, “Probabilistic latent semantic indexing,” in Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, 1999, pp. 50–57.
[29] S.-M. Kim and E. Hovy, “Determining the sentiment of opinions,” in Proceedings of the 20th international conference on Computational Linguistics, 2004.
[30] N. Kobayashi, K. Inui, and Y. Matsumoto, “Extracting aspect-evaluation and aspect-of relations in opinion mining,” in Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), 2007, pp. 1065–1074.
[31] B. Pang and L. Lee, “A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts,” in Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, 2004.
[32] J. M. Wiebe, R. F. Bruce, and T. P. O?Hara, “Development and use of a gold-standard data set for subjectivity classifications,” in Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, 1999, pp. 246–253.
[33] S.-M. Kim and E. Hovy, “Crystal: Analyzing predictive opinions on the web,” in Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), 2007, pp. 1056–1064.
[34] F. Benamara, V. Popescu, B. Chardon, Y. Mathieu, and Others, “Towards context-based subjectivity analysis,” in Proceedings of 5th International Joint Conference on Natural Language Processing, 2011, pp. 1180–1188.
[35] L. Barbosa and J. Feng, “Robust sentiment detection on Twitter from biased and noisy data,” in Proceedings of the 23rd International Conference on Computational Linguistics: Posters, 2010, pp. 36–44.
[36] J. M. Wiebe, “Learning subjective adjectives from corpora,” in Proceedings of the National Conference on Artificial Intelligence, 2000, pp. 735–741.
[37] V. Hatzivassiloglou and K. R. McKeown, “Predicting the semantic orientation of adjectives,” in Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics, 1997, pp. 174–181.
[38] E. Riloff and J. Wiebe, “Learning extraction patterns for subjective expressions,” in Proceedings of the 2003 conference on Empirical methods in natural language processing, 2003, pp. 105–112.
[39] M. Hu and B. Liu, “Opinion extraction and summarization on the web,” in Proceedings Of The National Conference On Artificial Intelligence, 2006, vol. 21, no. 2, p. 1621.
[40] S. Blair-Goldensohn, K. Hannan, R. McDonald, T. Neylon, G. A. Reis, and J. Reynar, “Building a sentiment summarizer for local service reviews,” in WWW Workshop on NLP in the Information Explosion Era, 2008.
[41] P. D. Turney and M. L. Littman, “Measuring praise and criticism: Inference of semantic orientation from association,” ACM Trans. Inf. Syst., vol. 21, no. 4, pp. 315–346, Oct. 2003.
[42] J. Kamps, M. Marx, R. J. Mokken, and M. de Rijke, “Using wordnet to measure semantic orientation of adjectives,” in Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004), 2004, vol. IV, pp. 1115–1118.
[43] L. Velikovich, S. Blair-Goldensohn, K. Hannan, and R. McDonald, “The viability of web-derived polarity lexicons,” in Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2010, pp. 777–785.
[44] H. Kanayama and T. Nasukawa, “Fully automatic lexicon expansion for domain-oriented sentiment analysis,” in Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, 2006, pp. 355–363.
[45] X. Ding, B. Liu, and P. S. Yu, “A holistic lexicon-based approach to opinion mining,” in Proceedings of the international conference on Web search and web data mining, 2008, pp. 231–240.
[46] L. Zhang and B. Liu, “Identifying noun product features that imply opinions,” in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers, 2011, vol. 2, pp. 575– 580.
[47] W. Du, S. Tan, X. Cheng, and X. Yun, “Adapting information bottleneck method for automatic construction of domainoriented sentiment lexicon,” in Proceedings of the third ACM international conference on Web search and data mining, 2010, pp. 111–120.
[48] G. Qiu, B. Liu, J. Bu, and C. Chen, “Expanding domain sentiment lexicon through double propagation,” in Proceedings of the 21st international jont conference on Artifical intelligence, 2009, pp. 1199–1204.
[49] G. Qiu, B. Liu, J. Bu, and C. Chen, “Opinion word expansion and target extraction through double propagation,” Comput. Linguist., vol. 37, no. 1, pp. 9–27, 2011.
[50] S. Feng, R. Bose, and Y. Choi, “Learning general connotation of words using graph-based algorithms,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2011, pp. 1092–1103.
[51] P. University, “About WordNet,” Princeton University, 2010. [Online]. Available: http://wordnet.princeton.edu.
[52] J. Liu and S. Seneff, “Review sentiment scoring via a parseand- paraphrase paradigm,” in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1, 2009, pp. 161–169.
[53] L. Qu, G. Ifrim, and G. Weikum, “The Bag-of-opinions Method for Review Rating Prediction from Sparse Text Patterns,” in Proceedings of the 23rd International Conference on Computational Linguistics, 2010, pp. 913– 921.
[54] P. D. P. Turney, “Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, 2002, no. July, pp. 417–424.
[55] M. Hu and B. Liu, “Mining and summarizing customer reviews,” in Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 2004, pp. 168–177.
[56] K. P. P. Shein and T. T. S. Nyunt, “Sentiment Classification Based on Ontology and SVM Classifier,” in Second International Conference on Communication Software and Networks, 2010, pp. 169–172.
[57] H. Yu and V. Hatzivassiloglou, “Towards answering opinion questions: separating facts from opinions and identifying the polarity of opinion sentences,” in Proceedings of the 2003 conference on Empirical methods in natural language processing, 2003, pp. 129–136.
[58] M. Gamon and A. Aue, “Pulse: Mining Customer Opinions from Free Text,” in Advances in Intelligent Data Analysis VI, vol. 3646, A. Famili, A.Fazel and Kok, JoostN. and Peña, JoséM. and Siebes, Arno and Feelders, Ed. Springer Berlin Heidelberg, 2005, pp. 121–132.
[59] R. Narayanan, B. Liu, and A. Choudhary, “Sentiment analysis of conditional sentences,” in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1, 2009, pp. 180–189.
[60] L. Zhuang, F. Jing, and X.-Y. Zhu, “Movie review mining and summarization,” in Proceedings of the 15th ACM international conference on Information and knowledge management, 2006, pp. 43–50.
[61] L. Jiang, M. Yu, M. Zhou, X. Liu, and T. Zhao, “Targetdependent twitter sentiment classification,” in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011, vol. 1, pp. 151–160.
[62] W. Wei and J. A. Gulla, “Sentiment learning on product reviews via sentiment ontology tree,” in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 2010, pp. 404–413.
[63] K. Moilanen and S. Pulman, “Sentiment composition,” in Proceedings of the Recent Advances in Natural Language Processing International Conference, 2007, pp. 378–382.
[64] L. Polanyi and A. Zaenen, “Contextual Valence Shifters,” in Computing Attitude and Affect in Text: Theory and Applications, J. Shanahan, JamesG. and Qu, Yan and Wiebe, Ed. Springer Netherlands, 2006, pp. 1–10.
[65] N. Jindal and B. Liu, “Identifying comparative sentences in text documents,” in Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, 2006, pp. 244–251.
[66] W. Jin and H. H. Ho, “A novel lexicalized HMM-based learning framework for web opinion mining.,” in Proceedings of the 26th Annual International Conference on Machine Learning, 2009, pp. 465–472.
[67] A. Mukherjee and B. Liu, “aspect extraction through Semi- Supervised modeling,” in Proceedings of 50th anunal meeting of association for computational Linguistics (ACL-2012), 2012, no. July, pp. 339–348.
[68] B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up?: sentiment classification using machine learning techniques,” in Proceedings of the ACL-02 conference on Empirical methods in natural language processing - Volume 10, 2002, no. July, pp. 79–86.
[69] T. Wilson, J. Wiebe, and R. Hwa, “Just how mad are you? Finding strong and weak opinion clauses,” in Proceedings of the National Conference on Artificial Intelligence, 2004, pp. 761–769.
[70] M. Hu and B. Liu, “Opinion feature extraction using class sequential rules,” in Proceedings of the Spring Symposia on Computational Approaches to Analyzing Weblogs, 2006, no. 3.
[71] B. Liu, M. Hu, and J. Cheng, “Opinion observer: analyzing and comparing opinions on the Web,” in Proceedings of the 14th international conference on World Wide Web, 2005, pp. 342–351.
[72] A.-M. Popescu, B. Nguyen, and O. Etzioni, “OPINE: extracting product features and opinions from reviews,” in Proceedings of HLT/EMNLP on Interactive Demonstrations, 2005, pp. 32–33.
[73] W. Zhang, H. Xu, and W. Wan, “Weakness Finder: Find product weakness from Chinese reviews by using aspects based sentiment analysis,” Expert Syst. Appl., vol. 39, no. 11, pp. 10283–10291, Sep. 2012.
[74] J. Yi, T. Nasukawa, R. Bunescu, and W. Niblack, “Sentiment analyzer: extracting sentiments about a given topic using natural language processing techniques,” in Third IEEE International Conference on Data Mining ICDM 2003., 2003, pp. 427–434.
[75] L. Shang, H. Wang, X. Dai, and M. Zhang, “Opinion Target Extraction for Short Comments,” PRICAI 2012 Trends Artif. Intell., vol. 7458, pp. 434–439, 2012.
[76] A. Bhattarai, N. Niraula, V. Rus, and K. Lin, “A Domain Independent Framework to Extract and Aggregate Analogous Features in Online Reviews,” in Computational Linguistics and Intelligent Text Processing, vol. 7181, A. Gelbukh, Ed. Springer Berlin Heidelberg, 2012, pp. 568–579.
[77] L. Zhang, B. Liu, S. H. S. H. Lim, and E. O?Brien-Strain, “Extracting and ranking product features in opinion documents,” in Proceedings of the 23rd International Conference on Computational Linguistics: Posters, 2010, no. August, pp. 1462–1470.
[78] Z. Hai, K. Chang, and G. Cong, “One seed to find them all: mining opinion features via association,” in Proceedings of the 21st ACM international conference on Information and knowledge management, 2012, pp. 255–264.
[79] B. Wang and H. Wang, “Bootstrapping both product features and opinion words from chinese customer reviews with crossinducing,” in Proceedings of The Third International Joint Conference on Natural Language Processing (IJCNLP), 2008.
[80] L. Ku, Y. Liang, and H. Chen, “Opinion extraction, summarization and tracking in news and blog corpora,” in Proceedings of AAAI-2006 Spring Symposium on Computational Approaches to Analyzing Weblogs, 2006, pp. 100–107.
[81] L. Tesnière and J. Fourquet, Eléments de syntaxe structurale, vol. 1965. Klincksieck Paris, 1959.
[82] I. Titov and R. McDonald, “A joint model of text and aspect ratings for sentiment summarization,” in Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics, 2008, pp. 308–316.
[83] D. M. Blei and J. D. McAuliffe, “Supervised topic models,” in Advances in Neural Information Processing Systems 20 (NIPS 2007), 2007, pp. 121–128.
[84] L. Chen, L. Qi, and F. Wang, “Comparison of feature-level learning methods for mining online consumer reviews,” Expert Syst. Appl., vol. 39, no. 10, pp. 9588–9601, Aug. 2012.
[85] Q. Su, X. Xu, GuoHonglei, Z. Guo, X. Wu, Z. Xiaoxun, B. Swen, and Z. Su, “Hidden sentiment association in chinese web opinion mining,” in Proceedings of the 17th international conference on World Wide Web, 2008, pp. 959– 968.
[86] L. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” Proc. IEEE, vol. 77, no. 2, pp. 257–286, 1989.
[87] N. Jakob and I. Gurevych, “Extracting Opinion Targets in a Single- and Cross-domain Setting with Conditional Random Fields,” in Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, 2010, pp. 1035– 1045.
[88] J. Lafferty, A. Mccallum, and F. C. N. Pereira, “Conditional Random Fields : Probabilistic Models for Segmenting and Labeling Sequence Data,” in Proceedings of ICML?01, 2001, vol. 2001, no. Icml, pp. 282–289.
[89] J. Yu, Z.-J. Zha, M. Wang, K. Wang, and T.-S. Chua, “Domain-assisted product aspect hierarchy generation: towards hierarchical organization of unstructured consumer reviews,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2011, pp. 140– 150.
[90] H. Jin, M. Huang, and X. Zhu, “Sentiment Analysis with Multi-source Product Reviews,” Intell. Comput. Technol., pp. 301–308, 2012.
[91] Q. Mei, X. Ling, M. Wondra, H. Su, and C. Zhai, “Topic sentiment mixture: modeling facets and opinions in weblogs,” in Proceedings of the 16th international conference on World Wide Web, 2007, pp. 171–180.
[92] F. Su and K. Markert, “From words to senses: a case study of subjectivity recognition,” in Proceedings of the 22nd International Conference on Computational Linguistics - Volume 1, 2008, pp. 825–832.
[93] J. Hai, Zhen and Chang, Kuiyu and Kim, “Implicit Feature Identification via Co-occurrence Association Rule Mining,” in Computational Linguistics and Intelligent Text Processing, A. Gelbukh, Ed. Springer Berlin Heidelberg, 2011, pp. 393– 404.
[94] Z. Zhai, B. Liu, H. Xu, and P. Jia, “Constrained LDA for Grouping Product Features in Opinion Mining,” in Advances in Knowledge Discovery and Data Mining, vol. 6634, J. Huang, L. Cao, and J. Srivastava, Eds. Springer Berlin Heidelberg, 2011, pp. 448–459.
[95] Z. Zhai, B. Liu, H. Xu, and P. Jia, “Grouping product features using semi-supervised learning with soft-constraints,” in Proceedings of the 23rd International Conference on Computational Linguistics, 2010, pp. 1272–1280.
[96] S. Mukherjee and P. Bhattacharyya, “Feature Specific Sentiment Analysis for Product Reviews,” in Computational Linguistics and Intelligent Text Processing, vol. 7181, A. Gelbukh, Ed. Springer Berlin / Heidelberg, 2012, pp. 475– 487.
[97] D. Andrzejewski, X. Zhu, and M. Craven, “Incorporating domain knowledge into topic modeling via Dirichlet Forest priors,” in Proceedings of the 26th Annual International Conference on Machine Learning, 2009, vol. 382, no. 26, pp. 25–32.
[98] H. Guo, H. Zhu, Z. Guo, X. Zhang, and Z. Su, “Product feature categorization with multilevel latent semantic association,” in Proceedings of the 18th ACM conference on Information and knowledge management, 2009, pp. 1087– 1096.
[99] G. Carenini, R. T. Ng, and E. Zwart, “Extracting knowledge from evaluative text,” in Proceedings of the 3rd international conference on Knowledge capture, 2005, pp. 11–18.
[100] A. Dempster, N. Laird, and D. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. R. Stat. Soc. Ser. B, pp. 1–38, 1977.
[101] R. Ng and A. Pauls, “Multi-document summarization of evaluative text,” in In Proceedings of the 11st Conference of the European Chapter of the Association for Computational Linguistics, 2006.
[102] S. Tata and B. Di Eugenio, “Generating fine-grained reviews of songs from album reviews,” in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 2010, pp. 1376–1385.
[103] K. Lerman, S. Blair-Goldensohn, and R. McDonald, “Sentiment summarization: evaluating and learning user preferences,” in Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, 2009, pp. 514–522.
[104] J. Liu, Y. Cao, C.-Y. Lin, Y. Huang, and M. Zhou, “Lowquality product review detection in opinion summarization,” in Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), 2007, pp. 334–342.
[105] Y. Lu, H. Duan, H. Wang, and C. Zhai, “Exploiting structured ontology to organize scattered online opinions,” in Proceedings of the 23rd International Conference on Computational Linguistics, 2010, pp. 734–742.
[106] Y. Lu, C. Zhai, and N. Sundaresan, “Rated aspect summarization of short comments,” in Proceedings of the 18th international conference on World wide web, 2009, pp. 131–140.
[107] J. Wiebe, T. Wilson, and C. Cardie, “Annotating Expressions of Opinions and Emotions in Language,” Lang. Resour. Eval., vol. 39, no. 2–3, pp. 165–210, 2005.
[108] C. Toprak, N. Jakob, and I. Gurevych, “Sentence and expression level annotation of opinions in user-generated discourse,” in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 2010, pp. 575– 584.
[109] X. Ding and B. Liu, “Resolving object and attribute coreference in opinion mining,” in Proceedings of the 23rd International Conference on Computational Linguistics, 2010, pp. 268–276.
[110] C. Akkaya, J. Wiebe, and R. Mihalcea, “Subjectivity word sense disambiguation,” in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1, 2009, pp. 190–199.

Keywords
Software Quality-in-use, Clustering, Topic Models, Opinion Mining Tasks.