Text Mining in Twitter with Spark and Scala

Απλή σελίδα τεκμηρίου
dc.contributor.authorAdam, Simitosen
dc.date.accessioned2017-05-09T07:15:24Z
dc.date.available2017-05-10T00:00:17Z
dc.date.issued2017-05-09
dc.identifier.urihttps://repository.ihu.edu.gr//xmlui/handle/11544/15305
dc.rightsDefault License
dc.titleText Mining in Twitter with Spark and Scalaen
heal.abstractThis dissertation was written as a part of the MSc in “Mobile and Web Computing” at the International Hellenic University, Thessaloniki, Greece. Text Mining is a research area that tries to solve the document overabundance problem by using Data Mining, Machine Learning, Natural Language Processing, Information Retrieval, and Knowledge Management techniques. Text Mining’s main purpose is the automate documents categorization in classes. People’s thoughts and opinions have always been studied and researched by the sciences of sociology and history. Social Media revolution has made opinion expression a very easy, simple and quick procedure. Thanks to Social Media an Internet user can propagate their opinion and read other users’ opinions as well. As a result, the Internet is “flooded” by a vast volume of data that is difficult to be managed. Social Media is one of the factors that contribute to the phenomenon called “Big Data” in computer science. The object of this master thesis is the collection and manipulation of social media users’ opinions about political situation in Greece by using text mining methods. Specifically, the application developed crawls opinions for Greek parliament members from Twitter social medium and categorizes them in positive, neutral, and negative. Statistics produced are indicative for each member’s popularity.en
heal.academicPublisherIHUen
heal.academicPublisherIDihuen_US
heal.accessfreeen_US
heal.advisorNamePapadopoulos, Apostolosen
heal.classificationInformation Technologyen
heal.committeeMemberNameBerberidis, Christosen
heal.committeeMemberNameAmpatzoglou, Apostolosen
heal.committeeMemberNameGatzianas, Mariosen
heal.creatorID.dhareIDa.simitos@ihu.edu.gr
heal.fileFormatpdfen_US
heal.keywordURI.LCSHData mining
heal.keywordURI.LCSHData mining--Computer programs
heal.keywordURI.LCSHData mining--Data processing
heal.keywordURI.LCSHData mining--Social aspects
heal.keywordURI.LCSHData mining--Statistical methods
heal.keywordURI.LCSHSocial media
heal.keywordURI.LCSHSocial media--Political aspects
heal.keywordURI.LCSHTwitter
heal.keywordURI.LCSHTwitter--Political aspects--Greece
heal.keywordURI.LCSHTwitter--Social aspects
heal.keywordURI.LCSHSpark (Electronic resource : Apache Software Foundation)
heal.keywordURI.LCSHScala (Computer program language)
heal.keywordURI.LCSHSPARK (Computer program language)
heal.keywordURI.LCSHInformation retrieval
heal.keywordURI.LCSHInformation retrieval--Data processing
heal.keywordURI.LCSHInformation retrieval--Technological innovations
heal.languageenen_US
heal.licensehttp://creativecommons.org/licenses/by-nc/4.0en_US
heal.numberOfPages78en_US
heal.publicationDate2016-12-23
heal.recordProviderSchool of Science and Technology, MSc in Mobile and Web Computingen_US
heal.secondaryTitleTwitter as Political Barometer in Greeceen
heal.spatialCoverageGreeceen
heal.tableOfContentsAbstract Contents List of Pictures List of Tables 1 Introduction ........................ 2 Big Data ............................ 2.1 What is Big Data............................. 2.2 Big Data Challenges ................... 2.3 Managing Big Data ....................... 2.3.1 Spark .......................... Spark stack .......................... Spark Core ..................................... Spark SQL ............................ Spark Streaming............................. MLlib ...................... GraphX .............................. Cluster Managers .......................................... Spark Runtime Architecture .................................. The Driver ..................................... Executors ..................................... Cluster Manager .............................................. 2.3.2 Scala ..................................... 3 Twitter ........................................ 3.1 Twitter Analytics ........................................... 3.2 Crawling Twitter Data ...................................... 3.2.1 Open Authentication .................................... 3.2.2 Collecting search results Collecting tweets using REST API ................. Collecting tweets using Streaming API .................. 3.3 Tweets Sentiment Analysis ............................. 3.4 Twitter and Politics .......................................... 3.4.1 Twitter for political communication ...................... 3.4.2 Twitter users as voters ................................... 3.4.3 Twitter in Greek political reality ..................... 4 Text Mining ....................................................... 4.1 Text Retrieval Methods .................................. 4.2 Finding Similar Documents ............................. 4.3 Document Classification Analysis ................. 4.4 Text retrieval evaluation methods .................... 4.5 Latent Semantic Indexing ................................ 5 The PolBar Application ...................................... 5.1 Tweets Collection ............................................ 5.1.1 Communicating with Twitter API ....................... 5.1.2 Choosing the suitable search keyword ..................... 5.1.3 Organizing keywords............................................................................... 5.1.4 Crawling and preprocessing tweets .............. 5.2 Tweets Storage ................................................ 5.3 Tweets Analysis and Classification ................. 5.3.1 Creating the training dataset ......................... Stopwords ......................................................... 5.3.2 Classifiers evaluation ...................................... Logistic Regression ................................................ Naïve Bayes .......................................................... Decision Tree ......................................................... Random Forest ........................................................ 5.4 Results Presentation .................................... 5.5 Extra Experiments .......................................... 5.5.1 Experiment with different datasets types ............ 5.5.2 Experiment with different datasets size ........... 6 Conclusions .................................................... 7 Future Prospects ............................................... Bibliography ............................................................ Appendix A .......................................................... Instance of Data Table........................................... Appendix B ............................................................. Instance of Month Statistics Table .......................... Appendix C .......................................................... Instance of Total Statistics Table ............................en
heal.typemasterThesisen_US

Αρχεία

Πρωτότυπος φάκελος/πακέτο

Τώρα δείχνει 1 - 1 από 1
Μικρογραφία εικόνας
Ονομα:
Simitos_Master_Thesis.pdf
Μέγεθος:
2.91 MB
Μορφότυπο:
Adobe Portable Document Format
Περιγραφή:
Κατεβάστε

Φάκελος/Πακέτο αδειών

Τώρα δείχνει 1 - 1 από 1
Δεν υπάρχει διαθέσιμη μικρογραφία
Ονομα:
license.txt
Μέγεθος:
2.58 KB
Μορφότυπο:
Item-specific license agreed upon to submission
Περιγραφή:
Κατεβάστε

Συλλογές

Theses and dissertations