====== crawlingCompanyRelatedWeiboOriginalPost(stock_id, company_name) ====== ===== Description ===== Extract company related original weibo post from sina weibo and insert to database. ===== Parameters ===== ^ Parameter ^ Necessity ^ Type ^ Description ^ | stock_id | required | int | company id | | company_name | required | string | company name need to crawl | ===== Output ===== ^ Parameters ^ Type ^ Description ^ | status | string | show the crawler running status | ===== Implementation ===== - masterStart(). Create multiple processes to begin crawling data. - wapLogIn(). Log in sina Account. - weiBoWapSearch(company_name, stock_id). Use company name and company id to search company related weibi and insert in database. * extractTopic(company_name, company_id). Extract weibo text and insert to database. ===== Related Work ===== None ===== Issues About The Crawler ===== - Sina Weibo API is not not so effective, it need to be authorized but the crawler would not pass sina's examine and verify. - Using browser’s cookies to log in sina account. - Using the url weibo.cn instead of www.weibo.com to crawl data, because the latter one’s tweet data is sealed in javascript and it’s difficult to extract. - Using multiple proxies to prevent sina block our ip. - For speeding up the crawler, using multiple processes and accounts.