Table of Contents
crawlingCompanyRelatedWeiboOriginalPost(stock_id, company_name)
Description
Extract company related original weibo post from sina weibo and insert to database.
Parameters
| Parameter | Necessity | Type | Description |
|---|---|---|---|
| stock_id | required | int | company id |
| company_name | required | string | company name need to crawl |
Output
| Parameters | Type | Description |
|---|---|---|
| status | string | show the crawler running status |
Implementation
- masterStart(). Create multiple processes to begin crawling data.
- wapLogIn(). Log in sina Account.
- weiBoWapSearch(company_name, stock_id). Use company name and company id to search company related weibi and insert in database.
- extractTopic(company_name, company_id). Extract weibo text and insert to database.
Related Work
None
Issues About The Crawler
- Sina Weibo API is not not so effective, it need to be authorized but the crawler would not pass sina's examine and verify.
- Using browser’s cookies to log in sina account.
- Using the url weibo.cn instead of www.weibo.com to crawl data, because the latter one’s tweet data is sealed in javascript and it’s difficult to extract.
- Using multiple proxies to prevent sina block our ip.
- For speeding up the crawler, using multiple processes and accounts.
