Navigation

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.

====== crawlingPeopleRelatedWeiboOriginalPost(pid, person_name) ======
===== Description =====
Extract people related original weibo post from sina weibo and insert to database.
===== Parameters =====
^  Parameter  ^  Necessity  ^  Type  ^  Description  ^
|  pid  |  required  |  int  |  person id  |
|  person_name  |  required  |  string  |  person name need to search  |
===== Output =====
None
===== Implementation =====
  - masterStart()
  - wapLogIn()
  - weiBoWapSearch(person_name, pid)

===== Related Work =====
None
===== Issues About The Crawler =====
  - Sina Weibo API is not not so effective, it need to be authorized but the crawler would not pass sina's examine and verify. 
  - Using browser’s cookies to log in sina account.
  - Using the url weibo.cn instead of www.weibo.com to crawl data, because the latter one’s tweet data is sealed in javascript and it’s difficult to extract.
  - Using multiple proxies to prevent sina block our ip.
  - For speeding up the crawler, using multiple processes and accounts.

Navigation

About Us

Publications

Professional Activities

Lab and Projects

Affiliated Labs

Organizations

Current Activities

See Irwin King In

Book Titles

MISC