Twitter数据抓取的方法(一)

Scraping Tweets Directly from Twitters Search Page – Part 1

Published January 8, 2015

EDIT – Since I wrote this post, Twitter has updated how you get the next list of tweets for your result. Rather than using scroll_cursor, it uses max_position. I’ve written a bit more in detail here.

In fairly recent news,
Twitter has started indexing it’s entire history of Tweets going all
the way back to 2006. Hurrah for data scientists! However, even with
this news (at time of writing), their search API is still restricted to
the past seven days of Tweets. While I doubt this will be the case
permanently, as a useful exercise this post presents how we can search
for Tweets from Twitter without necessarily using their API. Besides the
indexing, there is also the advantage that Twitter is a little more
liberal with rate limits, and you don’t require any authentication keys.

The post will be split up into two parts, this first part looking at
what we can extract from Twitter and how we might start to go about it,
and the second a tutorial on how we can implement this in Java.

Right, to begin, lets say we want to search Twitter for all tweets
related to the query “Babylon 5”. You can access Twitters advanced
search without being logged in: https://twitter.com/search-advanced

If we take a look at the URL that’s constructed when we perform the search we get:

https://twitter.com/search?q=Babylon%205&src=typd

As we can see, there are two query parameters, q (our query encoded) and src (assumed to be the source of the query, i.e. typed).  However, by default, Twitter returns top results, rather than all, so on the displayed page, if you click on All the URL changes to:

https://twitter.com/search?f=realtime&q=Babylon%205&src=typd

The difference here is the f=realtime parameter that appears to specify we receive Tweets in realtime as opposed to a subset of top Tweets. Useful to know, but currently we’re only getting the first 25 Tweets back. If we scroll down though, we notice that more Tweets are loaded on the page via AJAX. Logging all XMLHttpRequests in whatever dev tool you choose to use, we can see that everytime we reach the bottom of the page, Twitter makes an AJAX call a URL similar to:

https://twitter.com/i/search/timeline?f=realtime&q=Babylon%205&src=typd&include_available_features=1&include_entities=1&last_note_ts=85&scroll_cursor=TWEET-553069642609344512-553159310448918528-BD1UO2FFu9QAAAAAAAAETAAAAAcAAAASAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

On further inspection, we see that it is also a JSON response, which is very useful! Before we look at the response though, let’s have a look at that URL and some of it’s parameters.

First off, it’s slightly different to the default search URL. The path is /i/search/timeline as opposed to /search. Secondly, while we notice our familiar parameters q, f, and src, from before, there are several additional ones. The most important and essential new one though is scroll_cursor. This is what Twitter uses to paginate the results. If you remove scroll_cursor from that URL, you end up with your first page of results again.

Now lets take a look now at the JSON response that Twitter provides:

{
has_more_items: boolean,
items_html: "...",
is_scrolling_request: boolean,
is_refresh_request: boolean,
scroll_cursor: "...",
refresh_cursor: "...",
focused_refresh_interval: int
}

Again, not all parameters for this post are important to take note of, but the ones that are include: has_more_items, items_html, and scroll_cursor.

has_more_items – This lets you know with a boolean value whether or not there are any more results after this query.

items_html – This is where all the tweets are which Twitter uses to append to the bottom of their timeline. It requires parsing, but there is a good amount of information in there to be extracted which we will look at in a minute.

scroll_cursor – A pagination value that allows us to extract the next page of results.

Remember our scroll_cursor parameter from earlier on? Well for each search request you make to twitter, the value of this key in the response provides you with the next set of tweets, allowing you to recursively call Twitter until either has_more_items is false, or your previous scroll_cursor equals the last scroll_cursor you had.

Now that we know how to access Twitters own search functionality, lets turn our attention to the tweets themselves. As mentioned before, items_html in the response is where all the tweets are at. However, it comes in a block of HTML as Twitter injects that block at the bottom of the page each time that call is made. The HTML inside is a list of li elements, each element a Tweet. I won’t post the HTML for one here, as even one tweet has a lot of HTML in it, but if you want to look at it, copy the items_html value (omiting the quotes around the HTML content) and paste it into something like JSBeautifier to see the formatted results for yourself.

If we look over the HTML, aside from the tweets text, there is actually a lot of useful information encapsulated in this data packet. The most important item is the Tweet id itself. If you check, it’s actually in the root li element. Now, we could stop here as with that ID, you can query Twitters official API, and if it’s a public Tweet, you can get all kinds of information. However, that’d defeat the purpose of not using the API, so lets see what we can extract from what we already have.

The table below shows various CSS selector queries that you can use to extract the information with.

Embedded Tweet Data
Selector Value
div.original-tweet[data-tweet-id] The authors twitter handle
div.original-tweet[data-name] The name of the author
div.original-tweet[data-user-id] The user ID of the author
span._timestamp[data-time] Timestamp of the post
span._timestamp[data-time-ms] Timestamp of the post in ms
p.tweet-text  Text of Tweet
span.ProfileTweet-action–retweet > span.ProfileTweet-actionCount[data-tweet-stat-count] Number of Retweets
span.ProfileTweet-action–favorite > span.ProfileTweet-actionCount[data-tweet-stat-count]  Number of Favourites

That’s quite a sizeable amount of information in that HTML. From looking through, we can extract a bunch of stuff about the author, the time stamp of the tweet, the text, and number of retweets and favourites.

What have we learned here? Well, to summarize, we know how to construct a Twitter URL query, the response we get from said query, and the information we can extract from said response. The second part of this tutorial (to follow shortly) will introduce some code as to how we can implement the above.

时间: 2024-08-30 00:08:07

Twitter数据抓取的方法(一)的相关文章

Twitter数据抓取的方法(二)

Scraping Tweets Directly from Twitters Search Page – Part 2 Published January 11, 2015 In the previous post we covered effectively the theory of how we can search and extract tweets from Twitter without having to use their API. First, let’s have a qu

Twitter数据抓取的方法(三)

Scraping Tweets Directly from Twitters Search – Update Published August 1, 2015 Sorry for my delayed response to this as I’ve seen several comments on this topic, but I’ve been pretty busy with some other stuff recently, and this is the first chance

Twitter数据抓取

说明:这里分三个系列介绍Twitter数据的非API抓取方法,内容主要来自于Tom Dickinson的博客. Tom Dickinson Milton Keynes,I am currently a PhD student at KMI, currently researching extraction of memorable events from social media. My areas of expertise lie in data mining, machine learnin

Twitter数据非API采集方法

说明:这里分三个系列介绍Twitter数据的非API抓取方法. 在一个老外的博看上看到的,想详细了解的可以自己去看原文. 这种方法可以采集基于关键字在twitter上搜索的结果推文,已经实现自动翻页功能.对于有账号的用户还可以采集起关注的好友列表和关注自己的好友列表. 1.Twitter数据抓取(一) 2.Twitter数据抓取(二) 3.Twitter数据抓取(三)

python爬虫数据抓取

概要:利用python进行web数据抓取简单方法和实现. 1.python进行网页数据抓取有两种方式:一种是直接依据url链接来拼接使用get方法得到内容,一种是构建post请求改变对应参数来获得web返回的内容. 一.第一种方法通常用来获取静态页面内容,比如豆瓣电影内容分类下动画对应的链接: http://www.douban.com/tag/%E5%8A%A8%E7%94%BB/?focus=movie     纪录片对应的链接: http://www.douban.com/tag/%E7%

ngrep环回接口数据抓取方法,使用-d lo参数

ngrep环回接口数据抓取方法,使用-d lo参数,注意顺序: ngrep -W byline -d lo port 80

数据抓取的一个类,包含一些常用的方法

原文:数据抓取的一个类,包含一些常用的方法 using System;using System.Configuration;using System.IO;using System.Net;using System.Text;using System.Text.RegularExpressions; namespace XXX{    /// <summary>    /// Func 的摘要说明.    /// </summary>    public class Func   

浅谈数据抓取的几种方法

在下抓数据也小有研究,现分享几个自己研究出来的抓数据的技术,可能会有很多不足的地方,欢迎大家指正补充哈哈! 方法一:直接抓取网页源码优点:速度快.缺点:1,正由于速度快,易被服务器端检测,可能会限制当前ip的抓取.对于这点,可以尝试使用ip代码解决.   2,如果你要抓取的数据,是在网页加载完后,js修改了网页元素,无法抓取.   3,遇到抓取一些大型网站,如果需要抓取如登录后的页面,可能需要破解服务器端帐号加密算法以及各种加密算法,及其考验技术性.适用场景:网页完全静态化,并且你要抓取的数据在

delphi 用idhttp做web页面数据抓取 注意事项

这里不讨论webbrowse方式了 .直接采用indy的 idhttp  Get post 可以很方便的获取网页数据. 但如果要抓取大量数据 程序稳定运行不崩溃就不那么容易了.这几年也做了不少类似工具 总结了几点 好记性不如烂笔头. 内存泄露 获取页面文本 少不了用到html解析 具体到delphi 估计采用mshtml htmltotext 方法的不少,这个方案再大数据量时就会内存溢出 导致程序崩溃,而这并不是每个程序员都知道.解决的方案:采用自己的html解析类 这里我要感谢 武稀松(csd