python利用beautifulSoup写爬虫

以前讲过利用phantomjs做爬虫抓网页?http://99jty.com/?p=1058 ?是配合选择器做的

利用? beautifulSoup(文档：http://www.crummy.com/software/BeautifulSoup/bs4/doc/)这个python模块，可以很轻松的抓取网页内容

# coding=utf-8import urllibfrom bs4 import BeautifulSoupurl ='http://www.baidu.com/s'values ={'wd':'网球'}encoded_param = urllib.urlencode(values)full_url = url +'?'+ encoded_paramresponse = urllib.urlopen(full_url)soup =BeautifulSoup(response)alinks = soup.find_all('a')

上面可以抓取百度搜出来结果是网球的记录。

beautifulSoup内置了很多非常有用的方法。

几个比较好用的特性：

构造一个node元素

soup = BeautifulSoup('<b class="boldest">Extremely bold</b>')tag = soup.btype(tag)# <class 'bs4.element.Tag'>

属性可以使用attr拿到，结果是字典

tag.attrs# {u'class': u'boldest'}

或者直接tag.class取属性也可。

也可以自由操作属性

tag['class'] = 'verybold'tag['id'] = 1tag# <blockquote class="verybold" id="1">Extremely bold</blockquote>del tag['class']del tag['id']tag# <blockquote>Extremely bold</blockquote>tag['class']# KeyError: 'class'print(tag.get('class'))# None

还可以随便操作，查找dom元素，比如下面的例子

1.构建一份文档

html_doc = """&lt;html&gt;&lt;head&gt;&lt;title&gt;The Dormouse's story&lt;/title&gt;&lt;/head&gt;&lt;p&gt;&lt;b&gt;The Dormouse's story&lt;/b&gt;&lt;/p&gt;&lt;p&gt;Once upon a time there were three little sisters; and their names were&lt;a href="http://example.com/elsie" id="link1"&gt;Elsie&lt;/a&gt;,&lt;a href="http://example.com/lacie" id="link2"&gt;Lacie&lt;/a&gt; and&lt;a href="http://example.com/tillie" id="link3"&gt;Tillie&lt;/a&gt;;and they lived at the bottom of a well.&lt;/p&gt;&lt;p&gt;...&lt;/p&gt;"""from bs4 import BeautifulSoupsoup = BeautifulSoup(html_doc)

2.各种搞

soup.head# <head><title>The Dormouse's story</title></head>soup.title# <title>The Dormouse's story</title>soup.body.b# <b>The Dormouse's story</b>soup.a# <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>soup.find_all('a')# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]head_tag = soup.headhead_tag# <head><title>The Dormouse's story</title></head>head_tag.contents[<title>The Dormouse's story</title>]title_tag = head_tag.contents[0]title_tag# <title>The Dormouse's story</title>title_tag.contents# [u'The Dormouse's story']len(soup.contents)# 1soup.contents[0].name# u'html'text = title_tag.contents[0]text.contentsfor child in title_tag.children:    print(child)head_tag.contents# [<title>The Dormouse's story</title>]for child in head_tag.descendants:    print(child)# <title>The Dormouse's story</title># The Dormouse's storylen(list(soup.children))# 1len(list(soup.descendants))# 25title_tag.string# u'The Dormouse's story'

原文地址：python利用beautifulSoup写爬虫, 感谢原作者分享。生活不要太劳累，弄得自己很疲惫，快乐幸福多体会，

相关文章：

你感兴趣的文章：

标签云：