BS4

BeautifulSoup是用来从HTML or XML中提取数据的Python lib。BeautifulSoup将文档转化为树形结构（DOM），每个节点都是下述四种类型的Python对象：

BeautifulSoup <class 'bs4.BeatifulSoup'>
Tag <class 'bs4.element.Tag'>
NavigableString <class 'bs4.element.NavigableString'>
Comment <class

从集合角度理解以上4中类的关系（类概念上并不准确）

BeautifulSoup 为全集（将Document以入参传入生成BeautifulSoup object），包含 Tag子集
Tag 包含 NavigableString 子集
Comment 为 NavigableString 特殊集合

Usage

BeautifulSoup的第一个入参是Document，第二个入参指定Document parser 类型.

from bs4 import BeautifulSoup
import requests, re

url = 
# get whole HTTP response
response = requests.get(url)
# args[0] is HTML document, args[1] select LXML parser. returned BeautifulSoup object
soup = BeautifulSoup( response.text, 'lxml')
print soup.name
# [document]'
print type(soup)
# <class 'bs4.BeatifulSoup'>

Sample codes for Tag objects

# BeutifulSoup --> Tag 
# get the Tag object(title)
res = soup.title
print res
# <title>KDS Life</title>

res = soup.title.name
print res
# title

# attribules of a Tag object
res = soup.section
print type(res)
# <class 'bs4.element.Tag'>

print res['class']
# ['forum-head-hot', 'clearfix']

# All the attributes of section Tag object, returned a dict
print res.attrs
#{'class': ['forum-head-hot', 'clearfix']}

Sample codes for NavigableString object

# NavigableString object describes the string in Tag object
res = soup.title
print res.string
# KDS Life
print type(res.string)
# <class 'bs4.element.NavigableString'>

Sample codes for Comment object

# Comment, is a special NavigableString object
markup = "<b><!--Hey, buddy. Want to buy a used parser?--></b>"
soup = BeautifulSoup(markup)
comment = soup.b.string
print type(comment)
# <class

BS4 Parser

按照优先顺序自动解析，'lxml' --> 'html5lib' --> 'html.parser'

常用Tag对象方法

find_all()

find_all(name,attrs,recursive,text,**kwargs) 不解释，直接看代码

# filter, returned a matching list
# returned [] if matching nothing
title = soup.find_all('title')
print title
#[<title>Google</title>]

res = soup.find_all('div', 'topAd')
print res

# find all the elements whose id is 'gb-main'
res = soup.find_all(id='topAd')
print res
#[<div id="topAd">...</div>]

# find all the elements with 'img' tag and 'src' attribute matching the specific pattern
res = soup.find_all('img', 
print res
# 
#...]

select()

# css selector
# select those whose tag's id = wrapperto
res = soup.select('#wrapperto')
print res
# [<div class="swiper-wrapper clearfix" id="wrapperto"></div>]

# select those 'img' tags who have 'src' attribute
res = soup.select('img[src]')
print res
 <im
#g 

# select those 'img' tags whose 'src' attribute is 
res = 
print res
#

Other

# get_text()
markup = '<a  a link to 
soup = BeautifulSoup(markup,'lxml')
res = soup.get_text()
print res
#  a link to 

res = soup.i.get_text()
print res
# 

# .stripped_string
res = soup.stripped_strings
print list(res)
# [u'a link to',

最后贴上一个简单的KDS图片爬虫

Note

BeautifulSoup进行了编码检测并自动转为Unicode. soup.original_encoding属性来获取自动识别编码的结果。
Input converts to unicode, output encodes with utf-8
在BS使用中，可配合 XPath expression使用

[Python]BeautifulSoup 4 notes