jacksoncode.github.io/tutorials/python/78.html at master · jacksoncode/jacksoncode.github.io

Repository navigation

History

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

<!DOCTYPE html>

<head>

<title>16.10. urllib - CodeClub</title>

<style>

.tutorial-container { max-width: 960px; margin: 70px auto 40px; padding: 0 20px; }

.tutorial-sidebar { position: fixed; top: 70px; left: 20px; width: 260px; max-height: calc(100vh - 90px); overflow-y: auto; background: var(--bg-secondary, #fff); border-radius: 8px; padding: 16px; box-shadow: 0 2px 8px rgba(0,0,0,0.08); }

.tutorial-sidebar h3 { font-size: 16px; margin-bottom: 12px; color: var(--text-primary, #1a1a2e); }

.tutorial-sidebar a { display: block; padding: 6px 10px; color: var(--text-secondary, #666); text-decoration: none; font-size: 14px; border-radius: 4px; }

.tutorial-sidebar a:hover, .tutorial-sidebar a.active { background: var(--primary, #3b82f6); color: #fff; }

.tutorial-sidebar .sub-link { padding-left: 24px; font-size: 13px; }

.tutorial-content { margin-left: 300px; background: var(--bg-secondary, #fff); border-radius: 8px; padding: 32px; box-shadow: 0 2px 8px rgba(0,0,0,0.08); }

.tutorial-content h1 { font-size: 28px; margin-bottom: 20px; color: var(--text-primary, #1a1a2e); border-bottom: 2px solid var(--primary, #3b82f6); padding-bottom: 12px; }

.tutorial-content h2 { font-size: 22px; margin: 24px 0 16px; color: var(--text-primary, #1a1a2e); }

.tutorial-content h3 { font-size: 18px; margin: 20px 0 12px; }

.tutorial-content p { line-height: 1.8; margin: 12px 0; color: var(--text-secondary, #333); }

.tutorial-content pre { background: #1e1e2e; color: #cdd6f4; padding: 16px; border-radius: 8px; overflow-x: auto; margin: 16px 0; }

.tutorial-content code { font-family: 'Fira Code', 'Cascadia Code', monospace; font-size: 14px; }

.tutorial-content p code { background: #f1f5f9; padding: 2px 6px; border-radius: 4px; color: #e83e8c; }

.tutorial-content blockquote { border-left: 4px solid var(--primary, #3b82f6); padding: 12px 20px; margin: 16px 0; background: #f8fafc; border-radius: 0 8px 8px 0; }

.tutorial-content ul, .tutorial-content ol { padding-left: 24px; margin: 12px 0; }

.tutorial-content li { margin: 6px 0; line-height: 1.7; }

.tutorial-content img { max-width: 100%; border-radius: 8px; margin: 16px 0; }

.tutorial-nav { display: flex; justify-content: space-between; margin-top: 32px; padding-top: 20px; border-top: 1px solid var(--border, #e5e7eb); }

.tutorial-nav a { color: var(--primary, #3b82f6); text-decoration: none; font-weight: 500; }

.tutorial-nav a:hover { text-decoration: underline; }

.tutorial-index { max-width: 960px; margin: 70px auto 40px; padding: 0 20px; }

.tutorial-card { background: var(--bg-secondary, #fff); border-radius: 12px; padding: 24px; margin: 16px 0; box-shadow: 0 2px 8px rgba(0,0,0,0.08); transition: transform 0.2s; }

.tutorial-card:hover { transform: translateY(-2px); }

.tutorial-card h3 { margin: 0 0 8px; color: var(--text-primary, #1a1a2e); }

.tutorial-card p { color: var(--text-secondary, #666); margin: 0 0 12px; }

.tutorial-card .badge { display: inline-block; padding: 4px 10px; border-radius: 20px; font-size: 12px; font-weight: 600; margin-right: 8px; }

.badge-python { background: #3b82f6; color: #fff; }

.badge-git { background: #f97316; color: #fff; }

.badge-ai { background: #8b5cf6; color: #fff; }

@media (max-width: 768px) {

.tutorial-sidebar { display: none; }

.tutorial-content { margin-left: 0; }

}

</style>

</head>

<body>

<span>CodeClub</span>

</a>

</button>

</ul>

</div>

</nav>

<div class="tutorial-container" style="max-width: 960px; margin: 80px auto 40px; padding: 0 20px;"><div class="tutorial-sidebar"><h3>目录</h3><a href="0.html">1. 简介</a><a href="1.html">2. Python历史</a><a href="2.html">3. 安装Python</a><a href="3.html">3.1. Python解释器</a><a href="4.html">4. 第一个Python程序</a><a href="5.html">4.1. 使用文本编辑器</a><a href="6.html">4.2. 输入和输出</a><a href="7.html">5. Python基础</a><a href="8.html">5.1. 数据类型和变量</a><a href="9.html">5.2. 字符串和编码</a><a href="10.html">5.3. 使用list和tuple</a><a href="11.html">5.4. 条件判断</a><a href="12.html">5.5. 模式匹配</a><a href="13.html">5.6. 循环</a><a href="14.html">5.7. 使用dict和set</a><a href="15.html">6. 函数</a><a href="16.html">6.1. 调用函数</a><a href="17.html">6.2. 定义函数</a><a href="18.html">6.3. 函数的参数</a><a href="19.html">6.4. 递归函数</a><a href="20.html">7. 高级特性</a><a href="21.html">7.1. 切片</a><a href="22.html">7.2. 迭代</a><a href="23.html">7.3. 列表生成式</a><a href="24.html">7.4. 生成器</a><a href="25.html">7.5. 迭代器</a><a href="26.html">8. 函数式编程</a><a href="27.html">8.1. 高阶函数</a><a href="28.html">8.1.1. map/reduce</a><a href="29.html">8.1.2. filter</a><a href="30.html">8.1.3. sorted</a><a href="31.html">8.2. 返回函数</a><a href="32.html">8.3. 匿名函数</a><a href="33.html">8.4. 装饰器</a><a href="34.html">8.5. 偏函数</a><a href="35.html">9. 模块</a><a href="36.html">9.1. 使用模块</a><a href="37.html">9.2. 安装第三方模块</a><a href="38.html">10. 面向对象编程</a><a href="39.html">10.1. 类和实例</a><a href="40.html">10.2. 访问限制</a><a href="41.html">10.3. 继承和多态</a><a href="42.html">10.4. 获取对象信息</a><a href="43.html">10.5. 实例属性和类属性</a><a href="44.html">11. 面向对象高级编程</a><a href="45.html">11.1. 使用__slots__</a><a href="46.html">11.2. 使用@property</a><a href="47.html">11.3. 多重继承</a><a href="48.html">11.4. 定制类</a><a href="49.html">11.5. 使用枚举类</a><a href="50.html">11.6. 使用元类</a><a href="51.html">12. 错误、调试和测试</a><a href="52.html">12.1. 错误处理</a><a href="53.html">12.2. 调试</a><a href="54.html">12.3. 单元测试</a><a href="55.html">12.4. 文档测试</a><a href="56.html">13. IO编程</a><a href="57.html">13.1. 文件读写</a><a href="58.html">13.2. StringIO和BytesIO</a><a href="59.html">13.3. 操作文件和目录</a><a href="60.html">13.4. 序列化</a><a href="61.html">14. 进程和线程</a><a href="62.html">14.1. 多进程</a><a href="63.html">14.2. 多线程</a><a href="64.html">14.3. ThreadLocal</a><a href="65.html">14.4. 进程 vs. 线程</a><a href="66.html">14.5. 分布式进程</a><a href="67.html">15. 正则表达式</a><a href="68.html">16. 常用内建模块</a><a href="69.html">16.1. datetime</a><a href="70.html">16.2. collections</a><a href="71.html">16.3. argparse</a><a href="72.html">16.4. base64</a><a href="73.html">16.5. struct</a><a href="74.html">16.6. hashlib</a><a href="75.html">16.7. hmac</a><a href="76.html">16.8. itertools</a><a href="77.html">16.9. contextlib</a><a href="78.html">16.10. urllib</a><a href="79.html">16.11. XML</a><a href="80.html">16.12. HTMLParser</a><a href="81.html">16.13. venv</a><a href="82.html">17. 常用第三方模块</a><a href="83.html">17.1. Pillow</a><a href="84.html">17.2. requests</a><a href="85.html">17.3. chardet</a><a href="86.html">17.4. psutil</a><a href="87.html">18. 图形界面</a><a href="88.html">18.1. 海龟绘图</a><a href="89.html">19. 网络编程</a><a href="90.html">19.1. TCP/IP简介</a><a href="91.html">19.2. TCP编程</a><a href="92.html">19.3. UDP编程</a><a href="93.html">20. 电子邮件</a><a href="94.html">20.1. SMTP发送邮件</a><a href="95.html">20.2. POP3收取邮件</a><a href="96.html">21. 访问数据库</a><a href="97.html">21.1. 使用SQLite</a><a href="98.html">21.2. 使用MySQL</a><a href="99.html">21.3. 使用SQLAlchemy</a><a href="100.html">22. Web开发</a><a href="101.html">22.1. HTTP协议简介</a><a href="102.html">22.2. HTML简介</a><a href="103.html">22.3. WSGI接口</a><a href="104.html">22.4. 使用Web框架</a><a href="105.html">22.5. 使用模板</a><a href="106.html">23. 异步IO</a><a href="107.html">23.1. 协程</a><a href="108.html">23.2. 使用asyncio</a><a href="109.html">23.3. 使用aiohttp</a><a href="110.html">24. FAQ</a><a href="111.html">25. 期末总结</a></div><div class="tutorial-content"><h1>16.10. urllib</h1><div class="tutorial-body"><p><code>urllib</code>的<code>request</code>模块可以非常方便地抓取URL内容，也就是发送一个GET请求到指定的页面，然后返回HTTP的响应：</p>

<p>例如，对豆瓣的一个URL<code>https://api.douban.com/v2/book/2129650</code>进行抓取，并返回响应：</p>

<pre class="hljs"><code class="language-python"><span class="hljs-keyword">from</span> urllib <span class="hljs-keyword">import</span> request

<span class="hljs-keyword">with</span> request.urlopen(<span class="hljs-string">'https://api.douban.com/v2/book/2129650'</span>) <span class="hljs-keyword">as</span> f:

data = f.read()

<span class="hljs-built_in">print</span>(<span class="hljs-string">'Status:'</span>, f.status, f.reason)

<span class="hljs-keyword">for</span> k, v <span class="hljs-keyword">in</span> f.getheaders():

<span class="hljs-built_in">print</span>(<span class="hljs-string">'%s: %s'</span> % (k, v))

<span class="hljs-built_in">print</span>(<span class="hljs-string">'Data:'</span>, data.decode(<span class="hljs-string">'utf-8'</span>))

</code></pre>

<p>可以看到HTTP响应的头和JSON数据：</p>

<pre class="hljs"><code class="language-plain">Status: 200 OK

Server: nginx

Date: Tue, 26 May 2015 10:02:27 GMT

Content-Type: application/json; charset=utf-8

Content-Length: 2049

Connection: close

Expires: Sun, 1 Jan 2006 01:00:00 GMT

Pragma: no-cache

Cache-Control: must-revalidate, no-cache, private

X-DAE-Node: pidl1

Data: {"rating":{"max":10,"numRaters":16,"average":"7.4","min":0},"subtitle":"","author":["廖雪峰编著"],"pubdate":"2007-6",...}

</code></pre>

<p>如果我们要想模拟浏览器发送GET请求，就需要使用<code>Request</code>对象，通过往<code>Request</code>对象添加HTTP头，我们就可以把请求伪装成浏览器。例如，模拟iPhone去请求豆瓣首页：</p>

<pre class="hljs"><code class="language-python"><span class="hljs-keyword">from</span> urllib <span class="hljs-keyword">import</span> request

req = request.Request(<span class="hljs-string">'http://www.douban.com/'</span>)

req.add_header(<span class="hljs-string">'User-Agent'</span>, <span class="hljs-string">'Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25'</span>)

<span class="hljs-keyword">with</span> request.urlopen(req) <span class="hljs-keyword">as</span> f:

<span class="hljs-built_in">print</span>(<span class="hljs-string">'Status:'</span>, f.status, f.reason)

<span class="hljs-keyword">for</span> k, v <span class="hljs-keyword">in</span> f.getheaders():

<span class="hljs-built_in">print</span>(<span class="hljs-string">'%s: %s'</span> % (k, v))

<span class="hljs-built_in">print</span>(<span class="hljs-string">'Data:'</span>, f.read().decode(<span class="hljs-string">'utf-8'</span>))

</code></pre>

<p>这样豆瓣会返回适合iPhone的移动版网页：</p>

<pre class="hljs"><code class="language-plain">...

...

</code></pre>

<p>如果要以POST发送一个请求，只需要把参数<code>data</code>以bytes形式传入。</p>

<p>我们模拟一个微博登录，先读取登录的邮箱和口令，然后按照weibo.cn的登录页的格式以<code>username=xxx&password=xxx</code>的编码传入：</p>

<pre class="hljs"><code class="language-python"><span class="hljs-keyword">from</span> urllib <span class="hljs-keyword">import</span> request, parse

<span class="hljs-built_in">print</span>(<span class="hljs-string">'Login to weibo.cn...'</span>)

email = <span class="hljs-built_in">input</span>(<span class="hljs-string">'Email: '</span>)

passwd = <span class="hljs-built_in">input</span>(<span class="hljs-string">'Password: '</span>)

login_data = parse.urlencode([

(<span class="hljs-string">'username'</span>, email),

(<span class="hljs-string">'password'</span>, passwd),

(<span class="hljs-string">'entry'</span>, <span class="hljs-string">'mweibo'</span>),

(<span class="hljs-string">'client_id'</span>, <span class="hljs-string">''</span>),

(<span class="hljs-string">'savestate'</span>, <span class="hljs-string">'1'</span>),

(<span class="hljs-string">'ec'</span>, <span class="hljs-string">''</span>),

(<span class="hljs-string">'pagerefer'</span>, <span class="hljs-string">'https://passport.weibo.cn/signin/welcome?entry=mweibo&r=http%3A%2F%2Fm.weibo.cn%2F'</span>)

])

req = request.Request(<span class="hljs-string">'https://passport.weibo.cn/sso/login'</span>)

req.add_header(<span class="hljs-string">'Origin'</span>, <span class="hljs-string">'https://passport.weibo.cn'</span>)

req.add_header(<span class="hljs-string">'Referer'</span>, <span class="hljs-string">'https://passport.weibo.cn/signin/login?entry=mweibo&res=wel&wm=3349&r=http%3A%2F%2Fm.weibo.cn%2F'</span>)

<span class="hljs-keyword">with</span> request.urlopen(req, data=login_data.encode(<span class="hljs-string">'utf-8'</span>)) <span class="hljs-keyword">as</span> f:

<span class="hljs-built_in">print</span>(<span class="hljs-string">'Status:'</span>, f.status, f.reason)

<span class="hljs-keyword">for</span> k, v <span class="hljs-keyword">in</span> f.getheaders():

<span class="hljs-built_in">print</span>(<span class="hljs-string">'%s: %s'</span> % (k, v))

<span class="hljs-built_in">print</span>(<span class="hljs-string">'Data:'</span>, f.read().decode(<span class="hljs-string">'utf-8'</span>))

</code></pre>

<p>如果登录成功，我们获得的响应如下：</p>

<pre class="hljs"><code class="language-plain">Status: 200 OK

Server: nginx/1.2.0

...

Set-Cookie: SSOLoginState=1432620126; path=/; domain=weibo.cn

...

Data: {"retcode":20000000,"msg":"","data":{...,"uid":"1658384301"}}

</code></pre>

<p>如果登录失败，我们获得的响应如下：</p>

<pre class="hljs"><code class="language-plain">...

Data: {"retcode":50011015,"msg":"\u7528\u6237\u540d\u6216\u5bc6\u7801\u9519\u8bef","data":{"username":"example@python.org","errline":536}}

</code></pre>

<h3>Handler</h3>

<p>如果还需要更复杂的控制，比如通过一个Proxy去访问网站，我们需要利用<code>ProxyHandler</code>来处理，示例代码如下：</p>

<pre class="hljs"><code class="language-python">proxy_handler = urllib.request.ProxyHandler({<span class="hljs-string">'http'</span>: <span class="hljs-string">'http://www.example.com:3128/'</span>})

proxy_auth_handler = urllib.request.ProxyBasicAuthHandler()

proxy_auth_handler.add_password(<span class="hljs-string">'realm'</span>, <span class="hljs-string">'host'</span>, <span class="hljs-string">'username'</span>, <span class="hljs-string">'password'</span>)

opener = urllib.request.build_opener(proxy_handler, proxy_auth_handler)

<span class="hljs-keyword">with</span> opener.<span class="hljs-built_in">open</span>(<span class="hljs-string">'http://www.example.com/login.html'</span>) <span class="hljs-keyword">as</span> f:

</code></pre>

<p><code>urllib</code>提供的功能就是利用程序去执行各种HTTP请求。如果要模拟浏览器完成特定功能，需要把请求伪装成浏览器。伪装的方法是先监控浏览器发出的请求，再根据浏览器的请求头来伪装，<code>User-Agent</code>头就是用来标识浏览器的。</p>

<p>利用<code>urllib</code>读取JSON，然后将JSON解析为Python对象：</p>

<pre class="hljs"><code class="language-python"><span class="hljs-keyword">from</span> urllib <span class="hljs-keyword">import</span> request

<span class="hljs-keyword">def</span> <span class="hljs-title function_">fetch_data</span>(<span class="hljs-params">url</span>):

<span class="hljs-keyword">return</span> <span class="hljs-string">''</span>

URL = <span class="hljs-string">'https://api.weatherapi.com/v1/current.json?key=b4e8f86b44654e6b86885330242207&q=Beijing&aqi=no'</span>

data = fetch_data(URL)

<span class="hljs-built_in">print</span>(data)

<span class="hljs-keyword">assert</span> data[<span class="hljs-string">'location'</span>][<span class="hljs-string">'name'</span>] == <span class="hljs-string">'Beijing'</span>

<span class="hljs-built_in">print</span>(<span class="hljs-string">'ok'</span>)

</code></pre>

<p><a href="use_urllib.py">use_urllib.py</a></p>

</div>

<a href="/books/python/built-in-modules/contextlib/index.html"

onclick="gs_load_chapter('/books/python/built-in-modules/contextlib/index.html',true);return false;"

class="inline-block my-2">

stroke="currentColor">

<path stroke-linecap="round" stroke-linejoin="round"

d="M18.75 19.5l-7.5-7.5 7.5-7.5m-6 15L5.25 12l7.5-7.5" />

</svg>

contextlib

</a>

</div>

<a href="/books/python/built-in-modules/xml/index.html"

onclick="gs_load_chapter('/books/python/built-in-modules/xml/index.html',true);return false;"

class="inline-block my-2">

XML

stroke="currentColor">

<path stroke-linecap="round" stroke-linejoin="round"

d="M11.25 4.5l7.5 7.5-7.5 7.5m-6-15l7.5 7.5-7.5 7.5" />

</svg>

</a></div><div style="display: flex; justify-content: space-between; margin-top: 32px; padding-top: 20px; border-top: 1px solid var(--border, #e5e7eb);"><a href="77.html" style="color: var(--primary, #3b82f6); text-decoration: none; font-weight: 500;">← 16.9. contextlib</a><a href="79.html" style="color: var(--primary, #3b82f6); text-decoration: none; font-weight: 500;">16.11. XML →</a></div><p style="margin-top: 16px; font-size: 13px; color: var(--text-secondary, #999);">原文：<a href="https://liaoxuefeng.com/books/python/built-in-modules/urllib/index.html" style="color: var(--primary, #3b82f6);">廖雪峰 - 16.10. urllib</a></p></div></div>

</div>

</footer>

</body>

</html>

FilesExpand file tree

78.html

Latest commit

History

78.html

File metadata and controls