jacksoncode.github.io/tutorials/python/85.html at master · jacksoncode/jacksoncode.github.io

Repository navigation

History

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

<!DOCTYPE html>

<head>

<title>17.3. chardet - CodeClub</title>

<style>

.tutorial-container { max-width: 960px; margin: 70px auto 40px; padding: 0 20px; }

.tutorial-sidebar { position: fixed; top: 70px; left: 20px; width: 260px; max-height: calc(100vh - 90px); overflow-y: auto; background: var(--bg-secondary, #fff); border-radius: 8px; padding: 16px; box-shadow: 0 2px 8px rgba(0,0,0,0.08); }

.tutorial-sidebar h3 { font-size: 16px; margin-bottom: 12px; color: var(--text-primary, #1a1a2e); }

.tutorial-sidebar a { display: block; padding: 6px 10px; color: var(--text-secondary, #666); text-decoration: none; font-size: 14px; border-radius: 4px; }

.tutorial-sidebar a:hover, .tutorial-sidebar a.active { background: var(--primary, #3b82f6); color: #fff; }

.tutorial-sidebar .sub-link { padding-left: 24px; font-size: 13px; }

.tutorial-content { margin-left: 300px; background: var(--bg-secondary, #fff); border-radius: 8px; padding: 32px; box-shadow: 0 2px 8px rgba(0,0,0,0.08); }

.tutorial-content h1 { font-size: 28px; margin-bottom: 20px; color: var(--text-primary, #1a1a2e); border-bottom: 2px solid var(--primary, #3b82f6); padding-bottom: 12px; }

.tutorial-content h2 { font-size: 22px; margin: 24px 0 16px; color: var(--text-primary, #1a1a2e); }

.tutorial-content h3 { font-size: 18px; margin: 20px 0 12px; }

.tutorial-content p { line-height: 1.8; margin: 12px 0; color: var(--text-secondary, #333); }

.tutorial-content pre { background: #1e1e2e; color: #cdd6f4; padding: 16px; border-radius: 8px; overflow-x: auto; margin: 16px 0; }

.tutorial-content code { font-family: 'Fira Code', 'Cascadia Code', monospace; font-size: 14px; }

.tutorial-content p code { background: #f1f5f9; padding: 2px 6px; border-radius: 4px; color: #e83e8c; }

.tutorial-content blockquote { border-left: 4px solid var(--primary, #3b82f6); padding: 12px 20px; margin: 16px 0; background: #f8fafc; border-radius: 0 8px 8px 0; }

.tutorial-content ul, .tutorial-content ol { padding-left: 24px; margin: 12px 0; }

.tutorial-content li { margin: 6px 0; line-height: 1.7; }

.tutorial-content img { max-width: 100%; border-radius: 8px; margin: 16px 0; }

.tutorial-nav { display: flex; justify-content: space-between; margin-top: 32px; padding-top: 20px; border-top: 1px solid var(--border, #e5e7eb); }

.tutorial-nav a { color: var(--primary, #3b82f6); text-decoration: none; font-weight: 500; }

.tutorial-nav a:hover { text-decoration: underline; }

.tutorial-index { max-width: 960px; margin: 70px auto 40px; padding: 0 20px; }

.tutorial-card { background: var(--bg-secondary, #fff); border-radius: 12px; padding: 24px; margin: 16px 0; box-shadow: 0 2px 8px rgba(0,0,0,0.08); transition: transform 0.2s; }

.tutorial-card:hover { transform: translateY(-2px); }

.tutorial-card h3 { margin: 0 0 8px; color: var(--text-primary, #1a1a2e); }

.tutorial-card p { color: var(--text-secondary, #666); margin: 0 0 12px; }

.tutorial-card .badge { display: inline-block; padding: 4px 10px; border-radius: 20px; font-size: 12px; font-weight: 600; margin-right: 8px; }

.badge-python { background: #3b82f6; color: #fff; }

.badge-git { background: #f97316; color: #fff; }

.badge-ai { background: #8b5cf6; color: #fff; }

@media (max-width: 768px) {

.tutorial-sidebar { display: none; }

.tutorial-content { margin-left: 0; }

}

</style>

</head>

<body>

<span>CodeClub</span>

</a>

</button>

</ul>

</div>

</nav>

<div class="tutorial-container" style="max-width: 960px; margin: 80px auto 40px; padding: 0 20px;"><div class="tutorial-sidebar"><h3>目录</h3><a href="0.html">1. 简介</a><a href="1.html">2. Python历史</a><a href="2.html">3. 安装Python</a><a href="3.html">3.1. Python解释器</a><a href="4.html">4. 第一个Python程序</a><a href="5.html">4.1. 使用文本编辑器</a><a href="6.html">4.2. 输入和输出</a><a href="7.html">5. Python基础</a><a href="8.html">5.1. 数据类型和变量</a><a href="9.html">5.2. 字符串和编码</a><a href="10.html">5.3. 使用list和tuple</a><a href="11.html">5.4. 条件判断</a><a href="12.html">5.5. 模式匹配</a><a href="13.html">5.6. 循环</a><a href="14.html">5.7. 使用dict和set</a><a href="15.html">6. 函数</a><a href="16.html">6.1. 调用函数</a><a href="17.html">6.2. 定义函数</a><a href="18.html">6.3. 函数的参数</a><a href="19.html">6.4. 递归函数</a><a href="20.html">7. 高级特性</a><a href="21.html">7.1. 切片</a><a href="22.html">7.2. 迭代</a><a href="23.html">7.3. 列表生成式</a><a href="24.html">7.4. 生成器</a><a href="25.html">7.5. 迭代器</a><a href="26.html">8. 函数式编程</a><a href="27.html">8.1. 高阶函数</a><a href="28.html">8.1.1. map/reduce</a><a href="29.html">8.1.2. filter</a><a href="30.html">8.1.3. sorted</a><a href="31.html">8.2. 返回函数</a><a href="32.html">8.3. 匿名函数</a><a href="33.html">8.4. 装饰器</a><a href="34.html">8.5. 偏函数</a><a href="35.html">9. 模块</a><a href="36.html">9.1. 使用模块</a><a href="37.html">9.2. 安装第三方模块</a><a href="38.html">10. 面向对象编程</a><a href="39.html">10.1. 类和实例</a><a href="40.html">10.2. 访问限制</a><a href="41.html">10.3. 继承和多态</a><a href="42.html">10.4. 获取对象信息</a><a href="43.html">10.5. 实例属性和类属性</a><a href="44.html">11. 面向对象高级编程</a><a href="45.html">11.1. 使用__slots__</a><a href="46.html">11.2. 使用@property</a><a href="47.html">11.3. 多重继承</a><a href="48.html">11.4. 定制类</a><a href="49.html">11.5. 使用枚举类</a><a href="50.html">11.6. 使用元类</a><a href="51.html">12. 错误、调试和测试</a><a href="52.html">12.1. 错误处理</a><a href="53.html">12.2. 调试</a><a href="54.html">12.3. 单元测试</a><a href="55.html">12.4. 文档测试</a><a href="56.html">13. IO编程</a><a href="57.html">13.1. 文件读写</a><a href="58.html">13.2. StringIO和BytesIO</a><a href="59.html">13.3. 操作文件和目录</a><a href="60.html">13.4. 序列化</a><a href="61.html">14. 进程和线程</a><a href="62.html">14.1. 多进程</a><a href="63.html">14.2. 多线程</a><a href="64.html">14.3. ThreadLocal</a><a href="65.html">14.4. 进程 vs. 线程</a><a href="66.html">14.5. 分布式进程</a><a href="67.html">15. 正则表达式</a><a href="68.html">16. 常用内建模块</a><a href="69.html">16.1. datetime</a><a href="70.html">16.2. collections</a><a href="71.html">16.3. argparse</a><a href="72.html">16.4. base64</a><a href="73.html">16.5. struct</a><a href="74.html">16.6. hashlib</a><a href="75.html">16.7. hmac</a><a href="76.html">16.8. itertools</a><a href="77.html">16.9. contextlib</a><a href="78.html">16.10. urllib</a><a href="79.html">16.11. XML</a><a href="80.html">16.12. HTMLParser</a><a href="81.html">16.13. venv</a><a href="82.html">17. 常用第三方模块</a><a href="83.html">17.1. Pillow</a><a href="84.html">17.2. requests</a><a href="85.html">17.3. chardet</a><a href="86.html">17.4. psutil</a><a href="87.html">18. 图形界面</a><a href="88.html">18.1. 海龟绘图</a><a href="89.html">19. 网络编程</a><a href="90.html">19.1. TCP/IP简介</a><a href="91.html">19.2. TCP编程</a><a href="92.html">19.3. UDP编程</a><a href="93.html">20. 电子邮件</a><a href="94.html">20.1. SMTP发送邮件</a><a href="95.html">20.2. POP3收取邮件</a><a href="96.html">21. 访问数据库</a><a href="97.html">21.1. 使用SQLite</a><a href="98.html">21.2. 使用MySQL</a><a href="99.html">21.3. 使用SQLAlchemy</a><a href="100.html">22. Web开发</a><a href="101.html">22.1. HTTP协议简介</a><a href="102.html">22.2. HTML简介</a><a href="103.html">22.3. WSGI接口</a><a href="104.html">22.4. 使用Web框架</a><a href="105.html">22.5. 使用模板</a><a href="106.html">23. 异步IO</a><a href="107.html">23.1. 协程</a><a href="108.html">23.2. 使用asyncio</a><a href="109.html">23.3. 使用aiohttp</a><a href="110.html">24. FAQ</a><a href="111.html">25. 期末总结</a></div><div class="tutorial-content"><h1>17.3. chardet</h1><div class="tutorial-body"><p>对于未知编码的<code>bytes</code>，要把它转换成<code>str</code>，需要先“猜测”编码。猜测的方式是先收集各种编码的特征字符，根据特征字符判断，就能有很大概率“猜对”。</p>

<p>当然，我们肯定不能从头自己写这个检测编码的功能，这样做费时费力。<code>chardet</code>这个第三方库正好就派上了用场。用它来检测编码，简单易用。</p>

<h3>安装chardet</h3>

<p>如果安装了Anaconda，<code>chardet</code>就已经可用了。否则，需要在命令行下通过pip安装：</p>

<pre class="hljs"><code class="language-plain">$ pip install chardet

</code></pre>

<p>如果遇到Permission denied安装失败，请加上sudo重试。</p>

<h3>使用chardet</h3>

<p>当我们拿到一个<code>bytes</code>时，就可以对其检测编码。用<code>chardet</code>检测编码，只需要一行代码：</p>

<pre class="hljs"><code class="language-plain">>>> chardet.detect(b'Hello, world!')

{'encoding': 'ascii', 'confidence': 1.0, 'language': ''}

</code></pre>

<p>检测出的编码是<code>ascii</code>，注意到还有个<code>confidence</code>字段，表示检测的概率是1.0（即100%）。</p>

<p>我们来试试检测GBK编码的中文：</p>

<pre class="hljs"><code class="language-plain">>>> data = '离离原上草，一岁一枯荣'.encode('gbk')

>>> chardet.detect(data)

{'encoding': 'GB2312', 'confidence': 0.7407407407407407, 'language': 'Chinese'}

</code></pre>

<p>检测的编码是<code>GB2312</code>，注意到GBK是GB2312的超集，两者是同一种编码，检测正确的概率是74%，<code>language</code>字段指出的语言是<code>'Chinese'</code>。</p>

<p>对UTF-8编码进行检测：</p>

<pre class="hljs"><code class="language-plain">>>> data = '离离原上草，一岁一枯荣'.encode('utf-8')

>>> chardet.detect(data)

{'encoding': 'utf-8', 'confidence': 0.99, 'language': ''}

</code></pre>

<p>我们再试试对日文进行检测：</p>

<pre class="hljs"><code class="language-plain">>>> data = '最新の主要ニュース'.encode('euc-jp')

>>> chardet.detect(data)

{'encoding': 'EUC-JP', 'confidence': 0.99, 'language': 'Japanese'}

</code></pre>

<p>可见，用<code>chardet</code>检测编码，使用简单。获取到编码后，再转换为<code>str</code>，就可以方便后续处理。</p>

<p><code>chardet</code>支持检测的编码列表请参考官方文档<a href="https://chardet.readthedocs.io/en/latest/supported-encodings.html" target="_blank">Supported encodings</a>。</p>

<p>使用<code>chardet</code>检测编码非常容易，<code>chardet</code>支持检测中文、日文、韩文等多种语言。</p>

</div>

<a href="/books/python/third-party-modules/requests/index.html"

onclick="gs_load_chapter('/books/python/third-party-modules/requests/index.html',true);return false;"

class="inline-block my-2">

stroke="currentColor">

<path stroke-linecap="round" stroke-linejoin="round"

d="M18.75 19.5l-7.5-7.5 7.5-7.5m-6 15L5.25 12l7.5-7.5" />

</svg>

requests

</a>

</div>

<a href="/books/python/third-party-modules/psutil/index.html"

onclick="gs_load_chapter('/books/python/third-party-modules/psutil/index.html',true);return false;"

class="inline-block my-2">

psutil

stroke="currentColor">

<path stroke-linecap="round" stroke-linejoin="round"

d="M11.25 4.5l7.5 7.5-7.5 7.5m-6-15l7.5 7.5-7.5 7.5" />

</svg>

</a></div><div style="display: flex; justify-content: space-between; margin-top: 32px; padding-top: 20px; border-top: 1px solid var(--border, #e5e7eb);"><a href="84.html" style="color: var(--primary, #3b82f6); text-decoration: none; font-weight: 500;">← 17.2. requests</a><a href="86.html" style="color: var(--primary, #3b82f6); text-decoration: none; font-weight: 500;">17.4. psutil →</a></div><p style="margin-top: 16px; font-size: 13px; color: var(--text-secondary, #999);">原文：<a href="https://liaoxuefeng.com/books/python/third-party-modules/chardet/index.html" style="color: var(--primary, #3b82f6);">廖雪峰 - 17.3. chardet</a></p></div></div>

</div>

</footer>

</body>

</html>

FilesExpand file tree

85.html

Latest commit

History

85.html

File metadata and controls