[转]认识python的多线程[实例]
我们在做软件开发的时候很多要用到多线程技术。例如如果做一个下载软件像flashget就要用到、像在线视频工具realplayer也要用到因为要同时下载media stream还要播放。其实例子是很多的。
线程相对进程来说是“轻量级”的，操作系统用较少的资源创建和管理线程。程序中的线程在相同的内存空间中执行，并共享许多相同的资源。
1) 在python中如何创建一个线程对象？
如果你要创建一个线程对象，很简单，只要你的类继承threading.Thread，然后在__init__里首先调用threading.Thread的__init__方法即可：
import threading
class MyThread(threading.Thread):
def __init__(self,threadname):
threading.Thread.__init__(self,name=threadname)
这才仅仅是个空线程，我可不是要他拉空车的，他可得给我干点实在活。很简单，重写类的run()方法即可，把你要在线程执行时做的事情都放到里面
import threading,time
class MyThread(threading.Thread):
def __init__(self,threadname):
threading.Thread.__init__(self,name=threadname)
def run(self):
for i in xrange(10):
print self.getName,i
time.sleep(1)
以上代码如果被执行之后会每隔1秒输出一次信息到屏幕，10次后结束。其中这里getName()是threading.Thread类的一个方法，用来获得这个线程对象的name。还有一个方法setName()当然就是来设置这个线程对象的name的了。
如果要创建一个线程，首先就要先创建一个线程对象
my = MyThread('test')
一个线程对象被创建后，他就处于“born”（诞生状态），如何让这个线程对象开始运行呢?只要调用线程对象的start()方法即可
import threading,time
class MyThread(threading.Thread):
def __init__(self,threadname):
threading.Thread.__init__(self,name=threadname)
def run(self):
for i in xrange(10):
print self.getName(),i
time.sleep(1)
my = MyThread('test')
my.start()
现在线程就处于“ready”状态或者也称为“runnable”状态。
奇
怪吗？不是已经start了吗？为什么不称为“running”状态呢？其实是有原因的。因为我们的计算机一般是不具有真正并行处理能力的。我们所谓的多
线程只是把时间分成片段，然后隔一个时间段就让一个线程执行一下，然后进入“sleeping
”状态，然后唤醒另一个在“sleeping”的线程，如此循环runnable->sleeping->runnable...
，只是因为计算机执行速度很快，而时间片段间隔很小，我们感受不到，以为是同时进行的。所以说一个线程在start了之后只是处在了可以运行的状态，他什
么时候运行还是由系统来进行调度的。
那一个线程什么时候会“dead”呢？一般来说当线程对象的run方法执行结束或者在执行中抛出异常的话，那么这个线程就会结束了。系统会自动对“dead”状态线程进行清理。
如果一个线程A在执行的过程中需要等待另一个线程tB执行结束后才能运行的话，那就可以在A在调用B的B.join()方法，另外还可以给join()传入等待的时间。
线程对象的setDaemon()方法可以让子线程随着主线程的退出而结束，不过注意的是setDaemon()方法必须在线程对象没有调用start()方法之前调用(默认情况下；在python中，主线程结束后，会默认等待子线程结束后，主线程才退出)。
t1 = MyThread('t1')
print t1.getName(),t1.isDaemon()
t1.setDaemon(True)
print t1.getName(),t1.isDaemon()
t1.start()
print 'main thread exit'
当
执行到 print 'main thread exit'
后，主线程就退出了，当然t1这个线程也跟着结束了。但是如果不使用t1线程对象的setDaemon()方法的话，即便主线程结束了，还要等待t1线程
自己结束才能退出进程。isDaemon()是用来获得一个线程对象的Daemonflag状态的
如何来获得与线程有关的信息呢？
获得当前正在运行的线程的引用
running = threading.currentThread()
获得当前所有活动对象（即run方法开始但是未终止的任何线程）的一个列表
threadlist = threading.enumerate()
获得这个列表的长度
threadcount = threading.activeCount()
查看一个线程对象的状态调用这个线程对象的isAlive()方法，返回1代表处于“runnable”状态且没有“dead”
threadflag = threading.isAlive()
**2) 线程同歩队列？**
** **我
们经常会采用生产者/消费者关系的两个线程来处理一个共享缓冲区的数据。例如一个生产者线程接受用户数据放入一个共享缓冲区里，等待一个消费者线程对数据
取出处理。但是如果缓冲区的太小而生产者和消费者两个异步线程的速度不同时，容易出现一个线程等待另一个情况。为了尽可能的缩短共享资源并以相同速度工作
的各线程的等待时间，我们可以使用一个“队列”来提供额外的缓冲区。
创建一个“队列”对象
import Queue
myqueue = Queue.Queue(maxsize = 10)
** **Queue.Queue类即是一个队列的同步实现。队列长度可为无限或者有限。可通过Queue的构造函数的可选参数maxsize来设定队列长度。如果maxsize小于1就表示队列长度无限。
将一个值放入队列中
myqueue.put(10)
调
用队列对象的put()方法在队尾插入一个项目。put()有两个参数，第一个item为必需的，为插入项目的值；第二个block为可选参数，默认为
1。如果队列当前为空且block为1，put()方法就使调用线程暂停,直到空出一个数据单元。如果block为0，put方法将引发Full异常。
将一个值从队列中取出
myqueue.get()
调用队列对象的get()方法从队头删除并返回一个项目。可选参数为block，默认为1。如果队列为空且block为1，get()就使调用线程暂停，直至有项目可用。如果block为0，队列将引发Empty异常。
另外： q.task_done():表示q.get()返回的项目已被处理； q.join():表示直到队列中所有项目均被处理。
我们用一个例子来展示如何使用Queue(实参考于：[IBM](http://www.ibm.com/developerworks/cn/aix/library/au-threadingpython/))
#coding:utf-8
'''
今天我们来学习一下python里的多线程问题，并用一个多线程爬虫程序来实验。
@author FC_LAMP
有几点要说明一下：
1) 线程之间共享状态、内存、资源,并且它们相互间易于通信。
'''
import threading,urllib2
import datetime,time
import Queue
hosts = ['http://www.baidu.com','http://news.163.com/','http://xiaomizhou.net','http://iw3c.com']
class ThreadClass(threading.Thread):
def __init__(self,queue):
threading.Thread.__init__(self)
self.queue = queue
def run(self):
'''
run 方法用于要执行的功能
'''
#getName()用于获取线程名称
while True:
#从队列中获取一个任务
host = self.queue.get()
#抓取工作
url = urllib2.urlopen(host)
print url.read(500)
#标记队列工作已完成
self.queue.task_done()
def main():
#创建队列实例
queue = Queue.Queue()
#生成一个线程池
for i in range(len(hosts)):
t = ThreadClass(queue)
#主程序退出时，子线程也立即退出
t.setDaemon(True)
#启动线程
t.start()
#向队列中填充数数
for host in hosts:
queue.put(host)
#只到所有任务完成后，才退出主程序
queue.join()
if __name__=='__main__':
st = time.time()
main()
print '%f'%(time.time()-st)
另外一个更实际的例子，我们将多线程来下载某网站的图片：
#coding:utf-8
'''
@author:FC_LAMP
'''
import urllib2,urllib,socket
import os,re,threading,Queue
import cookielib,time,Image as image
import StringIO
#30 S请求
socket.setdefaulttimeout(30)
#详情页
class spiderDetailThread(threading.Thread):
header = {
'User-Agent':'Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2',
'Referer':'http://www.xxx.com' #这里是某图片网站
}
dir_path = 'D:/test/'
<span class="highlight-k">def</span> <span class="highlight-fm">__init__</span><span class="highlight-p">(</span><span class="highlight-bp">self</span><span class="highlight-p">,</span><span class="highlight-n">queue</span><span class="highlight-p">):</span>
<span class="highlight-n">threading</span><span class="highlight-o">.</span><span class="highlight-n">Thread</span><span class="highlight-o">.</span><span class="highlight-fm">__init__</span><span class="highlight-p">(</span><span class="highlight-bp">self</span><span class="highlight-p">)</span>
<span class="highlight-n">cookie</span> <span class="highlight-o">=</span> <span class="highlight-n">cookielib</span><span class="highlight-o">.</span><span class="highlight-n">CookieJar</span><span class="highlight-p">()</span>
<span class="highlight-n">cookieproc</span> <span class="highlight-o">=</span> <span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">HTTPCookieProcessor</span><span class="highlight-p">(</span><span class="highlight-n">cookie</span><span class="highlight-p">)</span>
<span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">install_opener</span><span class="highlight-p">(</span><span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">build_opener</span><span class="highlight-p">(</span><span class="highlight-n">cookieproc</span><span class="highlight-p">))</span>
<span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">queue</span> <span class="highlight-o">=</span> <span class="highlight-n">queue</span>
<span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">dir_path</span> <span class="highlight-o">=</span> <span class="highlight-n">dir_address</span>
<span class="highlight-k">def</span> <span class="highlight-nf">run</span><span class="highlight-p">(</span><span class="highlight-bp">self</span><span class="highlight-p">):</span>
<span class="highlight-k">while</span> <span class="highlight-kc">True</span><span class="highlight-p">:</span>
<span class="highlight-n">urls</span> <span class="highlight-o">=</span> <span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">queue</span><span class="highlight-o">.</span><span class="highlight-n">get</span><span class="highlight-p">()</span>
<span class="highlight-k">for</span> <span class="highlight-n">url</span> <span class="highlight-ow">in</span> <span class="highlight-n">urls</span><span class="highlight-p">:</span>
<span class="highlight-n">res</span> <span class="highlight-o">=</span> <span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">urlopen</span><span class="highlight-p">(</span><span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">Request</span><span class="highlight-p">(</span><span class="highlight-n">url</span><span class="highlight-o">=</span><span class="highlight-n">url</span><span class="highlight-p">,</span><span class="highlight-n">headers</span><span class="highlight-o">=</span><span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">header</span><span class="highlight-p">))</span><span class="highlight-o">.</span><span class="highlight-n">read</span><span class="highlight-p">()</span>
<span class="highlight-n">patt</span> <span class="highlight-o">=</span> <span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">compile</span><span class="highlight-p">(</span><span class="highlight-sa">r</span><span class="highlight-s1">'<title>([^<]+)</title>'</span><span class="highlight-p">,</span><span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">I</span><span class="highlight-p">)</span>
<span class="highlight-n">patt</span> <span class="highlight-o">=</span> <span class="highlight-n">patt</span><span class="highlight-o">.</span><span class="highlight-n">search</span><span class="highlight-p">(</span><span class="highlight-n">res</span><span class="highlight-p">)</span>
<span class="highlight-k">if</span> <span class="highlight-n">patt</span><span class="highlight-o">==</span><span class="highlight-kc">None</span><span class="highlight-p">:</span>
<span class="highlight-k">continue</span>
<span class="highlight-c1">#获取TITLE</span>
<span class="highlight-n">title</span> <span class="highlight-o">=</span> <span class="highlight-n">patt</span><span class="highlight-o">.</span><span class="highlight-n">group</span><span class="highlight-p">(</span><span class="highlight-mi">1</span><span class="highlight-p">)</span><span class="highlight-o">.</span><span class="highlight-n">split</span><span class="highlight-p">(</span><span class="highlight-s1">'_'</span><span class="highlight-p">)[</span><span class="highlight-mi">0</span><span class="highlight-p">]</span><span class="highlight-c1">#'abc/:*?"<>|'</span>
<span class="highlight-k">for</span> <span class="highlight-n">i</span> <span class="highlight-ow">in</span> <span class="highlight-p">[</span><span class="highlight-s1">''</span><span class="highlight-p">,</span><span class="highlight-s1">'/'</span><span class="highlight-p">,</span><span class="highlight-s1">':'</span><span class="highlight-p">,</span><span class="highlight-s1">'*'</span><span class="highlight-p">,</span><span class="highlight-s1">'?'</span><span class="highlight-p">,</span><span class="highlight-s1">'"'</span><span class="highlight-p">,</span><span class="highlight-s2">"'"</span><span class="highlight-p">,</span><span class="highlight-s1">'<'</span><span class="highlight-p">,</span><span class="highlight-s1">'>'</span><span class="highlight-p">,</span><span class="highlight-s1">'|'</span><span class="highlight-p">]:</span>
<span class="highlight-n">title</span><span class="highlight-o">=</span><span class="highlight-n">title</span><span class="highlight-o">.</span><span class="highlight-n">replace</span><span class="highlight-p">(</span><span class="highlight-n">i</span><span class="highlight-p">,</span><span class="highlight-s1">''</span><span class="highlight-p">)</span>
<span class="highlight-n">title</span> <span class="highlight-o">=</span> <span class="highlight-n">unicode</span><span class="highlight-p">(</span><span class="highlight-n">title</span><span class="highlight-p">,</span><span class="highlight-s1">'utf-8'</span><span class="highlight-p">)</span><span class="highlight-o">.</span><span class="highlight-n">encode</span><span class="highlight-p">(</span><span class="highlight-s1">'gbk'</span><span class="highlight-p">)</span>
<span class="highlight-nb">print</span> <span class="highlight-n">title</span>
<span class="highlight-c1">#获取图片</span>
<span class="highlight-n">cid</span> <span class="highlight-o">=</span> <span class="highlight-n">url</span><span class="highlight-o">.</span><span class="highlight-n">split</span><span class="highlight-p">(</span><span class="highlight-s1">'/'</span><span class="highlight-p">)[</span><span class="highlight-o">-</span><span class="highlight-mi">1</span><span class="highlight-p">]</span><span class="highlight-o">.</span><span class="highlight-n">split</span><span class="highlight-p">(</span><span class="highlight-s1">'c'</span><span class="highlight-p">)[</span><span class="highlight-o">-</span><span class="highlight-mi">1</span><span class="highlight-p">]</span><span class="highlight-o">.</span><span class="highlight-n">split</span><span class="highlight-p">(</span><span class="highlight-s1">'.'</span><span class="highlight-p">)[</span><span class="highlight-mi">0</span><span class="highlight-p">]</span>
<span class="highlight-n">patt</span> <span class="highlight-o">=</span> <span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">compile</span><span class="highlight-p">(</span><span class="highlight-sa">r</span><span class="highlight-s1">'news+Array(".*?<div[^>]+>(.*?)</div>")'</span><span class="highlight-p">,</span><span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">I</span><span class="highlight-o">|</span><span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">S</span><span class="highlight-p">)</span>
<span class="highlight-n">patt</span> <span class="highlight-o">=</span><span class="highlight-n">patt</span><span class="highlight-o">.</span><span class="highlight-n">search</span><span class="highlight-p">(</span><span class="highlight-n">res</span><span class="highlight-p">)</span>
<span class="highlight-k">if</span> <span class="highlight-ow">not</span> <span class="highlight-n">patt</span><span class="highlight-p">:</span>
<span class="highlight-k">continue</span>
<span class="highlight-n">patt</span> <span class="highlight-o">=</span> <span class="highlight-n">patt</span><span class="highlight-o">.</span><span class="highlight-n">group</span><span class="highlight-p">(</span><span class="highlight-mi">1</span><span class="highlight-p">)</span>
<span class="highlight-n">src_patt</span> <span class="highlight-o">=</span> <span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">compile</span><span class="highlight-p">(</span><span class="highlight-sa">r</span><span class="highlight-s1">'.*?src='</span><span class="highlight-p">(</span><span class="highlight-o">.*</span><span class="highlight-err">?</span><span class="highlight-p">)</span><span class="highlight-s1">'.*?'</span><span class="highlight-p">,</span><span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">I</span><span class="highlight-o">|</span><span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">S</span><span class="highlight-p">)</span>
<span class="highlight-n">src_patt</span> <span class="highlight-o">=</span> <span class="highlight-n">src_patt</span><span class="highlight-o">.</span><span class="highlight-n">findall</span><span class="highlight-p">(</span><span class="highlight-n">patt</span><span class="highlight-p">)</span>
<span class="highlight-k">if</span> <span class="highlight-ow">not</span> <span class="highlight-n">src_patt</span><span class="highlight-p">:</span>
<span class="highlight-k">continue</span>
<span class="highlight-c1">#创建目录</span>
<span class="highlight-k">try</span><span class="highlight-p">:</span>
<span class="highlight-n">path</span> <span class="highlight-o">=</span> <span class="highlight-n">os</span><span class="highlight-o">.</span><span class="highlight-n">path</span><span class="highlight-o">.</span><span class="highlight-n">join</span><span class="highlight-p">(</span><span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">dir_path</span><span class="highlight-p">,</span><span class="highlight-n">title</span><span class="highlight-p">)</span>
<span class="highlight-k">if</span> <span class="highlight-ow">not</span> <span class="highlight-n">os</span><span class="highlight-o">.</span><span class="highlight-n">path</span><span class="highlight-o">.</span><span class="highlight-n">exists</span><span class="highlight-p">(</span><span class="highlight-n">path</span><span class="highlight-p">):</span>
<span class="highlight-n">os</span><span class="highlight-o">.</span><span class="highlight-n">makedirs</span><span class="highlight-p">(</span><span class="highlight-n">path</span><span class="highlight-p">)</span>
<span class="highlight-k">except</span> <span class="highlight-ne">Exception</span> <span class="highlight-k">as</span> <span class="highlight-n">e</span><span class="highlight-p">:</span>
<span class="highlight-k">pass</span>
<span class="highlight-k">if</span> <span class="highlight-ow">not</span> <span class="highlight-n">os</span><span class="highlight-o">.</span><span class="highlight-n">path</span><span class="highlight-o">.</span><span class="highlight-n">exists</span><span class="highlight-p">(</span><span class="highlight-n">path</span><span class="highlight-p">):</span>
<span class="highlight-k">continue</span>
<span class="highlight-k">for</span> <span class="highlight-n">src</span> <span class="highlight-ow">in</span> <span class="highlight-n">src_patt</span><span class="highlight-p">:</span>
<span class="highlight-n">name</span> <span class="highlight-o">=</span> <span class="highlight-n">src</span><span class="highlight-o">.</span><span class="highlight-n">split</span><span class="highlight-p">(</span><span class="highlight-s1">'/'</span><span class="highlight-p">)[</span><span class="highlight-o">-</span><span class="highlight-mi">1</span><span class="highlight-p">]</span>
<span class="highlight-c1">#小图</span>
<span class="highlight-n">s_path</span> <span class="highlight-o">=</span> <span class="highlight-n">os</span><span class="highlight-o">.</span><span class="highlight-n">path</span><span class="highlight-o">.</span><span class="highlight-n">join</span><span class="highlight-p">(</span><span class="highlight-n">path</span><span class="highlight-p">,</span><span class="highlight-n">name</span><span class="highlight-p">)</span>
<span class="highlight-n">img</span> <span class="highlight-o">=</span> <span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">urlopen</span><span class="highlight-p">(</span><span class="highlight-n">src</span><span class="highlight-p">)</span><span class="highlight-o">.</span><span class="highlight-n">read</span><span class="highlight-p">()</span>
<span class="highlight-n">im</span> <span class="highlight-o">=</span> <span class="highlight-n">image</span><span class="highlight-o">.</span><span class="highlight-n">open</span><span class="highlight-p">(</span><span class="highlight-n">StringIO</span><span class="highlight-o">.</span><span class="highlight-n">StringIO</span><span class="highlight-p">(</span><span class="highlight-n">img</span><span class="highlight-p">))</span>
<span class="highlight-n">im</span><span class="highlight-o">.</span><span class="highlight-n">save</span><span class="highlight-p">(</span><span class="highlight-n">s_path</span><span class="highlight-p">)</span>
<span class="highlight-c1">#中图</span>
<span class="highlight-n">src</span> <span class="highlight-o">=</span> <span class="highlight-n">src</span><span class="highlight-o">.</span><span class="highlight-n">replace</span><span class="highlight-p">(</span><span class="highlight-s1">'_s.'</span><span class="highlight-p">,</span><span class="highlight-s1">'_r.'</span><span class="highlight-p">)</span>
<span class="highlight-n">name</span> <span class="highlight-o">=</span> <span class="highlight-n">src</span><span class="highlight-o">.</span><span class="highlight-n">split</span><span class="highlight-p">(</span><span class="highlight-s1">'/'</span><span class="highlight-p">)[</span><span class="highlight-o">-</span><span class="highlight-mi">1</span><span class="highlight-p">]</span>
<span class="highlight-n">m_path</span> <span class="highlight-o">=</span> <span class="highlight-n">os</span><span class="highlight-o">.</span><span class="highlight-n">path</span><span class="highlight-o">.</span><span class="highlight-n">join</span><span class="highlight-p">(</span><span class="highlight-n">path</span><span class="highlight-p">,</span><span class="highlight-n">name</span><span class="highlight-p">)</span>
<span class="highlight-n">img</span> <span class="highlight-o">=</span> <span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">urlopen</span><span class="highlight-p">(</span><span class="highlight-n">src</span><span class="highlight-p">)</span><span class="highlight-o">.</span><span class="highlight-n">read</span><span class="highlight-p">()</span>
<span class="highlight-n">im</span> <span class="highlight-o">=</span> <span class="highlight-n">image</span><span class="highlight-o">.</span><span class="highlight-n">open</span><span class="highlight-p">(</span><span class="highlight-n">StringIO</span><span class="highlight-o">.</span><span class="highlight-n">StringIO</span><span class="highlight-p">(</span><span class="highlight-n">img</span><span class="highlight-p">))</span>
<span class="highlight-n">im</span><span class="highlight-o">.</span><span class="highlight-n">save</span><span class="highlight-p">(</span><span class="highlight-n">m_path</span><span class="highlight-p">)</span>
<span class="highlight-c1">#大图</span>
<span class="highlight-n">src</span> <span class="highlight-o">=</span> <span class="highlight-n">src</span><span class="highlight-o">.</span><span class="highlight-n">replace</span><span class="highlight-p">(</span><span class="highlight-s1">'smallcase'</span><span class="highlight-p">,</span><span class="highlight-s1">'case'</span><span class="highlight-p">)</span>
<span class="highlight-n">src</span> <span class="highlight-o">=</span> <span class="highlight-n">src</span><span class="highlight-o">.</span><span class="highlight-n">replace</span><span class="highlight-p">(</span><span class="highlight-s1">'_r.'</span><span class="highlight-p">,</span><span class="highlight-s1">'.'</span><span class="highlight-p">)</span>
<span class="highlight-n">name</span> <span class="highlight-o">=</span> <span class="highlight-n">src</span><span class="highlight-o">.</span><span class="highlight-n">split</span><span class="highlight-p">(</span><span class="highlight-s1">'/'</span><span class="highlight-p">)[</span><span class="highlight-o">-</span><span class="highlight-mi">1</span><span class="highlight-p">]</span>
<span class="highlight-n">b_path</span> <span class="highlight-o">=</span> <span class="highlight-n">os</span><span class="highlight-o">.</span><span class="highlight-n">path</span><span class="highlight-o">.</span><span class="highlight-n">join</span><span class="highlight-p">(</span><span class="highlight-n">path</span><span class="highlight-p">,</span><span class="highlight-n">name</span><span class="highlight-p">)</span>
<span class="highlight-n">img</span> <span class="highlight-o">=</span> <span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">urlopen</span><span class="highlight-p">(</span><span class="highlight-n">src</span><span class="highlight-p">)</span><span class="highlight-o">.</span><span class="highlight-n">read</span><span class="highlight-p">()</span>
<span class="highlight-n">im</span> <span class="highlight-o">=</span> <span class="highlight-n">image</span><span class="highlight-o">.</span><span class="highlight-n">open</span><span class="highlight-p">(</span><span class="highlight-n">StringIO</span><span class="highlight-o">.</span><span class="highlight-n">StringIO</span><span class="highlight-p">(</span><span class="highlight-n">img</span><span class="highlight-p">))</span>
<span class="highlight-n">im</span><span class="highlight-o">.</span><span class="highlight-n">save</span><span class="highlight-p">(</span><span class="highlight-n">b_path</span><span class="highlight-p">)</span>
<span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">queue</span><span class="highlight-o">.</span><span class="highlight-n">task_done</span><span class="highlight-p">()</span>
#例表页
class spiderlistThread(threading.Thread):
header = {
'User-Agent':'Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2',
'Referer':'http://www.xxx.com' #这里某图片网站
}
<span class="highlight-k">def</span> <span class="highlight-fm">__init__</span><span class="highlight-p">(</span><span class="highlight-bp">self</span><span class="highlight-p">,</span><span class="highlight-n">queue</span><span class="highlight-p">,</span><span class="highlight-n">url</span><span class="highlight-p">):</span>
<span class="highlight-n">threading</span><span class="highlight-o">.</span><span class="highlight-n">Thread</span><span class="highlight-o">.</span><span class="highlight-fm">__init__</span><span class="highlight-p">(</span><span class="highlight-bp">self</span><span class="highlight-p">)</span>
<span class="highlight-n">cookie</span> <span class="highlight-o">=</span> <span class="highlight-n">cookielib</span><span class="highlight-o">.</span><span class="highlight-n">CookieJar</span><span class="highlight-p">()</span>
<span class="highlight-n">cookieproc</span> <span class="highlight-o">=</span> <span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">HTTPCookieProcessor</span><span class="highlight-p">(</span><span class="highlight-n">cookie</span><span class="highlight-p">)</span>
<span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">install_opener</span><span class="highlight-p">(</span><span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">build_opener</span><span class="highlight-p">(</span><span class="highlight-n">cookieproc</span><span class="highlight-p">))</span>
<span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">queue</span> <span class="highlight-o">=</span> <span class="highlight-n">queue</span>
<span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">url</span> <span class="highlight-o">=</span> <span class="highlight-n">url</span>
<span class="highlight-k">def</span> <span class="highlight-nf">run</span><span class="highlight-p">(</span><span class="highlight-bp">self</span><span class="highlight-p">):</span>
<span class="highlight-n">i</span> <span class="highlight-o">=</span> <span class="highlight-mi">1</span>
<span class="highlight-k">while</span> <span class="highlight-mi">1</span><span class="highlight-p">:</span>
<span class="highlight-n">url</span> <span class="highlight-o">=</span> <span class="highlight-s1">'</span><span class="highlight-si">%s</span><span class="highlight-s1">list0-</span><span class="highlight-si">%d</span><span class="highlight-s1">.html'</span><span class="highlight-o">%</span><span class="highlight-p">(</span><span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">url</span><span class="highlight-p">,</span><span class="highlight-n">i</span><span class="highlight-p">)</span>
<span class="highlight-n">res</span> <span class="highlight-o">=</span> <span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">urlopen</span><span class="highlight-p">(</span><span class="highlight-n">urllib2</span><span class="highlight-o">.</span><span class="highlight-n">Request</span><span class="highlight-p">(</span><span class="highlight-n">url</span><span class="highlight-o">=</span><span class="highlight-n">url</span><span class="highlight-p">,</span><span class="highlight-n">headers</span><span class="highlight-o">=</span><span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">header</span><span class="highlight-p">))</span><span class="highlight-o">.</span><span class="highlight-n">read</span><span class="highlight-p">()</span>
<span class="highlight-n">patt</span> <span class="highlight-o">=</span> <span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">compile</span><span class="highlight-p">(</span><span class="highlight-sa">r</span><span class="highlight-s1">'<uls+id="container"[^>]+>(.*?)</ul>'</span><span class="highlight-p">,</span><span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">I</span><span class="highlight-o">|</span><span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">S</span><span class="highlight-p">)</span>
<span class="highlight-n">patt</span> <span class="highlight-o">=</span> <span class="highlight-n">patt</span><span class="highlight-o">.</span><span class="highlight-n">search</span><span class="highlight-p">(</span><span class="highlight-n">res</span><span class="highlight-p">)</span>
<span class="highlight-k">if</span> <span class="highlight-ow">not</span> <span class="highlight-n">patt</span><span class="highlight-p">:</span>
<span class="highlight-k">break</span>
<span class="highlight-k">else</span><span class="highlight-p">:</span>
<span class="highlight-n">res</span> <span class="highlight-o">=</span> <span class="highlight-n">patt</span><span class="highlight-o">.</span><span class="highlight-n">group</span><span class="highlight-p">(</span><span class="highlight-mi">1</span><span class="highlight-p">)</span>
<span class="highlight-n">patt</span> <span class="highlight-o">=</span> <span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">compile</span><span class="highlight-p">(</span><span class="highlight-sa">r</span><span class="highlight-s1">'<labels+class="a">.*?href="(.*?)".*?</label>'</span><span class="highlight-p">,</span><span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">I</span><span class="highlight-o">|</span><span class="highlight-n">re</span><span class="highlight-o">.</span><span class="highlight-n">S</span><span class="highlight-p">)</span>
<span class="highlight-n">patt</span> <span class="highlight-o">=</span> <span class="highlight-n">patt</span><span class="highlight-o">.</span><span class="highlight-n">findall</span><span class="highlight-p">(</span><span class="highlight-n">res</span><span class="highlight-p">)</span>
<span class="highlight-k">if</span> <span class="highlight-ow">not</span> <span class="highlight-n">patt</span><span class="highlight-p">:</span>
<span class="highlight-k">break</span>
<span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">queue</span><span class="highlight-o">.</span><span class="highlight-n">put</span><span class="highlight-p">(</span><span class="highlight-n">patt</span><span class="highlight-p">)</span>
<span class="highlight-n">i</span><span class="highlight-o">+=</span><span class="highlight-mi">1</span>
<span class="highlight-n">time</span><span class="highlight-o">.</span><span class="highlight-n">sleep</span><span class="highlight-p">(</span><span class="highlight-mi">3</span><span class="highlight-p">)</span>
<span class="highlight-bp">self</span><span class="highlight-o">.</span><span class="highlight-n">queue</span><span class="highlight-o">.</span><span class="highlight-n">task_done</span><span class="highlight-p">()</span>
'''
多线程图片抓取
'''
if name=='main':
print unicode('---=======图片抓取=====----n先请输入图片的保存地址(一定要是像这样的路径：D:/xxx/ 不然会出现一些未知错误)。n若不输入,则默认保存在D:/test/ 文件夹会自动创建','utf-8').encode('gbk')
dir_address = raw_input(u'地址(回车确定)：'.encode('gbk')).strip()
print unicode('抓取工作马上开始.......','utf-8').encode('gbk')
if not dir_address:
dir_address = 'D:/test/'
if not os.path.exists(dir_address):
#试着创建目录(多级)
try:
os.makedirs(dir_address)
except Exception as e:
raise Exception(u'无法创建目录%s'%(dir_address))
<span class="highlight-n">url</span> <span class="highlight-o">=</span> <span class="highlight-s1">'http://www.xxx.com/'</span> <span class="highlight-c1">#这里是某图片网站</span>
<span class="highlight-n">queue</span> <span class="highlight-o">=</span> <span class="highlight-n">Queue</span><span class="highlight-o">.</span><span class="highlight-n">Queue</span><span class="highlight-p">()</span>
<span class="highlight-n">t1</span> <span class="highlight-o">=</span> <span class="highlight-n">spiderlistThread</span><span class="highlight-p">(</span><span class="highlight-n">queue</span><span class="highlight-p">,</span><span class="highlight-n">url</span><span class="highlight-p">)</span>
<span class="highlight-n">t1</span><span class="highlight-o">.</span><span class="highlight-n">setDaemon</span><span class="highlight-p">(</span><span class="highlight-kc">True</span><span class="highlight-p">)</span>
<span class="highlight-n">t1</span><span class="highlight-o">.</span><span class="highlight-n">start</span><span class="highlight-p">()</span>
<span class="highlight-n">t2</span> <span class="highlight-o">=</span> <span class="highlight-n">spiderDetailThread</span><span class="highlight-p">(</span><span class="highlight-n">queue</span><span class="highlight-p">)</span>
<span class="highlight-n">t2</span><span class="highlight-o">.</span><span class="highlight-n">setDaemon</span><span class="highlight-p">(</span><span class="highlight-kc">True</span><span class="highlight-p">)</span>
<span class="highlight-n">t2</span><span class="highlight-o">.</span><span class="highlight-n">start</span><span class="highlight-p">()</span>
<span class="highlight-k">while</span> <span class="highlight-mi">1</span><span class="highlight-p">:</span>
<span class="highlight-k">pass</span>
来源：http://fc-lamp.blog.163.com/blog/static/17456668720127221363513/
友情链接