我正在运行多线程python脚本.它的作用是抓取网页并插入/更新到mysql中.这是我的代码
mythread.py
<code>import threading
import time
class MyThread (threading.Thread):
def __init__(self, threadID, threadname, q):
threading.Thread.__init__(self)
self.threadID = threadID
self.threadname = threadname
self.queue = q
self.__exitFlag = False
self.__signal_lock = threading.Lock()
def run(self):
print "Starting " + self.threadname
self.process_data()
print "Exiting " + self.threadname
def stop(self):
with self.__signal_lock:
self.__exitFlag = True
def process_data(self):
while not self.__exitFlag:
if not self.queue.empty():
data = self.queue.get()
# crawl data from the web...
# update to mysql
# assuming we have already connected mysql:
# db = MySQLDb()
# db.connect
query = ""
db.query(query)
</code>
mysql_db.py
<code>class MySQLDb:
conn = None
def connect(self):
self.conn = MySQLdb.connect(
host="127.0.0.1",
user = "root",
passwd = "password",
db = "moviestats")
self.cursor = self.conn.cursor(MySQLdb.cursors.DictCursor)
def query(self, sql):
try:
self.cursor.execute(sql)
self.conn.commit()
except (AttributeError, MySQLdb.OperationalError):
# solution to: MySQL server has gone away
self.cursor.close()
self.connect()
self.cursor = self.conn.cursor(MySQLdb.cursors.DictCursor)
self.cursor.execute(sql)
self.conn.commit()
</code>
这是错误日志：
<code>Process: Python [905] Path: /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python Identifier: Python Version: 2.7.7 (2.7.7) Code Type: X86-64 (Native) Parent Process: bash [751] Responsible: Terminal [410] User ID: 501 Date/Time: 2014-07-09 22:31:43.221 +0800 OS Version: Mac OS X 10.9.3 (13D65) Report Version: 11 .... .... Crashed Thread: 5 Exception Type: EXC_CRASH (SIGABRT) Exception Codes: 0x0000000000000000, 0x0000000000000000 Application Specific Information: abort() called *** error for object 0x100a4b600: pointer being freed was not allocated ...... Thread 5 Crashed: 0 libsystem_kernel.dylib 0x00007fff83153866 __pthread_kill + 10 1 libsystem_pthread.dylib 0x00007fff8de8735c pthread_kill + 92 2 libsystem_c.dylib 0x00007fff8ef88b1a abort + 125 3 libsystem_malloc.dylib 0x00007fff8220707f free + 411 4 libmysqlclient.18.dylib 0x0000000101027302 vio_delete + 44 5 libmysqlclient.18.dylib 0x000000010100709a end_server + 48 6 libmysqlclient.18.dylib 0x0000000101006f81 cli_safe_read + 49 7 libmysqlclient.18.dylib 0x000000010100b469 cli_read_query_result + 26 8 libmysqlclient.18.dylib 0x000000010100a648 mysql_real_query + 83 9 _mysql.so 0x0000000100533be8 _mysql_ConnectionObject_query + 85 10 org.python.python 0x00000001000c2fad PyEval_EvalFrameEx + 21405 11 org.python.python 0x00000001000c3bfa PyEval_EvalFrameEx + 24554 12 org.python.python 0x00000001000c3bfa PyEval_EvalFrameEx + 24554 13 org.python.python 0x00000001000c4fb3 PyEval_EvalCodeEx + 2115 14 org.python.python 0x00000001000c33f0 PyEval_EvalFrameEx + 22496 15 org.python.python 0x00000001000c3bfa PyEval_EvalFrameEx + 24554 16 org.python.python 0x00000001000c3bfa PyEval_EvalFrameEx + 24554 17 org.python.python 0x00000001000c4fb3 PyEval_EvalCodeEx + 2115 18 org.python.python 0x00000001000c33f0 PyEval_EvalFrameEx + 22496 19 org.python.python 0x00000001000c3bfa PyEval_EvalFrameEx + 24554 20 org.python.python 0x00000001000c3bfa PyEval_EvalFrameEx + 24554 21 org.python.python 0x00000001000c3bfa PyEval_EvalFrameEx + 24554 22 org.python.python 0x00000001000c4fb3 PyEval_EvalCodeEx + 2115 23 org.python.python 0x000000010003eac0 function_call + 176 24 org.python.python 0x000000010000ceb2 PyObject_Call + 98 25 org.python.python 0x000000010001f56d instancemethod_call + 365 26 org.python.python 0x000000010000ceb2 PyObject_Call + 98 27 org.python.python 0x00000001000bc957 PyEval_CallObjectWithKeywords + 87 28 org.python.python 0x0000000100102f27 t_bootstrap + 71 29 libsystem_pthread.dylib 0x00007fff8de86899 _pthread_body + 138 30 libsystem_pthread.dylib 0x00007fff8de8672a _pthread_start + 137 31 libsystem_pthread.dylib 0x00007fff8de8afc9 thread_start + 13 </code>
我用50个线程运行脚本.发生的错误是间歇性的,但它是可重复的.我缩小了问题,这是由于插入/更新mysql.我读到这可能是由于并发问题,但我该如何解决？
解决方法:
我使用MySQLdb在OSX上遇到了同样的malloc错误.导致我的错误的是在线程之间共享MySQLdb连接.每个线程使用一个连接为我修复它.
从文档http://mysql-python.sourceforge.net/MySQLdb.html：
The MySQL protocol can not handle multiple threads using the same
connection at once. Some earlier versions of MySQLdb utilized locking
to achieve a threadsafety of 2. While this is not terribly hard to
accomplish using the standard Cursor class (which uses
mysql_store_result()), it is complicated by SSCursor (which uses
mysql_use_result(); with the latter you must ensure all the rows have
been read before another query can be executed. It is further
complicated by the addition of transactions, since transactions start
when a cursor execute a query, but end when COMMIT or ROLLBACK is
executed by the Connection object. Two threads simply cannot share a
connection while a transaction is in progress, in addition to not
being able to share it during query execution. This excessively
complicated the code to the point where it just isn’t worth it.The general upshot of this is: Don’t share connections between
threads. It’s really not worth your effort or mine, and in the end,
will probably hurt performance, since the MySQL server runs a separate
thread for each connection. You can certainly do things like cache
connections in a pool, and give those connections to one thread at a
time. If you let two threads use a connection simultaneously, the
MySQL client library will probably upchuck and die. You have been
warned.
【说明】：本文章由站长整理发布，文章内容不代表本站观点，如文中有侵权行为，请与本站客服联系(QQ:254677821)！