#7795 test_version_race fails occassionally
closed
General
2015-08-20
2014-11-06
No
forgewiki.tests.test_models.TestPageSnapshots.test_version_race fails occasionally. I've seen it every 20 or so runs maybe?
File "/usr/lib64/python2.7/unittest/case.py", line 365, in run
testMethod()
File "/var/local/env-allura/lib/python2.7/site-packages/nose-1.3.4-py2.7.egg/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/jenkins/jenkins-1244/forge/Allura/allura/tests/decorators.py", line 82, in wrapped
return func(*args, **kw)
File "/home/jenkins/jenkins-1244/forge/ForgeWiki/forgewiki/tests/test_models.py", line 55, in test_version_race
assert page.history().count() == 21, page.history().count()
12
-------------------- >> begin captured stdout << ---------------------
Running setup_app() from allura.websetup
--------------------- >> end captured stdout << ----------------------
Standard Error
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/jenkins/jenkins-1244/forge/ForgeWiki/forgewiki/tests/test_models.py", line 46, in <lambda>
t1 = Thread(target=lambda: run(1))
File "/home/jenkins/jenkins-1244/forge/ForgeWiki/forgewiki/tests/test_models.py", line 44, in run
page.commit()
File "/home/jenkins/jenkins-1244/forge/ForgeWiki/forgewiki/model/wiki.py", line 135, in commit
v1 = self.get_version(self.version - 1)
File "/home/jenkins/jenkins-1244/forge/Allura/allura/model/artifact.py", line 524, in get_version
version=n)
File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/odm/mapper.py", line 284, in get
return self.find(kwargs).first()
File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/odm/odmsession.py", line 358, in first
return self.next()
File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/odm/odmsession.py", line 311, in next
return self._next_impl()
File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/odm/odmsession.py", line 286, in _next_impl
doc = self.ming_cursor.next()
File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/base.py", line 74, in next
doc = self.cursor.next()
File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/mim.py", line 567, in next
value = self.iterator.next()
File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/mim.py", line 516, in <genexpr>
result = (doc for doc,match in self._iterator_gen())
File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/mim.py", line 302, in _gen
for doc in self._data.itervalues():
RuntimeError: dictionary changed size during iteration
Closed #717.
The problem is that mim (mongo-in-memory) are not thread-safe. Occasionally we run into situation when one thread inserts something into page history collection (which is just plain dict) and another thread is in the middle of iterating this collection at the same time.
Fix in
ib/7795QA:
On
master:for i in {1..30}; do nosetests forgewiki.tests.test_models; done, make sure it raises exception as in ticket description (usually around 20th run). If you want exception faster - see below..
On
ib/7795:git revert -n ad6cdc9nosetests forgewiki.tests.test_models, make sure it fails and one of the threads raisesDuplicateKeyErrorgit reset --hard HEADto reset the reverted changesfor i in {1..30}; do nosetests forgewiki.tests.test_models; done, make sure all runs pass.
Some notes:
import sys; sys.setcheckinterval(10)at the beginning of the test. This will make python interpreter switch between threads more frequently. Don't set it to0, though, since threads will not be able to finishsetup_global_objects()in time and you'll get "No object has been registered for this thread" errors.Last edit: Igor Bondarenko 2015-02-03
Nice find and fix.