#7795 test_version_race fails occassionally

v1.2.1
closed
General
2015-08-20
2014-11-06
No

forgewiki.tests.test_models.TestPageSnapshots.test_version_race fails occasionally. I've seen it every 20 or so runs maybe?

  File "/usr/lib64/python2.7/unittest/case.py", line 365, in run
    testMethod()
  File "/var/local/env-allura/lib/python2.7/site-packages/nose-1.3.4-py2.7.egg/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/jenkins/jenkins-1244/forge/Allura/allura/tests/decorators.py", line 82, in wrapped
    return func(*args, **kw)
  File "/home/jenkins/jenkins-1244/forge/ForgeWiki/forgewiki/tests/test_models.py", line 55, in test_version_race
    assert page.history().count() == 21, page.history().count()
12
-------------------- >> begin captured stdout << ---------------------
Running setup_app() from allura.websetup

--------------------- >> end captured stdout << ----------------------

Standard Error

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/jenkins/jenkins-1244/forge/ForgeWiki/forgewiki/tests/test_models.py", line 46, in <lambda>
    t1 = Thread(target=lambda: run(1))
  File "/home/jenkins/jenkins-1244/forge/ForgeWiki/forgewiki/tests/test_models.py", line 44, in run
    page.commit()
  File "/home/jenkins/jenkins-1244/forge/ForgeWiki/forgewiki/model/wiki.py", line 135, in commit
    v1 = self.get_version(self.version - 1)
  File "/home/jenkins/jenkins-1244/forge/Allura/allura/model/artifact.py", line 524, in get_version
    version=n)
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/odm/mapper.py", line 284, in get
    return self.find(kwargs).first()
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/odm/odmsession.py", line 358, in first
    return self.next()
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/odm/odmsession.py", line 311, in next
    return self._next_impl()
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/odm/odmsession.py", line 286, in _next_impl
    doc = self.ming_cursor.next()
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/base.py", line 74, in next
    doc = self.cursor.next()
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/mim.py", line 567, in next
    value = self.iterator.next()
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/mim.py", line 516, in <genexpr>
    result = (doc for doc,match in self._iterator_gen())
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.4.7-py2.7.egg/ming/mim.py", line 302, in _gen
    for doc in self._data.itervalues():
RuntimeError: dictionary changed size during iteration

Related

Tickets: #7863

Discussion

  • Dave Brondsema

    Dave Brondsema - 2015-01-26
    • labels: --> sf-current, sf-2
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-01-27
    • Owner: Anonymous --> Igor Bondarenko
    • Labels: sf-current, sf-2 --> 42cc, sf-current, sf-2
    • Status: open --> in-progress
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-02-03
    • status: in-progress --> review
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-02-03

    Closed #717.

    The problem is that mim (mongo-in-memory) are not thread-safe. Occasionally we run into situation when one thread inserts something into page history collection (which is just plain dict) and another thread is in the middle of iterating this collection at the same time.

    Fix in ib/7795

    QA:

    On master:

    • cd ForgeWiki
    • run for i in {1..30}; do nosetests forgewiki.tests.test_models; done, make sure it raises exception as in ticket description (usually around 20th run). If you want exception faster - see below.

    .

    On ib/7795:

    • make sure test still serves its initial purpose:
      • git revert -n ad6cdc9
      • run nosetests forgewiki.tests.test_models, make sure it fails and one of the threads raises DuplicateKeyError
      • git reset --hard HEAD to reset the reverted changes
    • run for i in {1..30}; do nosetests forgewiki.tests.test_models; done, make sure all runs pass

    .

    Some notes:

    • You should run tests in entire file as described above, not individual test. I don't know why, but when I run it as individual test I can't reproduce the error.
    • If you want to generate error faster add import sys; sys.setcheckinterval(10) at the beginning of the test. This will make python interpreter switch between threads more frequently. Don't set it to 0, though, since threads will not be able to finish setup_global_objects() in time and you'll get "No object has been registered for this thread" errors.
     

    Last edit: Igor Bondarenko 2015-02-03
  • Dave Brondsema

    Dave Brondsema - 2015-02-03
    • status: review --> closed
    • Reviewer: Dave Brondsema
     
  • Dave Brondsema

    Dave Brondsema - 2015-02-03

    Nice find and fix.

     
  • Dave Brondsema

    Dave Brondsema - 2015-02-09
    • labels: 42cc, sf-current, sf-2 --> 42cc, sf-2
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-02-18
    • Milestone: unreleased --> asf_release_1.2.1
     

Log in to post a comment.