#5368 When flushing BatchIndexer, prevent exceeding max BSON doc size

v1.0.0
closed
Tim Van Steenburgh
migration (63) sf-1 (616)
General
nobody
2015-08-20
2012-11-30
Tim Van Steenburgh
No

Discussion

  • Tim Van Steenburgh - 2012-11-30

    forge-classic:tv/5368

     

  • Tim Van Steenburgh - 2012-11-30
    • status: in-progress --> code-review
    • qa: Dave Brondsema
    • size: --> 1
     

  • Dave Brondsema

    Dave Brondsema - 2012-11-30
    • labels: --> migration
     

  • Dave Brondsema

    Dave Brondsema - 2012-11-30
    • Milestone: forge-nov-30 --> forge-dec-14
     

  • Dave Brondsema

    Dave Brondsema - 2012-11-30
    • status: code-review --> closed
     

  • Dave Brondsema

    Dave Brondsema - 2013-01-12

    We got very close but failed. Perhaps best to catch the error and then split the list of args into two tasks. That'd be a general solution that would work for any server configuration (e.g. if we move the BatchIndexer into Allura proper and it's used on a server with a smaller max size)

    Traceback (most recent call last):
  File "/var/local/forge-classic/sfx/lib/migrate.py", line 270, in migrate
    BatchIndexer.flush()
  File "/var/local/forge-classic/sfx/lib/migrate.py", line 102, in flush
    index_tasks.add_artifacts.post(chunk)
  File "/var/local/allura/Allura/allura/lib/decorators.py", line 17, in post
    return M.MonQTask.post(func, args, kwargs)
  File "/var/local/allura/Allura/allura/model/monq_model.py", line 145, in post
    session(obj).flush(obj)
  File "/var/local/env-allura/lib/python2.7/site-packages/TimerMiddleware-0.4-py2.7.egg/timermiddleware/__init__.py", line 105, in wrapper
    return self.run_and_log(func, inst, *args, **kwargs)
  File "/var/local/env-allura/lib/python2.7/site-packages/TimerMiddleware-0.4-py2.7.egg/timermiddleware/__init__.py", line 114, in run_and_log
    return func(*args, **kwargs)
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/base.py", line 29, in inner
    result = func(obj, *args, **kwargs)
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/odmsession.py", line 58, in flush
    self.insert_now(obj, st)
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/base.py", line 29, in inner
    result = func(obj, *args, **kwargs)
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/odmsession.py", line 66, in insert_now
    mapper(obj).insert(obj, st, self, **kwargs)
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/base.py", line 29, in inner
    result = func(obj, *args, **kwargs)
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/mapper.py", line 55, in insert
    session.impl.insert(doc, validate=False)
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/session.py", line 20, in wrapper
    return func(self, doc, *args, **kwargs)
  File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/session.py", line 146, in insert
    bson = self._impl(doc).insert(data, safe=kwargs.get('safe', True))
  File "/var/local/env-allura/lib/python2.7/site-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/collection.py", line 306, in insert
    continue_on_error, self.__uuid_subtype), safe)
  File "/var/local/env-allura/lib/python2.7/site-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 732, in _send_message
    (request_id, data) = self.__check_bson_size(message)
  File "/var/local/env-allura/lib/python2.7/site-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 709, in __check_bson_size
    (max_doc_size, self.__max_bson_size))
InvalidDocument: BSON document too large (16906035 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
     

  • Dave Brondsema

    Dave Brondsema - 2013-01-12
    • Description has changed:

    Diff:

    --- old
+++ new
@@ -0,0 +1 @@
+
    • status: closed --> open
    • assigned_to: Tim Van Steenburgh --> nobody
    • qa: Dave Brondsema --> nobody
    • milestone: forge-dec-14 --> forge-jan-25
     

  • Dave Brondsema

    Dave Brondsema - 2013-01-13

    Need to re-run:

    • pidgin
    • xsltmp
    • ambulant
     

  • Tim Van Steenburgh - 2013-01-14
    • Description has changed:

    Diff:

    --- old
+++ new
@@ -1 +0,0 @@
-
    • status: open --> in-progress
    • assigned_to: Tim Van Steenburgh
     

  • Tim Van Steenburgh - 2013-01-14
    • status: in-progress --> code-review
     

  • Tim Van Steenburgh - 2013-01-14

    forge-classic:tv/5368

     

  • Dave Brondsema

    Dave Brondsema - 2013-01-14
    • status: code-review --> validation
     

  • Dave Brondsema

    Dave Brondsema - 2013-01-14
    • status: validation --> closed
     

Log in to post a comment.