#5368 When flushing BatchIndexer, prevent exceeding max BSON doc size

v1.0.0
closed
General
nobody
2015-08-20
2012-11-30
No

Discussion

  • Tim Van Steenburgh

    forge-classic:tv/5368

     
  • Tim Van Steenburgh

    • status: in-progress --> code-review
    • qa: Dave Brondsema
    • size: --> 1
     
  • Dave Brondsema

    Dave Brondsema - 2012-11-30
    • labels: --> migration
     
  • Dave Brondsema

    Dave Brondsema - 2012-11-30
    • Milestone: forge-nov-30 --> forge-dec-14
     
  • Dave Brondsema

    Dave Brondsema - 2012-11-30
    • status: code-review --> closed
     
  • Dave Brondsema

    Dave Brondsema - 2013-01-12

    We got very close but failed. Perhaps best to catch the error and then split the list of args into two tasks. That'd be a general solution that would work for any server configuration (e.g. if we move the BatchIndexer into Allura proper and it's used on a server with a smaller max size)

    Traceback (most recent call last):
      File "/var/local/forge-classic/sfx/lib/migrate.py", line 270, in migrate
        BatchIndexer.flush()
      File "/var/local/forge-classic/sfx/lib/migrate.py", line 102, in flush
        index_tasks.add_artifacts.post(chunk)
      File "/var/local/allura/Allura/allura/lib/decorators.py", line 17, in post
        return M.MonQTask.post(func, args, kwargs)
      File "/var/local/allura/Allura/allura/model/monq_model.py", line 145, in post
        session(obj).flush(obj)
      File "/var/local/env-allura/lib/python2.7/site-packages/TimerMiddleware-0.4-py2.7.egg/timermiddleware/__init__.py", line 105, in wrapper
        return self.run_and_log(func, inst, *args, **kwargs)
      File "/var/local/env-allura/lib/python2.7/site-packages/TimerMiddleware-0.4-py2.7.egg/timermiddleware/__init__.py", line 114, in run_and_log
        return func(*args, **kwargs)
      File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/base.py", line 29, in inner
        result = func(obj, *args, **kwargs)
      File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/odmsession.py", line 58, in flush
        self.insert_now(obj, st)
      File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/base.py", line 29, in inner
        result = func(obj, *args, **kwargs)
      File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/odmsession.py", line 66, in insert_now
        mapper(obj).insert(obj, st, self, **kwargs)
      File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/base.py", line 29, in inner
        result = func(obj, *args, **kwargs)
      File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/odm/mapper.py", line 55, in insert
        session.impl.insert(doc, validate=False)
      File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/session.py", line 20, in wrapper
        return func(self, doc, *args, **kwargs)
      File "/var/local/env-allura/lib/python2.7/site-packages/Ming-0.3.2dev_20121101-py2.7.egg/ming/session.py", line 146, in insert
        bson = self._impl(doc).insert(data, safe=kwargs.get('safe', True))
      File "/var/local/env-allura/lib/python2.7/site-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/collection.py", line 306, in insert
        continue_on_error, self.__uuid_subtype), safe)
      File "/var/local/env-allura/lib/python2.7/site-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 732, in _send_message
        (request_id, data) = self.__check_bson_size(message)
      File "/var/local/env-allura/lib/python2.7/site-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 709, in __check_bson_size
        (max_doc_size, self.__max_bson_size))
    InvalidDocument: BSON document too large (16906035 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
    
     
  • Dave Brondsema

    Dave Brondsema - 2013-01-12
    • Description has changed:

    Diff:

    --- old
    +++ new
    @@ -0,0 +1 @@
    +
    
    • status: closed --> open
    • assigned_to: Tim Van Steenburgh --> nobody
    • qa: Dave Brondsema --> nobody
    • milestone: forge-dec-14 --> forge-jan-25
     
  • Dave Brondsema

    Dave Brondsema - 2013-01-13

    Need to re-run:

    • pidgin
    • xsltmp
    • ambulant
     
  • Tim Van Steenburgh

    • Description has changed:

    Diff:

    --- old
    +++ new
    @@ -1 +0,0 @@
    -
    
    • status: open --> in-progress
    • assigned_to: Tim Van Steenburgh
     
  • Tim Van Steenburgh

    • status: in-progress --> code-review
     
  • Tim Van Steenburgh

    forge-classic:tv/5368

     
  • Dave Brondsema

    Dave Brondsema - 2013-01-14
    • status: code-review --> validation
     
  • Dave Brondsema

    Dave Brondsema - 2013-01-14
    • status: validation --> closed
     

Log in to post a comment.