#7307 Broken handling of InvalidDocument: BSON document too large

v1.2.0
closed
Dave Brondsema
sf-2 (994)
General
Cory Johns
2015-08-20
2014-04-03
Dave Brondsema
No

Ming "helpfully" adds the value of the current doc, when pymongo errors are raised. In the case of docs that are too large, this puts a huge amount of text on the error, which often gets written to a log file or saved back to monq_task (which of course will fail again). Also, in our indexing code where we know we're making big documents, and check for this case so we can split the doc, the check is failing because of ming's slight change to the error object.

Discussion

  • Dave Brondsema

    Dave Brondsema - 2014-04-03
    • status: in-progress --> code-review
     

  • Dave Brondsema

    Dave Brondsema - 2014-04-03

    Truncating the error message in: https://sourceforge.net/p/merciless/code/merge-requests/16/

    Fixing the error checking in: db/7307

    And here's an example showing the differences between pymongo and ming exceptions:

    $ allurapaste shell /var/local/config/production.ini
>>> from ming.orm.ormsession import ThreadLocalORMSession
>>> from allura.model.notification import SiteNotification
>>> s = SiteNotification(content='\xab\xab')  # invalid encoding, but easier to test than doc too large
>>> try:
...     ThreadLocalORMSession.flush_all()
... except Exception as e:
...     pass
...

# ming-fiddled exception
>>> str(e)
'(\'strings in documents must be valid UTF-8\', "doc:  {\'active\': True, \'content\': \'\\\\xab\\\\xab\', \'_id\': ObjectId(\'533d76f4d8be357276af7b29\'), \'impressions\': 0}")'
>>> e.args
('strings in documents must be valid UTF-8', "doc:  {'active': True, 'content': '\\xab\\xab', '_id': ObjectId('533d74e1d8be35710fdeddb6'), 'impressions': 0}")
>>> e.args[0]
'strings in documents must be valid UTF-8'

# now compare to direct pymongo exception
>>> from pymongo.errors import InvalidDocument
>>> str(InvalidDocument('foobar'))
'foobar'
>>> InvalidDocument('foobar').args
('foobar',)
>>> str(InvalidDocument('foobar').args[0])
'foobar'
     

  • Cory Johns - 2014-04-04
    • status: code-review --> closed
    • QA: Cory Johns
     

  • Dave Brondsema

    Dave Brondsema - 2015-01-05
    • Milestone: unreleased --> asf_release_1.2.0
     

Log in to post a comment.