#6713 Slow /auth/bare_openid?url=/user/registration

v1.1.0
closed
Cory Johns
performance (126) sf-2 (994)
General
Tim Van Steenburgh
2015-08-20
2013-09-26
Dave Brondsema
No

/auth/bare_openid?url=/user/registration can be extremely slow in production sometimes. To get to this page naturally, log in as an openid-only user and then visit an Allura page like /p/add_project which will send you to that page. Of course you can access it directly too if you want.

On sandboxes, jinja_master/master.html you need to change {% if g.production_mode %}{{g.analytics.display()}}{% endif %} so that it runs on a sandbox. After doing that, I still haven't been able to replicate the issue on a sandbox. I can replicate it on production only the first time using a new openid-only account. Refreshing the page comes back fast the second time.

New Relic trace shows a lot of slow mongo calls, see http://screencast.com/t/SBAapm50R It would make sense that this is database query related, since it's all fast on sandboxes. I haven't been able to find relevant slow queries in prod logs.

This may be specific to the SF theme, since custom_tracking_js is set in the theme.

Discussion

  • Cory Johns - 2013-10-14

    Per the x-ray trace in New Relic, we gathered the following info:

    The slowness is coming from the call to User.my_projects() in rate_limit() while creating the user project on demand for an OpenID user. It looks like line 659 of allura/model/auth.py might be the source of the problem:

    reaching_roles = [ProjectRole.query.get(_id=i) for i in reaching_role_ids]

    Changing that to the following would be much more efficient:

    reaching_roles = ProjectRole.query.find(_id={'$in': reaching_role_ids}).all()

    However, the underlying issue seems to be that OpenID users are being associated with a single User instance that has a ton of ProjectRoles (probably not *anonymous since user_roles() has a short-circuit to return an empty list for *anon).

    A quick fix would be to short-circuit either the user account auto-gen, rate_limit(), or my_projects() for whatever user account gets used for OpenID users. Though, it might be worth while to determine why there are so many ProjectRoles associated with that user and see if we can remove & prevent those. User project auto-gen might be the best place, since the OpenID User instance should have a username value of '' (empty string).

     

  • Cory Johns - 2013-10-14
    • Milestone: forge-backlog --> forge-nov-01
     

  • Dave Brondsema

    Dave Brondsema - 2013-10-21
    • Size: --> 2
     

  • Cory Johns - 2013-10-23
    • status: open --> in-progress
    • assigned_to: Cory Johns
     

  • Cory Johns - 2013-10-23
    • status: in-progress --> code-review
     

  • Cory Johns - 2013-10-23

    allura:cj/6713

     

  • Cory Johns - 2013-10-23

    We should be able to drop the ProjectRoles associated with the empty-username user (ObjectId("4c2b6c3d0594ca6a1200003a")) and this fix ought to keep new ones from being created (due to the change in User.project_role()).

    There are currently 24,027.

     

  • Tim Van Steenburgh - 2013-10-23
    • QA: Tim Van Steenburgh
     

  • Tim Van Steenburgh - 2013-10-23
    • status: code-review --> closed
     

  • Dave Brondsema

    Dave Brondsema - 2013-10-28
    • private: Yes --> No
     

Log in to post a comment.