#7644 Make /nf/admin/new_projects faster

v1.2.0
closed
General
2015-08-20
2014-08-28
No

/nf/admin/new_projects can take a very long time when showing hundreds of projects. I don't have timermiddleware stats, that'll probably have to be reproduced on a sandbox for detailed analysis of what is slow. I do have data from New Relic which indicates hundreds of mongo queries are being run and it looks like they are invoked from within the template. Perhaps we are doing separate queries for each project and can do a batch query instead?

Related

Tickets: #7644

Discussion

  • Dave Brondsema

    Dave Brondsema - 2014-09-08
    • Size: --> 4
     
  • Igor Bondarenko

    Igor Bondarenko - 2014-09-19
    • labels: performance --> performance, 42cc
    • status: open --> in-progress
    • assigned_to: Igor Bondarenko
     
  • Igor Bondarenko

    Igor Bondarenko - 2014-09-22
    • status: in-progress --> code-review
     
  • Igor Bondarenko

    Igor Bondarenko - 2014-09-22

    I've generated ~2200 projects with scripts/perf/generate-projects.py to test this.

    Closed #658. je/42cc_7644

    The bottleneck is call to get project admins in the template, which generates a lot of queries to lookup ProjectRoles. Roles are cached, but for every call to admins() there's actually a cache miss :)

    There are two reasons for that:

    1. we ask for roles for each project exactly once within every request, and role cache is not preserved between different requests
    2. users_with_named_role() (which Project.admins() uses) does not actually use a cache and always generates a new query

    I've added cache pre-population before rendering template, which generates only one query to get all roles for all given projects and fixed reason (2) above.

    On my test data response time improved from ~12-14sec to ~9-11sec.

    That's better, but not as much as I want it to be :)

    Removing calls to project.admins() entirely gives response time of ~4-5sec for the same data. So another option could be to initially render page without displaying admins. Instead we can add a button "Show admins" for each project which will make an ajax call to get admins or even just a link to a page, which already contains a list of admins (don't sure such page exists, though).

    Any thoughts?

     
  • Anonymous - 2014-09-22

    Originally by: moorman

    Admins are needed as it allows correlation of username to project name, and
    grouping of projects by username.

    On Mon, Sep 22, 2014 at 5:23 AM, Igor Bondarenko jetmind@users.sf.net
    wrote:

    • status: in-progress --> code-review
    • Comment:

    I've generated ~2200 projects with scripts/perf/generate-projects.py to
    test this.

    Closed #658. je/42cc_7644

    The bottleneck is call to get project admins in the template, which
    generates a lot of queries to lookup ProjectRoles. Roles are cached, but
    for every call to admins() there's actually a cache miss :)

    There are two reasons for that:

    1. we ask for roles for each project exactly once within every
      request, and role cache is not preserved between different requests
    2. users_with_named_role() (which Project.admins() uses) does not
      actually use a cache and always generates a new query

    I've added cache pre-population before rendering template, which generates
    only one query to get all roles for all given projects and fixed reason (2)
    above.

    On my test data response time improved from ~12-14sec to ~9-11sec.

    That's better, but not as much as I want it to be :)

    Removing calls to project.admins() entirely gives response time of
    ~4-5sec for the same data. So another option could be to initially render
    page without displaying admins. Instead we can add a button "Show admins"
    for each project which will make an ajax call to get admins or even just a
    link to a page, which already contains a list of admins (don't sure such
    page exists, though).

    Any thoughts?

    Status: code-review
    Milestone: forge-sep-19
    Labels: performance 42cc
    Created: Thu Aug 28, 2014 02:24 PM UTC by Dave Brondsema
    Last Updated: Fri Sep 19, 2014 06:04 AM UTC
    Owner: Igor Bondarenko

    /nf/admin/new_projects can take a very long time when showing hundreds of
    projects. I don't have timermiddleware stats, that'll probably have to be
    reproduced on a sandbox for detailed analysis of what is slow. I do have
    data from New Relic which indicates hundreds of mongo queries are being run
    and it looks like they are invoked from within the template. Perhaps we are
    doing separate queries for each project and can do a batch query instead?


    Sent from sourceforge.net because you indicated interest in
    https://sourceforge.net/p/allura/tickets/7644/

    To unsubscribe from further messages, please visit
    https://sourceforge.net/auth/subscriptions/

     

    Related

    Tickets: #7644

  • Dave Brondsema

    Dave Brondsema - 2014-09-22
    • status: code-review --> closed
    • QA: Dave Brondsema
     
  • Dave Brondsema

    Dave Brondsema - 2015-01-05
    • Milestone: unreleased --> asf_release_1.2.0
     

Log in to post a comment.