#6388 Deep trace of single pageload

v1.1.0
closed
Dave Brondsema
performance (126) sf-2 (994)
General
Tim Van Steenburgh
2015-08-20
2013-06-24
Dave Brondsema
No

Let's do a trace (e.g. start with TimerMiddleware verbose mode to get every pymongo call; maybe try line_profiler later) of common functionality to every page. Focus on just the core parts of allura, not looking into the specifics of whatever page you're on. Might be helpful to test hitting an external link URL, so there's not much tool-specific going on, just core route handling, project/tool lookup, permission checks, etc. Identify opportunities for improvement.

Discussion

  • Dave Brondsema

    Dave Brondsema - 2013-09-24

    Automatically checking against mongo indexes would be awesome. Perhaps integrating with https://pypi.python.org/pypi/Dex/0.6 to use some of its functionality

     

  • Dave Brondsema

    Dave Brondsema - 2013-12-03
    • Milestone: forge-backlog --> forge-dec-13
     

  • Dave Brondsema

    Dave Brondsema - 2013-12-03
    • Size: --> 2
     

  • Dave Brondsema

    Dave Brondsema - 2013-12-09

    Based on New Relic "xray sessions", potential candidates to investigate more are:

    • controllers/root.py Neighborhood.query.get: 11% of avg _nav.json request. Time spent is all in mongo. Cache it for a while?
    • model/project.py install_anchored_tools: 15% of avg _nav.json request. Most of time spent is in mongo. Seems to be the self.app_configs property load?
    • lib/security.py obj.neighborhood_project: 7.3% of avg _nav.json request. Cache it for a while?
    • post_thread/post_widget.html has repeated sequential calls to Artifact.refs taking 13% of avg ticket view, backrefs (5.2%), user icon_url which is already memozied (8.4%)
     

  • Dave Brondsema

    Dave Brondsema - 2013-12-11
    • assigned_to: Dave Brondsema
     

  • Dave Brondsema

    Dave Brondsema - 2013-12-13
    • Milestone: forge-dec-13 --> forge-dec-27
     

  • Dave Brondsema

    Dave Brondsema - 2013-12-16
    • status: open --> in-progress
     

  • Dave Brondsema

    Dave Brondsema - 2013-12-19

    Neighborhood.query.get is all over the place. For example in a discussion thread, each post calls User.icon_url which calls private_project which uses self.neighborhood but even though that is a LazyProperty it is instance-level so doesn't get re-used much. There are many other places too.

    We could have a general purpose Neighborhood cache system and use it in lots of places. Some tests might not want it to be cached though. Another even generalized possibility is to extend Ming's identity map. It currently indexes objects by _id only, but we could make it do lookups by any unique key(s) for a given model (e.g. url_prefix for Neighborhood).

     

  • Dave Brondsema

    Dave Brondsema - 2013-12-20
    • status: in-progress --> code-review
     

  • Dave Brondsema

    Dave Brondsema - 2013-12-20

    allura:db/6388

    There are a few performance & configuration improvements in this branch, but none of the particular areas mentioned above are addressed. Most of the work in this branch is a script to have a standard benchmark of call counts that can be run for any git revision (can't go earlier than the creation of the script though).

    Run the main script to get current call counts, and to see specific calls allura/scripts/perf$ ./call_count.py -v

    Use the shell wrapper to create a CSV file with multiple commits recorded (suitable for graphing in a spreadsheet app). E.g.: allura/scripts/perf$ ./call_count_walk_git.sh bdde98d..HEAD

    For integration with Jenkins, we should be able to run call_count.py --data-file /some/permanent/location.csv and hook that up to https://wiki.jenkins-ci.org/display/JENKINS/Plot+Plugin

     

  • Tim Van Steenburgh - 2014-01-02
    • QA: Tim Van Steenburgh
     

  • Tim Van Steenburgh - 2014-01-02
    • status: code-review --> closed
     

Log in to post a comment.