https://forge-allura.apache.org/p/allura/tickets/You searched for labels:"performance"enThu, 14 Sep 2023 18:51:37 -0000https://forge-allura.apache.org/p/allura/tickets/8497/`g.forge_markdown` constructs a new `ForgeMarkdown` with a lot of extensions every time it is used. This can be a bit expensive. Re-using the same instance would probably require updates/fixes to some of our custom extensions. But lots of times `cached_convert` is used and gets a cached result so a full Markdown instance doesn't even need to be created then.Dave BrondsemaThu, 14 Sep 2023 18:51:37 -0000https://forge-allura.apache.org/p/allura/tickets/8497/https://forge-allura.apache.org/p/allura/tickets/8422/followup to [#4359] Like it did with backrefs can we consolidate queries for: - `ForumAttachment` queries - additional `ArtifactReference` queries, maybe these are regular ("forward") references from `def related_artifacts` calling `self.refs` Dave BrondsemaFri, 23 Sep 2022 16:10:19 -0000https://forge-allura.apache.org/p/allura/tickets/8422/https://forge-allura.apache.org/p/allura/tickets/8416/This helps performance issues with certain markdown content & pattern matchingDave BrondsemaFri, 23 Sep 2022 16:10:18 -0000https://forge-allura.apache.org/p/allura/tickets/8416/https://forge-allura.apache.org/p/allura/tickets/8410/Some markdown text can be very slow to process with the `FORGE_LINK_RE` regexes we're using. Long-term we should look at using markdown's `InlineProcessor` instead of `Pattern` for `ForgeLinkPattern` but we can mitigate it pretty well with a small regex tweakDave BrondsemaFri, 23 Sep 2022 16:10:18 -0000https://forge-allura.apache.org/p/allura/tickets/8410/https://forge-allura.apache.org/p/allura/tickets/8409/For every logged-out request, `authenticate_request` does a User query with username None, which is pointless and never finds anything. It also calls `M.User.anonymous()` which maybe could be in memory instead of a database lookup? Could speed up lots of other calls to anonymous() too.Dave BrondsemaFri, 23 Sep 2022 16:10:18 -0000https://forge-allura.apache.org/p/allura/tickets/8409/https://forge-allura.apache.org/p/allura/tickets/8381/Similar to [#8332] (limiting max size for syntax highlighting) set some overall max sizes. Downloading with `?format=raw` can be a higher limit, I've found reading files from a repo is generally quite fast, but rendering them into an html template (even with syntax highlighting omitted) can be slow for very large files.Dave BrondsemaMon, 17 May 2021 15:07:02 -0000https://forge-allura.apache.org/p/allura/tickets/8381/https://forge-allura.apache.org/p/allura/tickets/8360/Dave BrondsemaMon, 17 May 2021 15:07:02 -0000https://forge-allura.apache.org/p/allura/tickets/8360/https://forge-allura.apache.org/p/allura/tickets/8359/Dave BrondsemaMon, 17 May 2021 15:07:02 -0000https://forge-allura.apache.org/p/allura/tickets/8359/https://forge-allura.apache.org/p/allura/tickets/8341/Similar to [#8332] but for diffsDave BrondsemaMon, 17 May 2021 15:07:01 -0000https://forge-allura.apache.org/p/allura/tickets/8341/https://forge-allura.apache.org/p/allura/tickets/8332/Huge files (like 8MB) in code repos can generate very slow timings like this (from timermiddleware stats.log): ``` { "level": "INFO ", "message": { "call_counts": { "git_tool.__init__": 1, "git_tool._get_refs": 2, "git_tool._object": 2, "git_tool.blob_size": 1, "git_tool.commit": 1, "git_tool.is_empty": 3, "git_tool.open_blob": 1, "git_tool.shorthand_for_commit": 2, "git_tool.url_for_commit": 23, "jinja": 1, "ming": 222, "mongo": 170, "repo.Blob.__init__": 2, "repo.Blob.open": 1, "repo.Blob.path": 3, "repo.Commit.__repr__": 3, "repo.Commit.get_tree": 1, "repo.Commit.set_context": 1, "repo.Commit.shorthand_id": 2, "repo.Commit.url": 2, "repo.Tree.__getitem__": 2, "repo.Tree.__repr__": 3, "repo.Tree.get_blob": 1, "repo.Tree.path": 3, "repo.Tree.set_context": 2, "sidebar": 1, "sqlalchemy": 2, "total": 1 }, "request_category": "git", "timings": { "git_tool.__init__": 0, "git_tool._get_refs": 0, "git_tool._object": 1105, "git_tool.blob_size": 485, "git_tool.commit": 116, "git_tool.is_empty": 1191, "git_tool.open_blob": 1959, "git_tool.shorthand_for_commit": 0, "git_tool.url_for_commit": 1, "jinja": 524566, "ming": 2929, "mongo": 3029, "repo.Blob.__init__": 0, "repo.Blob.open": 1959, "repo.Blob.path": 0, "repo.Commit.__repr__": 0, "repo.Commit.get_tree": 191, "repo.Commit.set_context": 0, "repo.Commit.shorthand_id": 0, "repo.Commit.url": 0, "repo.Tree.__getitem__": 4, "repo.Tree.__repr__": 0, "repo.Tree.get_blob": 0, "repo.Tree.path": 0, "repo.Tree.set_context": 0, "sidebar": 3559, "sqlalchemy": 238, "total": 528318 }, "uptime": 3592, "url": "/p/mingw-w64/mingw-w64/ci/8331eb05125cb7fd26462b9be0a772f8d29ccf13/tree/mingw-w64-crt/Makefile.in" }, "name": "stats", "time": "2019-08-23 12:45:01,270" } ```Dave BrondsemaFri, 04 Oct 2019 21:42:55 -0000https://forge-allura.apache.org/p/allura/tickets/8332/https://forge-allura.apache.org/p/allura/tickets/8272/Some discussions with lots of threads can cause artifact_feed queries that are huge, like this: ``` 2019-03-12T23:38:23.950+0000 I COMMAND [conn6349421] warning: log line attempted (3226kB) over max size (10kB), printing beginning and end ... query project-data.artifact_feed query: { $query: { ref_id: { $in: [ "forgediscussion/model/forum/ForumThread#00841475", "forgediscussion/model/forum/ForumThread#00c62d67", "forgediscussion/model/forum/ForumThread#007b66b8", "forgediscussion/model/forum/ForumThread#00c67d7c", "forgediscussion/model/forum/ForumThread#00c84065", "forgediscussion/model/forum/ForumThread#004372fb", "forgediscussion/.......... /ForumThread#2f8fe61bcb", "forgediscussion/model/forum/ForumThread#e255b209ec", "forgediscussion/model/forum/ForumThread#cf784149a4", "forgediscussion/model/forum/ForumThread#94d83cb848", "forgediscussion/model/forum/ForumThread#eb607a976b", "forgediscussion/model/forum/ForumThread#48a736ec01", "forgediscussion/model/forum/ForumThread#706e08a2b4", "forgediscussion/model/forum/ForumThread#a8b9ad8eaf", "forgediscussion/model/forum/ForumThread#8eaf24bc77", "forgediscussion/model/forum/ForumThread#32d92622f7" ] } }, $orderby: { pubdate: -1 } } planSummary: IXSCAN { ref_id: 1, pubdate: -1 }, IXSCAN { ref_id: 1, pubdate: -1 } cursorid:421910004086 ntoreturn:10 ntoskip:0 keysExamined:52445 docsExamined:8252 hasSortStage:1 keyUpdates:0 writeConflicts:0 numYields:411 nreturned:10 reslen:9110 locks:{ Global: { acquireCount: { r: 824 } }, Database: { acquireCount: { r: 412 } }, Collection: { acquireCount: { r: 412 } } } 1758ms ``` Dave BrondsemaMon, 17 Jun 2019 15:19:14 -0000https://forge-allura.apache.org/p/allura/tickets/8272/https://forge-allura.apache.org/p/allura/tickets/8214/We added caching to merge requests' list of commits previously (`allura.model.repository.MergeRequest#commits`). But it still can sometimes be slow the very first time. This is particularly a problem when creating the merge request and there's been no chance for it to get cached. * if 'commits' are available (cached), show them on the page just like we do now * if they aren't, return the page without them. * start a background task to get them * use ajax to fetch the bg tasks results when complete, and insert them into the page There is a similar pattern in place already for seeing if the commits are cleanly mergeable (see "merge_task_status")Dave BrondsemaMon, 24 Sep 2018 19:36:24 -0000https://forge-allura.apache.org/p/allura/tickets/8214/https://forge-allura.apache.org/p/allura/tickets/8189/The forum listings page can be very slow to load when there are a large number of topics.Kenton TaylorWed, 14 Mar 2018 19:42:19 -0000https://forge-allura.apache.org/p/allura/tickets/8189/https://forge-allura.apache.org/p/allura/tickets/8188/On repos, the commit browse page and log page hit disk the most, so it would be useful to have config options to control the default limits for those.Dave BrondsemaWed, 14 Mar 2018 19:42:19 -0000https://forge-allura.apache.org/p/allura/tickets/8188/https://forge-allura.apache.org/p/allura/tickets/8109/The `File` class creates a `GridFS` for basically any operation. This includes viewing any attachment, thumbnail, project icon, project screenshot etc. The pymongo GridFS code creates an index on `{ files_id: 1, n: 1 }` every time (with a little caching from going through `ensure_index`) which is a really unnecessary. Pymongo 3.1 avoids all that index creation (see [here](http://api.mongodb.com/python/current/api/gridfs/index.html) but we aren't able to upgrade to pymongo 3.x since Ming doesn't support it yet: see https://sourceforge.net/p/merciless/bugs/27/ and https://sourceforge.net/p/merciless/mailman/merciless-discuss/thread/CAJfq2JEicmKpwb31uN1ju%3D1jS0bVDS6fGVjk3n-dpL-BEFX3fg%40mail.gmail.com/#msg34734599 We can work around it a little bit for the `GridFS()` constructor on 2.x But the index will still be created on deletes and writes (within the `GridIn` class)Dave BrondsemaThu, 28 Jul 2016 20:48:46 -0000https://forge-allura.apache.org/p/allura/tickets/8109/https://forge-allura.apache.org/p/allura/tickets/8068/The `ensure_index` paster command misses some indexes: `CommitRunDoc` which uses the `repo_commitrun` collection doesn't get its indexes created, because there is no `mapper(...)` line for it at the bottom of `allura/Allura/allura/model/repository.py`. (There is no class to map it to currently) Also the activitystream indexes on `activities` and `nodes` don'te get created, despite code in `EnsureIndexCommand` that uses `activity_odm_session` to try to do so. Must be a bug in it.Dave BrondsemaThu, 17 Mar 2016 15:18:04 -0000https://forge-allura.apache.org/p/allura/tickets/8068/https://forge-allura.apache.org/p/allura/tickets/8019/Some repos can get a large number of cached branches or tags. The large lists cause a ming performance penalty: e.g. one request timing had 296ms in mongo, 2304ms in ming. This is particular true for the sidebar when it hits `repo.forks` property (twice), which loads many repo objects into memory. We could look into ways to not load those fields when not needed, or skip ming validation on them (see `validate_ranges` option of ming `Array`). Or just querying for a count and not getting the objects at all. Or cap the size of the lists. From forge-allura's own mongo, look at the size of these cached branches lists: ~~~~ > db['git-repository'].find({}).forEach(function(u) { print(u.cached_branches && u.cached_branches.length, u.url_path) }); undefined /p/test/ undefined /u/masterbunnyfu/ undefined /u/shujisado/ undefined /p/allura/ 0 /u/swelltt/ undefined /u/mtem/ undefined /u/barki2015/ 0 /p/allura/ 0 /u/dnayak/ 24 /u/brondsem/ 776 /p/allura/ 547 /u/saintaardvark/ 725 /u/lxj/ 550 /u/testmr/ 550 /u/vansteenburgh/ 583 /u/alexluberg/ 543 /u/masterbunnyfu/ 643 /u/jetmind/ 649 /u/heiths/ 577 /u/shujisado/ 641 /u/mishravikas/ 695 /u/yesidid180/ 758 /u/jredd/ 776 /u/pranav/ ~~~~Dave BrondsemaWed, 24 Feb 2016 19:32:39 -0000https://forge-allura.apache.org/p/allura/tickets/8019/https://forge-allura.apache.org/p/allura/tickets/8006/When an activity happens on a project and the `create_timelines` task is run, that executes ActivityStream's `Aggregator.create_timeline`. In a case where there are no new records for some reason, then it calls `get_timeline`. This can be a problem because the pre-computed "timeline" there could potentially be thousands or millions of records. This takes a while and can take up a ton of memory which doesn't get reclaimed after the task is done. We should evaulate if that behavior is correct. If it is needed, we should pass a `limit` parameter in. Also perhaps see if we can figure out why these records aren't being garbage collected.Dave BrondsemaThu, 12 May 2016 18:51:47 -0000https://forge-allura.apache.org/p/allura/tickets/8006/https://forge-allura.apache.org/p/allura/tickets/7963/Currently git and mercurial detect copies when viewing commits. This adds a significant amount of time to each request and is not a common enough occurrence to justify such a large performance hit on every view. But we should also probably add an option in the .ini that gives the option to enable copy detection (noting the performance penalty). [#7925]Heith SeewaldMon, 24 Aug 2015 14:29:16 -0000https://forge-allura.apache.org/p/allura/tickets/7963/https://forge-allura.apache.org/p/allura/tickets/7961/Like [#7873] did for git, cache branch & tag info from Mercurial. Existing code from [#7873] probably could be re-used be refactoring it up into common repo code.Dave BrondsemaTue, 11 Aug 2015 20:34:49 -0000https://forge-allura.apache.org/p/allura/tickets/7961/https://forge-allura.apache.org/p/allura/tickets/7932/The `index` method on `allura.controllers.repository.CommitBrowser` has an issue where every page from the pagination consists of an identical copy of the entire list. so **?page=2** is the same as **?page=10** and they both contain a full (non-paged) version of the changes.Heith SeewaldMon, 27 Jul 2015 14:35:07 -0000https://forge-allura.apache.org/p/allura/tickets/7932/https://forge-allura.apache.org/p/allura/tickets/7925/In a git repo with a large amount of binary files, our diff processing can be very inefficient. We should test if a file is binary and exclude it from the diff processing section.Heith SeewaldMon, 10 Aug 2015 14:27:58 -0000https://forge-allura.apache.org/p/allura/tickets/7925/https://forge-allura.apache.org/p/allura/tickets/7914/From [docs](https://forge-allura.apache.org/docs/getting_started/administration.html#create-allura-sitemap-py): > This takes a while to run on a prod-sized data set. There are a couple of things that would make it faster, if we need/want to. > > 1. Monkeypatch `forgetracker.model.ticket.Globals.bin_count` to skip the refresh (Solr search) and just return zero for everything, since we don’t need bin counts for the sitemap. > 2. Use multiprocessing to distribute the offsets to n subprocesses. Remove speedup suggestions from docs when this is ready.Igor BondarenkoThu, 09 Jul 2015 08:42:14 -0000https://forge-allura.apache.org/p/allura/tickets/7914/https://forge-allura.apache.org/p/allura/tickets/7889/The `markdown_render_max_length` param is an absolute limit to how big something can be for markdown conversion. But `cached_convert()` was added later and is a better approach. E.g. a very large ticket only has to get converted once and then its good, so a `markdown_render_max_length` isn't needed (we even have it set very high in development.ini because of this). However, not all markdown conversions go through `cached_convert()`, for example, commit messages, outbound email bodies, or markdown files in repos. So we should have `markdown_render_max_length` apply only to non-cached conversions and change its default value to something not so extremely high, maybe 100000. Or possibly introduce a second option, one for cached and one for non-cached conversions. Or develop a new caching mechanism that works for markdown text that isn't part of an artifact. An example I came across was thousands of lines like this in the commit message itself, totaling around 500k chars: ~~~~ renamed: ts/5.1/fastboot/fastboot-mangle -> ts/5.2/fastboot/fastboot-mangle renamed: ts/5.1/fastboot/lib-boot -> ts/5.2/fastboot/lib-boot renamed: ts/5.1/fastboot/lib-rw -> ts/5.2/fastboot/lib-rw renamed: ts/5.1/fastboot/usr-rw -> ts/5.2/fastboot/usr-rw renamed: ts/5.1/kernel/alias/acpi -> ts/5.2/kernel/alias/acpi renamed: ts/5.1/kernel/alias/pcm -> ts/5.2/kernel/alias/pcm ~~~~Dave BrondsemaMon, 29 Jun 2015 14:23:15 -0000https://forge-allura.apache.org/p/allura/tickets/7889/