We currently deploy allura in a single-threaded mod_wsgi configuration. Using multiple threads in the past gave very poor performance. Figure out what doesn't work well with multiple threads.
Basic ab testing with 500 requests on a development machine using mod_wsgi shows reasonable behavior when adding threads. Note timings below end up varying between re-runs, so only consider significant changes. ab was run with concurrency matching mod_wsgi's proc*thread, and a few manual requests were made first to make sure it was warmed up
I simulated a test where I/O is blocking requests, and threading should give an advantage to overall throughput, by adding a sleep(3) to the controller requested.
Just to confirm that the above 2x net speedup was not due to just running ab with concurrency 4, I ran with the 2 proc 1 thread configuration again, and kept ab at concurrency 2. As you can see, total time was at the same range as the first test, and individual requests took 2x slower (6sec) because ab was making twice as many simultaneous requests as mod_wsgi could service, so they had to wait.
~~~~
Concurrency Level: 4
Time taken for tests: 158.275591 seconds
Requests per second: 0.63 [#/sec] (mean)
Time per request: 6331.024 [ms] (mean)
Time per request: 1582.756 [ms] (mean, across all concurrent requests)
Percentage of the requests served within a certain time (ms)
50% 6219
66% 6221
75% 6232
80% 6327
90% 6887
95% 7063
98% 7772
99% 7925
100% 7925 (longest request)
~~~~
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
History of our configuration changes are at https://control.siteops.geek.net/git/?p=mastertree.git;a=history;f=host/sfn-web/etc/httpd/conf.d/wsgi-daemon.inc;h=f59577826ca85f3d6b61d1a31e44b4ca047685d9;hb=HEAD Graphite data doesn't go back to the 2012-10-08 to 2012-10-11 range, but stats.log files may still be on the hosts, so we can look at that to learn from that attempt.
My recollection is that performance was very erratic. I don't recall errors, but we should look logs to check.
TimerMiddleware was made thread-safe on 2012-11-05, so performance data during the 2012-10 attempt is not trustworthy.
Basic
abtesting with 500 requests on a development machine using mod_wsgi shows reasonable behavior when adding threads. Note timings below end up varying between re-runs, so only consider significant changes.abwas run with concurrency matching mod_wsgi's proc*thread, and a few manual requests were made first to make sure it was warmed upbaseline: 2 proc, 1 thread
switch it around: 1 proc 2 threads
Slower, roughly by 2x total time. Makes sense since only 1 process can actually be busy
even more threads: 1 proc 5 threads, 5 concur
Roughly same total time to serve 500 requests, but most individual requests were slower.
2 proc 2 threads
Compared to first baseline, this just adds a thread to each process. Results are very comparable on all aspects
I simulated a test where I/O is blocking requests, and threading should give an advantage to overall throughput, by adding a sleep(3) to the controller requested.
baseline: 2 procs, 1 thread
add threads: 2 procs, 2 thread
This completed in about half the time, as expected. Upper end of requests were slower though.
2 procs, 1 thread
Just to confirm that the above 2x net speedup was not due to just running
abwith concurrency 4, I ran with the 2 proc 1 thread configuration again, and keptabat concurrency 2. As you can see, total time was at the same range as the first test, and individual requests took 2x slower (6sec) becauseabwas making twice as many simultaneous requests as mod_wsgi could service, so they had to wait.~~~~
Concurrency Level: 4
Time taken for tests: 158.275591 seconds
Requests per second: 0.63 [#/sec] (mean)
Time per request: 6331.024 [ms] (mean)
Time per request: 1582.756 [ms] (mean, across all concurrent requests)
Percentage of the requests served within a certain time (ms)
50% 6219
66% 6221
75% 6232
80% 6327
90% 6887
95% 7063
98% 7772
99% 7925
100% 7925 (longest request)
~~~~
Seems to work fine - can reopen if issues arise.