To avoid low rate limits for anonymous API access, we should use an oauth app. http://developer.github.com/v3/#rate-limiting
As best I can tell https://pypi.python.org/pypi/requests-oauthlib is the best oauth v2 library to use. (The "oauth2" library we already use, despite its name, only is for oauth v1) It's license is BSD/MIT style, based on the very good 'requests' library, has good docs and has an active git repo.
I am not super familiar with oauth v2 and github's setup, but based on what I know, here's how I think it should work. Each Allura instance (e.g. your development host, SourceForge, etc) will need to set up a their own Github OAuth App. Then those keys can be placed in the
ini file. Our github importer code will then do the oauth flow to authorize the user requesting an import. No scope is necessary since we're just doing public readonly fetching. We should store the appropriate user tokens (via
user.set_tool_data) so that they are available for the background task, and also can be re-used if the user wants to run another import.
This should all go through a shared mechanism (e.g. override the base
GitHubProjectExtractor) so that it's used for all github related API access. This code should also check the rate limit values and when it reaches the limit, log a warning, and sleep for the amount of time needed until the limit resets).
Of course, we can modify this as needed if my understanding of github oauth isn't correct.