Ticket search results

Github api deprecations

Dave Brondsema — Mon, 17 May 2021 15:07:01 -0000

July 1: https://developer.github.com/changes/2020-02-14-deprecating-oauth-app-endpoint/#changes-to-make Nov 13: https://developer.github.com/changes/2020-02-14-deprecating-oauth-auth-endpoint/#changes-to-make https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/

Github import error on deleted users

Dave Brondsema — Mon, 24 Sep 2018 19:36:24 -0000

``` Traceback (most recent call last): File "/var/local/allura/ForgeImporters/forgeimporters/base.py", line 131, in import_tool mount_point=mount_point, mount_label=mount_label, **kw) File "/var/local/allura/ForgeImporters/forgeimporters/github/tracker.py", line 137, in import_tool self.process_events(extractor, ticket, issue) File "/var/local/allura/ForgeImporters/forgeimporters/github/tracker.py", line 205, in process_events self.get_user_link(event['actor']['login'])) TypeError: 'NoneType' object has no attribute '__getitem__' ``` This occurs when a deleted-on-github user is involved in a thread. https://github.com/ampache/ampache/issues/1669 in this case

Check for wiki presence before importing it

Dave Brondsema — Thu, 20 Aug 2015 22:06:52 -0000

The wiki import (both individual and full-project, I suspect) will throw an error if the wiki is not available. We should catch and handle that situation (e.g. silently skip on full-project import, and error to user on individual import form maybe) so that we don't have unnecessary error emails. ~~~~ Importing the Wiki data from the Groove GitHub project to the groovejam SourceForge project failed with the error: 'git clone --bare -v https://github.com/radhouenerouached/Groove.wiki /tmp/tmpU1fq4V' returned exit status 128: fatal: remote error: access denied or repository not exported: /b/nw/bf/80/1f/107924511/28155028.wiki.git Traceback (most recent call last): File "/var/local/allura/ForgeImporters/forgeimporters/base.py", line 131, in import_tool mount_point=mount_point, mount_label=mount_label, **kw) File "/var/local/allura/ForgeImporters/forgeimporters/github/wiki.py", line 162, in import_tool self.import_pages(wiki_url, history=with_history) File "/var/local/allura/ForgeImporters/forgeimporters/github/wiki.py", line 259, in import_pages wiki = git.Repo.clone_from(wiki_url, to_path=wiki_path, bare=True) File "/var/local/env-allura/lib/python2.7/site-packages/GitPython-0.3.2.RC1_20140304-py2.7.egg/git/repo/base.py", line 742, in clone_from return cls._clone(Git(os.getcwd()), url, to_path, GitCmdObjectDB, progress, **kwargs) File "/var/local/env-allura/lib/python2.7/site-packages/GitPython-0.3.2.RC1_20140304-py2.7.egg/git/repo/base.py", line 694, in _clone finalize_process(proc) File "/var/local/env-allura/lib/python2.7/site-packages/GitPython-0.3.2.RC1_20140304-py2.7.egg/git/util.py", line 129, in finalize_process proc.wait() File "/var/local/env-allura/lib/python2.7/site-packages/GitPython-0.3.2.RC1_20140304-py2.7.egg/git/cmd.py", line 101, in wait raise GitCommandError(self.args, status, self.proc.stderr.read()) GitCommandError: 'git clone --bare -v https://github.com/radhouenerouached/Groove.wiki /tmp/tmpU1fq4V' returned exit status 128: fatal: remote error: access denied or repository not exported: /b/nw/bf/80/1f/107924511/28155028.wiki.git ~~~~ https://github.com/radhouenerouached/Groove

Use OAuth token for github project validation

Igor Bondarenko — Thu, 20 Aug 2015 22:07:00 -0000

Use OAuth token for project validation or else github will return 403 (probably only if request limit is reached) and user will see "This project is unavailable for import" message

Issues import from GitHub is broken

Igor Bondarenko — Thu, 20 Aug 2015 22:07:00 -0000

~~~~~ 5:08:33,519 ERROR [taskd:forgeimporters.base.import_tool:5475ecee04687d5bd4a43343:allura.lib.helpers] Failed after 0 retries on url with a timeout of 5.0: https://api.github.com/repos/atom/atom/issues?state=opened&access_token=<...>: HTTP Error 422: Unprocessable Entity Traceback (most recent call last): File "/home/ibondarenko/ibondarenko-1194/forge/Allura/allura/lib/helpers.py", line 1025, in urlopen return urllib2.urlopen(url, timeout=timeout) File "/var/local/env-allura/lib/python2.7/site-packages/TimerMiddleware-0.4.4-py2.7.egg/timermiddleware/__init__.py", line 121, in __call__ return self.run_and_log(self._inner, None, *args, **kwargs) File "/var/local/env-allura/lib/python2.7/site-packages/TimerMiddleware-0.4.4-py2.7.egg/timermiddleware/__init__.py", line 126, in run_and_log return func(*args, **kwargs) File "/usr/lib64/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib64/python2.7/urllib2.py", line 406, in open response = meth(req, response) File "/usr/lib64/python2.7/urllib2.py", line 519, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib64/python2.7/urllib2.py", line 444, in error return self._call_chain(*args) File "/usr/lib64/python2.7/urllib2.py", line 378, in _call_chain result = func(*args) File "/usr/lib64/python2.7/urllib2.py", line 527, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 422: Unprocessable Entity ~~~~~

Validate Github projects before importing

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

Like [#6895] for Google Code, our github importer should check to see if a user/repo pair is valid before importing (and erroring on a 404 later)

Default username for GitHub import

Cory Johns — Thu, 26 Feb 2015 06:09:49 -0000

Since we use oauth to connect to GitHub, we should default the username field on the GitHub import pages, since most people will be importing their own projects. We should still allow changing it, however, since that won't always be the case.

Attached images on comments not importing on GitHub Issues Importer

Cory Johns — Thu, 20 Aug 2015 22:07:10 -0000

Confirmed in [#6922] that the filename is (now) being matched properly and handed off to `Attachment` / `p.add_multiple_attachments()` properly, but it's not showing up on the imported ticket comment. E.g. https://sf-cjohns-1025.sb.sf.net/p/allura/huboard/48/#0df6 vs https://github.com/rauhryan/huboard/issues/48#issuecomment-5397695

Handle github's GH-123 ticket syntax

Dave Brondsema — Tue, 26 Nov 2013 23:01:35 -0000

We don't support GH-123 for linking to tickets (can't find it on github site docs either) like we do for #123 etc.

Handle github emoji

Dave Brondsema — Tue, 30 Oct 2018 14:21:19 -0000

Emoji (e.g. :+1: thumbs up) are heavily used on some GH projects and would be good to carry over during import. However, the images are not licensed for us to redistribute https://github.com/github/gemoji/blob/master/LICENSE Maybe we could use https://github.com/Genshin/PhantomOpenEmoji if the mapping is straightforward. We also need to decide if we want an emoji-like feature throughout our markdown or handle this just at import.

GH import freezes on comment with image

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

Testing `rauhryan/huboard` gets hung up. Running taskd manually and hitting ctrl-c let me see where it was at that point in time: ~~~~ File "/home/dbrondsema/dbrondsema-1019/forge/ForgeImporters/forgeimporters/base.py", line 122, in import_tool mount_point=mount_point, mount_label=mount_label, **kw) File "/home/dbrondsema/dbrondsema-1019/forge/ForgeImporters/forgeimporters/github/tracker.py", line 131, in import_tool self.process_comments(extractor, ticket, issue) File "/home/dbrondsema/dbrondsema-1019/forge/ForgeImporters/forgeimporters/github/tracker.py", line 182, in process_comments body, attachments = self._get_attachments(comment['body']) File "/home/dbrondsema/dbrondsema-1019/forge/ForgeImporters/forgeimporters/github/tracker.py", line 256, in _get_attachments for i, match in enumerate(found_matches): File "/var/local/env-allura/lib/python2.7/site-packages/async-0.6.1-py2.7-linux-x86_64.egg/async/__init__.py", line 20, in thread_interrupt_handler prev_handler(signum, frame) KeyboardInterrupt ~~~~ https://github.com/rauhryan/huboard/issues/48#issuecomment-5396859 is consistently the last comment to be saved to mongo. So something in the next one?

Github import: set timestamp on wiki revisions

Dave Brondsema — Sat, 07 Mar 2015 14:18:00 -0000

When electing to import wiki revisions, all the revisions have the same timestamp instead of the original historical timestamp. They also all get credited to the user who ran the import, but I don't think we can do much about that. Maybe make the user be anonymous instead.

Handle revoked OAuth tokens for GitHub import

Cory Johns — Wed, 14 Dec 2016 16:59:03 -0000

If the user revokes the access token granted to the GitHub importer and then tries to do another import, it fails with a generic message (general failed email to user, generic 401: Unauthorized to engr). It will continue to fail until the bad token is cleared but there is no way to do that other than directly updating the user record in mongo.

Handle directories in github wikis

Dave Brondsema — Tue, 31 Mar 2015 06:42:59 -0000

Currently they are ignored. Looks like github flattens it all out so directories don't show on the web, but it can be referenced within markdown. E.g. see repo https://github.com/snowplow/snowplow.wiki

Textile import issues

Dave Brondsema — Tue, 29 Oct 2013 19:46:14 -0000

Content within `

` tags is lost during import and a single `` tag is shown. e.g. https://github.com/mxcl/homebrew/wiki/Gems%2C-Eggs-and-Perl-Modules/  Somewhat similar, `...` tags are escaped so you see the actual tags when looking at the page.  e.g. https://github.com/mxcl/homebrew/wiki/Installation/ and `
` on https://github.com/mxcl/homebrew/wiki/FAQ

footnotes aren't linked up, e.g. https://github.com/mxcl/homebrew/wiki/Installation/

Support importing tables in github wiki

Dave Brondsema — Tue, 15 Oct 2013 18:58:04 -0000

For non-markdown syntax formats, github wiki uses html2text to import the content, and that doesn't handle tables. https://github.com/aaronsw/html2text/issues/50

Handle github images/files within the wiki

Dave Brondsema — Thu, 19 Sep 2013 15:47:52 -0000

Copied from https://sourceforge.net/p/allura/tickets/6534/#9e5f/6c13/82b8/6599/29c1 > You can have any type of file in the wiki repo and can reference it from the page. When referencing image from the page text it ends up embeded into a page, and when referencing something else (e.g. pdf) it just displayed as link to the file. > I've created example page showing this capabilities https://github.com/jetmind/dot/wiki/Files > Source looks like this: ~~~~ [[Example pdf|example.pdf|width=400px]] [[Link to image|/image.jpg|width=400px]] [[image.jpg|frame|alt=hello|width=400px]] [[http://eofdreams.com/data_images/dreams/image/image-07.jpg|width=400px]] ~~~~ > Also, when embedding an image there are couple of available options (like show in frame, resize, align, etc). Don't sure if we can convert all of those. > I think we can upload such files as attachments to pages from where they are referenced, and convert links/embedd tags to markdown format.

Import github pull requests better

Dave Brondsema — Mon, 23 Feb 2015 14:01:25 -0000

Currently our github ticket import includes pull requests and makes them like regular tickets in Allura. We should at least include some data about what commits were in the pull request. We can also consider making them into Allura merge requests. But Allura merge requests are separate from tickets and don't have all the features like labels, milestones, etc.

Github oauth application

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

To avoid low rate limits for anonymous API access, we should use an oauth app. http://developer.github.com/v3/#rate-limiting As best I can tell https://pypi.python.org/pypi/requests-oauthlib is the best oauth v2 library to use. (The "oauth2" library we already use, despite its name, only is for oauth v1) It's license is BSD/MIT style, based on the very good 'requests' library, has good docs and has an active git repo. I am not super familiar with oauth v2 and github's setup, but based on what I know, here's how I think it should work. Each Allura instance (e.g. your development host, SourceForge, etc) will need to set up a their own Github OAuth App. Then those keys can be placed in the `ini` file. Our github importer code will then do the oauth flow to authorize the user requesting an import. No [scope](http://developer.github.com/v3/oauth/#scopes) is necessary since we're just doing public readonly fetching. We should store the appropriate user tokens (via `user.set_tool_data`) so that they are available for the background task, and also can be re-used if the user wants to run another import. This should all go through a shared mechanism (e.g. override the base `ProjectExtractor.urlopen` in `GitHubProjectExtractor`) so that it's used for all github related API access. This code should also check the rate limit values and when it reaches the limit, log a warning, and sleep for the amount of time needed until the limit resets). Of course, we can modify this as needed if my understanding of github oauth isn't correct.

Apply audit log / import_id and error handling improvements to github importers

Dave Brondsema — Thu, 20 Aug 2015 22:07:20 -0000

[#6541] and [#6530] were developed at the same time as all the github importers. Once all of those are merged in, we should go back and apply the audit log / import_id and error handling logic to the github importers.

Convert or handle Github markdown extensions

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

When importing github content (tickets, wiki, comments) we should deal with their special markup. For example, code blocks with optional language specification: ~~~~ ```javascript function fancyAlert(arg) { if(arg) { $.facebox({div:'#foo'}) } } ``` ~~~~ should be converted to:

~~~~
function fancyAlert(arg) {
  if(arg) {
    $.facebox({div:'#foo'})
  }
}
~~~~

And strikethrough `~~example~~` should be converted to `~~example~~`. This we could possibly support directly in our Markdown renderer if we wanted to. That would also allow it to work for Markdown files in git repos (since we won't modify those during import). Emoji I don't think we should handle (yet?) Cross-reference syntax https://help.github.com/articles/github-flavored-markdown#references we may want to consider handling. See also Trac syntax [#6140] handling. Converting markdown can be tricky to get right, so we have to be careful that we only convert the right content. Nested markup, escaped markup, etc.

Ticket import for github

Dave Brondsema — Thu, 20 Aug 2015 22:07:20 -0000

Use the github api for issues http://developer.github.com/v3/issues/#list-issues-for-a-repository to import tickets into allura. You can use ForgeImporter's GoogleCodeTrackerImporter (at time of writing, cj/6464 on the apache repo has the latest code for it) and TracTicketImporter as reference. Import all the fields & values possible, including comments, events (e.g. status changes) and attachments. See the Google Code and Trac importers for examples of setting up the custom fields as needed, and referencing the remote users (keep the tickets & comments as anonymous and add identification with link into the body).

Wiki importer for github

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

Wikis are git repositories and can be accessed like `git clone https://github.com/OpenRefine/OpenRefine.wiki` for example. Check the main repo API first to see if the repo has wiki enabled. You can see https://sourceforge.net/p/googlecodewikiimporter/git/ for reference as an example of another wiki importer. It is a separate repo because it needs the "html2text" package to convert html to markdown, and that is a GPL library. Github supports many markup types. Find a full list and determine what the best way to convert them to markdown is. My guess is that few formats will have tools available to convert them directly to markdown, so my likely recommendation would be to render them as HTML (using [pypeline](http://pypeline.sourceforge.net/) as a generic way to handle many of those formats) and then html2text to get it into markdown. If html2text or any other GPL library is needed, this will have to be a separate repo from the main Allura repo. So please evaluate & test the conversion options first, before putting code into place. A second phase to all this (i.e. do it separately, after the basic import is all working) would be to handle revision history. This would mean going through each commit in the wiki git repo, and converting & updating every file that changes. This may be very time consuming, so when we get to it, we may want it to be a checkbox option, so users only do it if they want it.

Code importer for github

Dave Brondsema — Thu, 20 Aug 2015 22:07:20 -0000

Create a github importer for git repositories. See ForgeImporter's entry point & class for `GoogleRepoImporter` as reference. That is much more complex though, since it supports git, hg & svn. Github obviously is all git repos, so make the github repo importer nice and simple, no extra code like the Google repo importer needs to have for svn & hg.

Project metadata importer for github

Dave Brondsema — Thu, 20 Aug 2015 22:07:20 -0000

Create an importer that given a github repo name imports the basic information using the github API, into Allura. See ForgeImporter's entry points and the `GoogleCodeProjectImporter` and `TracProjectImporter` classes as similar examples. Github doesn't have a lot of metadata. `description` can go into the "summary" field. homepage can be set. The programming language could be mapped to a programming language trove in Allura. But determining all those mappings (by hand or with some algorithm) is going to be too much work, and not worth it I think.