Ticket search results

Remove Google Code importers

Dave Brondsema — Wed, 27 Jan 2016 18:43:59 -0000

Google Code is in archive mode, so importers as they are now don't work - and some tests fail too. Supporting the archive format would be nice, but at least for now we should just remove the importers.

Wiki not imported properly from Google Code

John Barrett — Mon, 21 Sep 2015 14:06:09 -0000

https://sourceforge.net/p/forge/site-support/11216/ ---- Details from Support Ticket: ---- Hi, I have imported following project from google code. https://sourceforge.net/p/openwonderland/ The code imported successfully but the wiki page didn't look same as on the google code. https://sourceforge.net/p/openwonderland/wiki/OpenWonderland https://code.google.com/p/openwonderland/wiki/OpenWonderland Is it not imported properly?

Error on google code import with paginated comments

Dave Brondsema — Thu, 20 Aug 2015 22:06:09 -0000

Importing https://code.google.com/p/usbloader-gui/issues/detail?id=1769 fails with the following error. I suspect it is due to the comment pagination which we probably don't handle at all. ~~~~ Traceback (most recent call last): File "/var/local/allura/ForgeImporters/forgeimporters/base.py", line 131, in import_tool mount_point=mount_point, mount_label=mount_label, **kw) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 123, in import_tool self.process_fields(ticket, issue) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 164, in process_fields ticket.mod_date = dateutil.parser.parse(issue.get_issue_mod_date()) File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 309, in get_issue_mod_date last_update = Comment(comments[-1], self.project_name) File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 391, in __init__ tag.find('span', 'author').find(True, 'userlink')) AttributeError: 'NoneType' object has no attribute 'find' ~~~~

Google code importer should handle Apache-Extras/EclipseLabs projects

Dave Brondsema — Thu, 20 Aug 2015 22:07:00 -0000

E.g. projects at https://code.google.com/a/eclipselabs.org/hosting/ and https://code.google.com/a/apache-extras.org/hosting/ are slightly different than regular projects.

GC import shouldn't error on "Multiple Licenses"

Dave Brondsema — Sat, 03 Jan 2015 10:06:47 -0000

E.g. http://code.google.com/p/go/ causes this error. ~~~~ Traceback (most recent call last): File "/var/local/allura/ForgeImporters/forgeimporters/google/tasks.py", line 37, in import_project_info extractor.get_license(c.project) File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 175, in get_license project.trove_license.append(trove._id) AttributeError: 'NoneType' object has no attribute '_id' ~~~~ Since it doesn't specify what those multiple licenses are, we can't pick any. So I think we just need to leave the license blank on the Allura side and avoid the error.

Some Google Code icons have wrong file type

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

http://code.google.com/p/multitheftauto/logo?cct=1329973869 for example has `Content-Type: text/html; charset=UTF-8` so `save_image` skips it. It is really a PNG though.

GC date parsing error

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

~~~~ File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 132, in import_tool self.process_fields(ticket, issue) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 171, in process_fields ticket.created_date = datetime.strptime(issue.get_issue_created_date(), '%c') File "/usr/lib64/python2.7/_strptime.py", line 325, in _strptime (data_string, format)) ValueError: time data 'Tue Jul 9 19:03:12 2013' does not match format '%c' ~~~~ Maybe we shouldn't use %c which is locale-dependent?

Validate GC project is publicly readable

Cory Johns — Thu, 20 Aug 2015 22:07:10 -0000

To give better feedback to the user in the case of a typo in the project name, and to prevent invalid projects from being created, the validation of the project name for the GC import should do a HEAD request and confirm a 200 response. Some example projects that fail: sdcardrecovery2013: 403 (private) tauroplay: 404 (typo?)

UnicodeDecodeError on GC ticket import

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

Got this on `umple` ticket import: ~~~~ Traceback (most recent call last): File "/var/local/allura/Allura/allura/model/monq_model.py", line 258, in __call__ self.result = func(*self.args, **self.kwargs) File "/var/local/allura/ForgeImporters/forgeimporters/base.py", line 121, in import_tool mount_point=mount_point, mount_label=mount_label, **kw) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 132, in import_tool self.process_fields(ticket, issue) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 187, in process_fields body=issue.get_issue_description(), File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 227, in get_issue_description return _as_markdown(self.page.find(id='hc0').pre, self.project_name) File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 93, in _as_markdown return ''.join(fragments).strip() UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 491: ordinal not in range(128) ~~~~

Google code cross-linking syntax

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

Google code supports r123 syntax for links, e.g. http://code.google.com/p/umple/issues/detail?can=1&colspec=ID%20Type%20Component%20Diffic%20Status%20Priority%20Owner%20Summary&id=315 And also "issue 123" syntax for links, e.g. http://code.google.com/p/umple/issues/detail?can=1&colspec=ID%20Type%20Component%20Diffic%20Status%20Priority%20Owner%20Summary&id=318 We should convert these into some sort of link when we import. They currently end up as plain text.

GC icon not imported

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

This is a regression. Tested with gmapcatcher and umple.

GC Import fail due to unknown license, with no error notification

Cory Johns — Thu, 20 Aug 2015 22:07:10 -0000

~~~~ Traceback (most recent call last): File "/var/local/allura/Allura/allura/model/monq_model.py", line 258, in __call__ self.result = func(*self.args, **self.kwargs) File "/var/local/allura/ForgeImporters/forgeimporters/google/tasks.py", line 33, in import_project_info extractor.get_license(c.project) File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 131, in get_license project.trove_license.append(trove._id) AttributeError: 'NoneType' object has no attribute '_id' ~~~~ There wasn't an error email sent for this, either, which needs to be investigated.

Make GC Issues importer resilient to upstream changes mid-import

Cory Johns — Thu, 28 Jul 2016 06:57:39 -0000

For long imports, tickets could be added, removed, or made private upstream while the import is running. This can cause the list of ticket numbers to change, and since they're not all pre-loaded, there is a higher chance of this happening and causing `DuplicateKeyError`s. Change `iter_issues` to: * Pull in all issue IDs prior to importing any tickets (this will reduce the window for changes) * Iterate the issue IDs in increasing order instead of the current random order (this will prevent adding new tickets, the most likely change, from causing problems) * Remove any duplicates from the list of issue IDs caused by tickets that were deleted or marked private * Once the import is completed, check for any newly added issues

GC issue parsing bug

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

Importing the Issues data from the heidisql Google Code project to the ctsai-test3 Allura project failed with the error: 'NoneType' object has no attribute 'strip' ~~~~ Traceback (most recent call last): File "/var/local/allura/ForgeImporters/forgeimporters/base.py", line 119, in import_tool mount_point=mount_point, mount_label=mount_label, **kw) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 131, in import_tool self.process_fields(ticket, issue) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 168, in process_fields ticket.summary = issue.get_issue_summary() File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 143, in get_issue_summary text = self.page.find(id='issueheader').findAll('td', limit=2)[1].span.string.strip() AttributeError: 'NoneType' object has no attribute 'strip' ~~~~

GC image attachments don't get thumbnails

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

I noticed these log lines, which come from the thumbnail code: ~~~~ 21:28:03,179 DEBUG [taskd:forgeimporters.base.import_tool:525f049bd8be351e4fb6dadd:allura.model.filesystem] Content type application/octet-stream from file screenshot_example.jpg not supported 21:28:03,187 DEBUG [taskd:forgeimporters.base.import_tool:525f049bd8be351e4fb6dadd:allura.model.filesystem] Content type application/octet-stream from file xGPS_map.db not supported ~~~~ We should try to get the mime-type handling better so that thumbnails for images do work.

GC import: 404 on ticket, list wiki pages, minor form tweaks

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

* drop "at this time" from Google Code notes; it implicitly promises that those features are being worked on. In fact, better to switch the tone and say what can be imported (i.e. all public tickets) and for things that we don't handle (e.g. home page content, additional repos), tell them how to make it happen themselves. Figure out a way to add a link to site-specific docs. * if no default wiki page from GC, set it to `browse_pages`. Need to make that not show up on the list of pages though. Delete the `Home` page. * allow full GC url to be used in project name field, parse out the project name * error when reaching http://code.google.com/p/gmapcatcher/issues/detail?id=373 see trace below: . ~~~~ Traceback (most recent call last): File "/var/local/allura/ForgeImporters/forgeimporters/base.py", line 119, in import_tool mount_point=mount_point, mount_label=mount_label, **kw) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 124, in import_tool for ticket_num, issue in GoogleCodeProjectExtractor.iter_issues(project_name): File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 138, in iter_issues yield (int(issue_id), cls(project_name, 'issue', issue_id=issue_id)) File "/var/local/allura/ForgeImporters/forgeimporters/base.py", line 139, in __init__ self.get_page(page_name, **kw) File "/var/local/allura/ForgeImporters/forgeimporters/base.py", line 172, in get_page parser(self.urlopen(self.url)) File "/var/local/allura/ForgeImporters/forgeimporters/base.py", line 145, in urlopen return h.urlopen(req, retries=retries, codes=codes) File "/var/local/allura/Allura/allura/lib/helpers.py", line 975, in urlopen raise e HTTPError: HTTP Error 404: Not Found ~~~~

StopIteration error on a google code ticket's mod_date logic

Dave Brondsema — Thu, 20 Aug 2015 22:07:20 -0000

~~~~ Importing the Issues data from the pychess Google Code project to the brondsem-pychess Allura project failed with the error: Traceback (most recent call last): File "/var/local/allura/ForgeImporters/forgeimporters/base.py", line 97, in import_tool mount_point=mount_point, mount_label=mount_label, **kw) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 138, in import_tool self.process_fields(ticket, issue) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 178, in process_fields ticket.mod_date = datetime.strptime(issue.get_issue_mod_date(), '%c') File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 165, in get_issue_mod_date last_update = Comment(comments[-1]) File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 241, in __init__ self._get_updates(tag) File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 249, in _get_updates self.updates = {field: updates.next() for field in updates} File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 249, in self.updates = {field: updates.next() for field in updates} StopIteration ~~~~

Error on Google Code ticket import

Tim Van Steenburgh — Thu, 20 Aug 2015 22:07:20 -0000

https://sourceforge.net/nf/admin/task_manager/view/5230c5bd90954737cf132a91 Google Code project: pychess ~~~~ Traceback (most recent call last): File "/var/local/allura/Allura/allura/model/monq_model.py", line 258, in __call__ self.result = func(*self.args, **self.kwargs) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 58, in import_tool importer.import_tool(c.project, c.user, **kw) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 135, in import_tool self.process_fields(ticket, issue) File "/var/local/allura/ForgeImporters/forgeimporters/google/tracker.py", line 192, in process_fields ticket.add_multiple_attachments(issue.get_issue_attachments()) File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 195, in get_issue_attachments return [Attachment(a.parent) for a in attachments.findAll('a', text='Download')] File "/var/local/allura/ForgeImporters/forgeimporters/google/__init__.py", line 277, in __init__ filename = parse_qs(urlparse(url).query)['name'][0] KeyError: 'name' ~~~~

Import multiple repos (of same type) from google code

Dave Brondsema — Fri, 28 Nov 2014 17:51:41 -0000

Git (e.g. http://code.google.com/p/tortoisegit/source/browse?repo=version) and Hg (e.g. http://code.google.com/p/httplib2/source/browse?repo=async2) can support multiple repositories. We should import all of them. I don't know if SVN can have multiple. Naming might be tricky. I think we'd want to preserve the original names if possible. However, the individual tool import form prompts for a single mount point & label - which doesn't align with multiple (automatically named?) repos being created.

Google code imported tickets don't show "votes" in list view

Dave Brondsema — Thu, 20 Aug 2015 22:07:20 -0000

Tickets imported from GC have stars converted to votes, and those show up when looking at an individual ticket, but not when viewing them in a list (milestone, search, etc). They all show 0 votes on the list view.

Fix icon loading & trove names in google code import

Dave Brondsema — Thu, 20 Aug 2015 22:07:54 -0000

~~~~ Traceback (most recent call last): File "/home/dbrondsema/dbrondsema-1019/forge/Allura/allura/model/monq_model.py", line 256, in __call__ self.result = func(*self.args, **self.kwargs) File "/home/dbrondsema/dbrondsema-1019/forge/ForgeImporters/forgeimporters/google/tasks.py", line 32, in import_project_info extractor.get_icon() File "/home/dbrondsema/dbrondsema-1019/forge/ForgeImporters/forgeimporters/google/__init__.py", line 66, in get_icon fp_ish = urllib2.urlopen(icon_url) File "/var/local/env-allura/lib/python2.7/site-packages/timermiddleware/__init__.py", line 120, in __call__ return self.run_and_log(self._inner, None, *args, **kwargs) File "/var/local/env-allura/lib/python2.7/site-packages/timermiddleware/__init__.py", line 125, in run_and_log return func(*args, **kwargs) File "/usr/lib64/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib64/python2.7/urllib2.py", line 392, in open protocol = req.get_type() File "/usr/lib64/python2.7/urllib2.py", line 254, in get_type raise ValueError, "unknown url type: %s" % self.__original ValueError: unknown url type: /p/tortoisesvn/logo?cct=1374605992 ~~~~ ~~~~ Traceback (most recent call last): File "/home/dbrondsema/dbrondsema-1019/forge/Allura/allura/model/monq_model.py", line 256, in __call__ self.result = func(*self.args, **self.kwargs) File "/home/dbrondsema/dbrondsema-1019/forge/ForgeImporters/forgeimporters/google/tasks.py", line 33, in import_project_info extractor.get_license() File "/home/dbrondsema/dbrondsema-1019/forge/ForgeImporters/forgeimporters/google/__init__.py", line 80, in get_license self.project.trove_license.append(trove._id) AttributeError: 'NoneType' object has no attribute '_id' ~~~~ Default image http://www.gstatic.com/codesite/ph/images/defaultlogo.png shouldn't be pulled in

Support private tickets in Google Code Issues importer

Cory Johns — Mon, 09 Mar 2015 03:26:17 -0000

Apparently, GC does in fact support private tickets via the label "private." (NB: when adding the "private" label, GC notes that you're using "an uncommon label" but it does make the ticket only visible by project members.) To support importing them, we will need to perform authentication via [AuthSub](https://developers.google.com/accounts/docs/AuthForWebApps), requiring additional info from the user during either tool or project import.

Create tracker importer for Google Code using CSV and scraping

Cory Johns — Thu, 20 Aug 2015 22:07:20 -0000

Since the Google Data API for Issues is deprecated and was scheduled to be shut down already (June 14th, 2013), we'll need to create an implementation using the CSV list and scraping to ensure that the Google Code importer continues to work. The importer should follow the framework discussed on the [mailing list](http://mail-archives.apache.org/mod_mbox/incubator-allura-dev/201307.mbox/%3CCAEMb8zUg7Kem2aDxVzAqF3U4aKEj7jL3UO=UpX=2+NfY_P8kXQ@mail.gmail.com%3E) and integrate with the project importer from [#6456]. The list of tickets and their metadata can be retrieved via the CSV export list, e.g., https://code.google.com/p/modwsgi/issues/csv but the ticket body and comments will need to be scraped from the web interface. The description and comments can be retrieved from, e.g., https://code.google.com/p/modwsgi/issues/detail?id=22 by iterating over the items with `id="hc\d+"` or `class="issuedescription|issuecomment"`. The description and comments on issues don't support wiki syntax or HTML, so we can just convert them to text. User mapping will have the same issues, so whatever we end up doing in [#6461] will apply here.

Create code importer for Google Code

Cory Johns — Mon, 05 Dec 2016 11:17:43 -0000

During project import, SCM needs to be included. Since Google Code only supports a single repository at a time, we should just have a single Code Importer that checks the project to discover the repo type and passes that information to the appropriate SCM tool's `clone_from_url` method. One way to discover the repo type would be to load `https://code.google.com/p/%s/source/browse/` and get the text of the item `id="crumb_root"`, which will be one of `" git/ "`, `" hg/ "`, or `" svn/ "`.

Create tracker importer for Google Code using gdata API

Cory Johns — Thu, 20 Aug 2015 22:07:55 -0000

Despite being marked for discontinuation on June 14th, 2013, the gdata API for Issues is still functional. Since the gdata library should be easy to implement, it is worth implementing an importer using it while it still works, even if we have to replace it. The importer should follow the framework discussed on the [mailing list](http://mail-archives.apache.org/mod_mbox/incubator-allura-dev/201307.mbox/%3CCAEMb8zUg7Kem2aDxVzAqF3U4aKEj7jL3UO=UpX=2+NfY_P8kXQ@mail.gmail.com%3E) and integrate with the project importer from [#6456]. The [API documentation](https://code.google.com/p/support/wiki/IssueTrackerAPIPython) indicates that authentication is required, but as there is no support for non-public issues and we're doing only reads, it would be good if we can avoid asking for a username / password. (The [feed itself](https://code.google.com/feeds/issues/p/modwsgi/issues/full) doesn't require authentication.) Another example using the API can be found [here](https://github.com/JetBrains/youtrack-rest-python-library/blob/master/python/googleCode2youtrack.py).