Apache Allura™

Hey, I am currently working on this.

Hi Fahri,

I saw your question on IRC about a forum export example, but then you disconnected. Here's a super simple one you can start with. I can give you more complex ones, with attachments, different users, etc, if you want. Also what problem are you getting with your own export? Does the taskd process log an error?

Hi Dave,

the current version of the discussion importer is able to import this example (This was the latest commit: https://forge-allura.apache.org/u/fahri/allurafork/ci/063507d5a23b611f3537f3434b4a7c172650b127/). I have added some screenshots as attachment.

If I try to export my discussion at https://forge-allura.apache.org/u/fahri/discussion/ I receive the following E-Mail:

The bulk export for project u/fahri is completed.

The following tools were exported:
- discussion

Follow the instructions below to get your exported data.

Sample instructions for u/fahri

I would be glad if you send me more examples of a discussion export. Thanks in advance :)

Ok the error you got looks to be something that changed with mongodb 3.6 and we haven't updated for. I created ticket [#8348] for that. You could try a prior version of mongo, or work on code changes for [#8348] (if you do that, please make sure they are separate commits from import work, so it could be merged separately)

Exports on https://forge-allura.apache.org aren't fully enabled. They run, but we don't have any way set up for users to access and download them.

I'll get back to you again before long with more export examples. Exciting to see you got something working!

Here's another forum export, illustrating a number of different situations. Some things (like sticky & announcement posts) don't seem to be marked in export so guess you can't do anything about that during the import. Should be added to the export/API at some point.

If update to the latest 'master' code, that works with new versions of Mongodb now ([#8348] is fixed), so you can test your own exports if you want.

Thank you Dave!
Unfortunately I have the next problem: If I try to clone or pull from my repository at forge allura I get this error:

fahri@debian:~/git/fahri-allurafork$ git pull
fatal: unable to access 'https://forge-allura.apache.org/git/u/fahri/allurafork/': The requested URL returned error: 500

But it is still possible to clone the "master" code:
git clone https://gitbox.apache.org/repos/asf/allura.git/ allura-git

Hey Dave,
I've imported the last export that you have sent to me. I've added some screenshots as attachments.

For the post with attachments I have used some other attachments. Because the URLs in the export weren't reachable anymore (the server reponded with a 404).

Hi,

For the post with attachments, you don't necessarily need to reach the original URL. Forum exports may or may not include a copy of the attachments, in this case they did, so you can use "path": "discussion/5e32098a07ae3108f6c98c23/251636cb3f/be44/keepass.png", to locate the attachment file within the zip file.

For the post with lots of replies, some of the comments should be nested replies. If you look at "slug": "0d1d/d421/f37d" for example in the json, that represents it is a nested child of "slug": "0d1d/d421" which is a nested child of "slug": "0d1d". If you can preserve the original slugs when creating the comments then I think that slug format should automatically make them be shown as nested.

Otherwise looking pretty great!

-Dave

While creating tests for the controller I encountered another thing which I don't understand. I do use forge/test_tracker.py as an example for writing my tests. In the "test_index"-method of the "TestForgeTrackerImportController" class there is this path of a URL that I don't really understand: '/p/test/admin/bugs/_importer/'. How is this path assembled and from where does the "bugs" part come from?

    @with_tracker
    def test_index(self):
        r = self.app.get('/p/test/admin/bugs/_importer/')
        self.assertIsNotNone(r.html.find(attrs=dict(name="tickets_json")))
        self.assertIsNotNone(r.html.find(attrs=dict(name="mount_label")))
        self.assertIsNotNone(r.html.find(attrs=dict(name="mount_point")))

I have used another path for my tests: "'/p/test/admin/ext/import/forge-discussion/'". When I use this path my tests pass. Is this also a valid solution?

This is my test for the Controller:

class TestForgeDiscussionController(TestController, TestCase):

     def setUp(self):
         super(TestForgeDiscussionController, self).setUp()
         from forgediscussion.forum_main import ForumAdminController
         ForumAdminController._importer = \                 discussion.ForgeDiscussionImportController(discussion.ForgeDiscussionImporter())


     @with_discussion
     def test_index(self):
         r = self.app.get('/p/test/admin/ext/import/forge-discussion/')
         self.assertIsNotNone(r.html.find(attrs=dict(name="discussions_json")))
         self.assertIsNotNone(r.html.find(attrs=dict(name="mount_label")))
         self.assertIsNotNone(r.html.find(attrs=dict(name="mount_point")))


     @with_discussion
     @mock.patch('forgeimporters.forge.discussion.save_importer_upload')
     @mock.patch('forgeimporters.base.import_tool')
     def test_create(self, import_tool, siu):
         project = M.Project.query.get(shortname='test')
         params = {
             'discussions_json': webtest.Upload('discussions.json', b'{"key": "val"}'),
             'mount_label': 'mylabel',
             'mount_point': 'mymount',
         }
         r = self.app.post('/p/test/admin/ext/import/forge-discussion/create', params,
                           status=302)
         self.assertEqual(r.location, 'http://localhost/p/test/admin/')
         siu.assert_called_once_with(project, 'discussions.json', '{"key": "val"}')
         self.assertEqual(
             'mymount', import_tool.post.call_args[1]['mount_point'])
         self.assertEqual(
             'mylabel', import_tool.post.call_args[1]['mount_label'])


     @with_discussion
     @mock.patch('forgeimporters.forge.discussion.save_importer_upload')
     @mock.patch('forgeimporters.base.import_tool')
     def test_create_limit(self, import_tool, siu):
         project = M.Project.query.get(shortname='test')
         project.set_tool_data('ForgeDiscussionImporter', pending=1)
         ThreadLocalORMSession.flush_all()
         params = {
             'discussions_json': webtest.Upload('discussions.json', b'{"key": "val"}'),
             'mount_label': 'mylabel',
             'mount_point': 'mymount',
         }
         r = self.app.post('/p/test/admin/ext/import/forge-discussion/create', params,
                           status=302).follow()
         self.assertIn('Please wait and try again', r)
         self.assertEqual(import_tool.post.call_count, 0)

Well, I have just realized that nested posts were still handled wrong by the importer. Here is a new screenshot of these posts which were imported by the new version of the importer.

• importing with a whole zip file would be nice to do, but could be a phase 2. It would take some additional work to be able to handle very large zip files, and existing importers don't handle them yet either. When it is added, it would be good to add it for discussions and tickets and share logic between them.
• for tests, the /p/test project automatically is created. And @with_tracker creates a "bugs" ticket tracker on the /p/test project. Similarly, @with_discussion creates a "discussion" discussion tool. As for the _importer part of the URL, I can see in the setUp that it added in the setUp but I don't know offhand why it does it that way. Using the real URL like /p/test/admin/ext/import/forge-discussion definitely seems better to me. And then you don't need to do anything with _importer in the setUp
• those nested posts look good now!

Nice job so far, looks like you're getting quite close to having this all ready :D

Thank you for explaining this! I've just cleaned up my scripts so they match pep8 code style guidelines and made a merge request.

Hi, I haven't had a chance to look at this yet and it might take several days more as it is a big contribution. But I wanted to let you know I've seen it and will get to it when I can. Thanks for working on this!

Done with https://forge-allura.apache.org/p/allura/git/merge-requests/334/

Hi,

After merging this I noticed a minor issue, but wanted to let you know and see if you were interested in doing a little bit more cleanup. PEP8 is a code style guideline for python and there are various tools to check and to enforce it. We're not super strict about it with Allura, but we like to follow the guidelines when possible. ForgeImporters/forgeimporters/tests/forge/test_discussion.py and ForgeImporters/forgeimporters/forge/discussion.py have quite a few places that don't follow PEP8 exactly. Do you want to see about tidying them up? Depending on your editor, you may have an option to automatically reformat the files. Or you can use a tool flake8 or autopep8. In all cases, please set the max line length to 119 (that is the setting we use for Allura, some tools may pick it up automatically but some might default to a smaller width can cause extra line wrapping which we don't like).

Thanks!