#4186 Import from mediawiki into wiki tool

v1.0.0
closed
nobody
import (116) 42cc (432)
General
Dave Brondsema
2015-08-20
2012-05-10
Dave Brondsema
No

We should have a script to import from a mediawiki database dump into a discussion tool. Maybe use sqlite to load the database dump? But also have an option to connect to a live database, since we'll be able to take advantage of that in some situations.

See scripts/teamforge-import.py for reference on how to create wiki pages programmatically. That script uses 2 phases, first to extract the data and save it, second to load the data into Allura. We should do that, but make the data files be free from phpbb details, so the loading script can be generic. Some day people can write extractors from other wiki software and use the same loading script.

This code should be within the ForgeWiki tool, and ideally exposed as a paster command.

The import should handle as much as is possible: wiki pages (including converting the format to markdown - not sure if there is any existing conversion libraries available), attachments, history of each page (our wiki pages are versioned too), permissions, "Talk" pages (can go into the discussion for a page - not sure the best way to split up into separate comments), other config options, etc.

Related

Tickets: #4186
Tickets: #4213
Tickets: #4660
Tickets: #5190

Discussion

<< < 1 2 (Page 2 of 2)

  • Yaroslav Luzin - 2012-06-26

    closed #85, all changes are in 42cc_4186

     

  • Yaroslav Luzin - 2012-06-26

    Run the script as following:

    $ paster wiki2markdown ../Allura/development.ini -d dump_dir -n Projects -p jetest -s mysql --user root --password 0000 --db mediawiki_test -a mediawiki_test/fancypants/

    Available options:

    $ paster wiki2markdown ../Allura/development.ini -h
Usage: paster wiki2markdown [options] [<ini file>]
Import wiki from mediawiki-dump to allura wiki

Options:
  -h, --help            show this help message and exit
  -v, --verbose         
  -e, --extract-only    Store data from the mediawiki-dump on the local
                        filesystem; not load into Allura
  -l, --load-only       Load into Allura previously-extracted data
  -d DUMP_DIR, --dump-dir=DUMP_DIR
                        Directory for dump files
  -n NBHD, --neighborhood=NBHD
                        Neighborhood name to load data
  -p PROJECT, --project=PROJECT
                        Project shortname to load data into
  -s SOURCE, --source=SOURCE
                        Database type to extract from (only mysql for now)
  --db_name=DB_NAME     Database name
  --host=HOST           Database host
  --port=PORT           Database port
  --user=USER           User for database connection
  --password=PASSWORD   Password for database connection
  -a ATTACHMENTS_DIR, --attachments-dir=ATTACHMENTS_DIR
                        Path to directory with mediawiki attachments dump
    • status: in-progress --> code-review
     

  • Yaroslav Luzin - 2012-06-28
    • status: code-review --> in-progress
     

  • Yaroslav Luzin - 2012-06-28

    created a new ticket #96: [#4186] Error handling and small amendments (1cp)

     

    Related

    Tickets: #4186

  • Yaroslav Luzin - 2012-07-03

    closed #96 and pushed changes into 42cc_4186

    • status: in-progress --> code-review
     

  • Dave Brondsema

    Dave Brondsema - 2012-08-07
    • status: code-review --> closed
    • qa: Dave Brondsema
    • milestone: forge-backlog --> forge-aug-10
     

  • Dave Brondsema

    Dave Brondsema - 2012-08-07

    It probably would've been better to use sqlalchemy rather than directly depend on mysql, but that's not really a big deal at all since mediawiki is always on mysql.

    I made a few changes to how imports are handled, so that the GPL'd mediawiki library is optional when using Allura in general.

    Merged to dev. Created [#4660] for some followup improvements.

     

    Related

    Tickets: #4660

<< < 1 2 (Page 2 of 2)

Log in to post a comment.