#6769 Make GC Issues importer resilient to upstream changes mid-import

v1.1.0
closed
Cory Johns
import (116) google-code (27) sf-2 (994)
General
nobody
2016-07-28
2013-10-17
Cory Johns
No

For long imports, tickets could be added, removed, or made private upstream while the import is running. This can cause the list of ticket numbers to change, and since they're not all pre-loaded, there is a higher chance of this happening and causing DuplicateKeyErrors.

Change iter_issues to:

  • Pull in all issue IDs prior to importing any tickets (this will reduce the window for changes)
  • Iterate the issue IDs in increasing order instead of the current random order (this will prevent adding new tickets, the most likely change, from causing problems)
  • Remove any duplicates from the list of issue IDs caused by tickets that were deleted or marked private
  • Once the import is completed, check for any newly added issues

Related

Tickets: #6769

Discussion

  • Cory Johns - 2013-10-17

    I'm pretty sure the failure of task 525ffd3ec431435c937496a9 was caused by this.

     

  • Cory Johns - 2013-10-17
    • status: in-progress --> code-review
     

  • Cory Johns - 2013-10-17

    allura:cj/6769

     

  • Cory Johns - 2013-10-18
    • QA: Wayne Witzel ❸
     

  • Anonymous - 2013-10-18

    Originally by: algorithms

    • status: code-review --> closed
     

  • Anonymous - 2013-10-18

    Originally by: algorithms

    Looks good. The tests properly exercise the described bug. Merged to master.

     

  • Anonymous - 2013-10-18

    Originally by: jormaral

    Hello, please admin, unsubscribe me from this mailiing list.

    Thanks in advance.

    On Fri, Oct 18, 2013 at 1:40 PM, Cory Johns masterbunnyfu@users.sf.netwrote:

    • QA: Wayne Witzel ❸

    Status: code-review
    Labels: import google-code
    Created: Thu Oct 17, 2013 09:42 PM UTC by Cory Johns
    Last Updated: Thu Oct 17, 2013 11:21 PM UTC
    Owner: Cory Johns

    For long imports, tickets could be added, removed, or made private
    upstream while the import is running. This can cause the list of ticket
    numbers to change, and since they're not all pre-loaded, there is a higher
    chance of this happening and causing DuplicateKeyErrors.

    Change iter_issues to:

    • Pull in all issue IDs prior to importing any tickets (this will
      reduce the window for changes)
    • Iterate the issue IDs in increasing order instead of the current
      random order (this will prevent adding new tickets, the most likely change,
      from causing problems)
    • Remove any duplicates from the list of issue IDs caused by tickets
      that were deleted or marked private
    • Once the import is completed, check for any newly added issues

    Sent from sourceforge.net because allura-dev@incubator.apache.org is
    subscribed to https://sourceforge.net/p/allura/tickets/

    To unsubscribe from further messages, a project admin can change settings
    at https://sourceforge.net/p/allura/admin/tickets/options. Or, if this is
    a mailing list, you can unsubscribe from the mailing list.

     

    Related

    Tickets: #6769

  • Cory Johns - 2013-10-18

    Jorge,

    It appears that you are subscribed to allura-dev@incubator.apache.org, which receives ticket notifications, currently. We are looking into moving the ticket notifications to a separate mailing list and / or making the ticket notifications less "noisy" (see this thread, and feel free to chime in).

    In the meantime, you have two options:

    • Filter ticket notifications (I use subject:("[allura:tickets]") list:"allura-dev.incubator.apache.org" in gmail, personally)
    • Unsubscribe to allura-dev by sending an email to unsubscribe-allura-dev@incubator.apache.org

    Apologies for the inconvenience.

     

  • Libri Nozze - 2016-07-28

    The link "sourceforge...../p/allura/admin/tickets/options" now is "Error 403".

     

Log in to post a comment.