#5076 svn hooks need conversion

asf_release_1.0.0
closed
Cory Johns
General
2015-02-24
2012-10-05
Chris Tsai
No

Due to differences in server configs, migrated svn hooks don't seem to work.

Errors with pre-commit hooks can cause commits to get blocked, ref: [forge:site-support:#874]

after upgrading the megameklab project, we can't checking anything into SVN now. We get the following error:
/svn/p/megameklab/code/hooks/pre-commit: line 3: /var/local/mastertree/host/sfp-svn/hook-scripts/check-case-insensitive.py: No such file or directory

I imagine other hooks would fail similarly. We should probably also remove any unnecessary hooks and update the data on other hooks. For example, the user in [forge:site-support:#862] had the following:

-bash-3.2$ pwd
/home/svn/p/callflow/code/hooks

-bash-3.2$ cat post-commit     
#!/bin/bash
# The following is required for site integration, do not remove/modify.
# Place user hook code in post-commit-user and it will be called from here.
curl -s http://sourceforge.net/auth/refresh_repo/p/callflow/code/

DIR="$(dirname "${BASH_SOURCE[0]}")"
if [ -x $DIR/post-commit-user ]; then  exec $DIR/post-commit-user "$@"

-bash-3.2$ cat post-commit-user 
#!/bin/sh
# THIS FILE IS AUTOGENERATED - DO NOT EDIT
/var/local/mastertree/host/sfp-svn/hook-scripts/sf-svn-stats-hook.py $*
/var/local/mastertree/host/sfp-svn/hook-scripts/keepsake -p "$1"
/var/local/mastertree/host/sfp-svn/hook-scripts/svnnotify --repos-path "$1" --revision "$2" --to "callflow-svn@lists.sourceforge.net" --subject-prefix "SF.net SVN: callflow:" --subject-cx --no-first-line --with-diff --viewcvs-url "http://callflow.svn.sourceforge.net/callflow/?rev=%s&view=rev" --user-domain "users.sourceforge.net" --footer "This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site." -l /usr/bin/svnlook --max-diff-length 100000
exit 0
  • We should probably first remove the "AUTOGENERATED" link, since that's no longer true.

  • I think we can probably just remove the sf classic specific hooks (sf-svn-stats-hook.py and keepsake).

  • For svnnotify, we should fix the path (iirc, it's in the standard location /usr/bin/svnnotify/ though just svnnotify works), and update the url to the code browser (ie., the --viewcvs-url option)

We have a few other pre-configured hooks users can add, but we didn't allow custom hooks for svn repos on classic, so it should be a finite list.

Discussion

1 2 > >> (Page 1 of 2)
  • Dave Brondsema
    Dave Brondsema
    2012-10-05

    • size: --> 1
     

  • Anonymous
    2012-10-16

    Originally by: maddes

    Hi there,

    I can not find any svnnotify on the server.
    Here's how I searched:

    -bash-3.2$ hostname
    shell-24006
    -bash-3.2$ pwd
    /home/svn/p/regshot/code/hooks
    -bash-3.2$ echo $PATH
    /usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin
    -bash-3.2$ which svnnotify
    /usr/bin/which: no svnnotify in (/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin)
    -bash-3.2$ find / -name "svnnotify"
    -bash-3.2$
    

    Similar issue for check-case-insensitive.py and check-mime-type.pl in the converted pre-commit hook:

    -bash-3.2$ find / -name "check-case-insensitive.py"
    -bash-3.2$
    -bash-3.2$ find / -name "check-mime-type.pl"
    /usr/share/doc/subversion-1.6.17/contrib/hook-scripts/check-mime-type.pl
    

    What are the correct pathes for:
    "svnnotify" in post-commit-user
    "check-case-insensitive.py" in pre-commit
    * "check-mime-type.pl" in pre-commit

    Or do have I to place them myself somewhere and call from there? If so, what is the recommended path?

    Maddes

    Current content of pre-commit after upgrade to Allura:

    -bash-3.2$ cat pre-commit
    #!/bin/sh -e
    # THIS FILE IS AUTOGENERATED - DO NOT EDIT
    /var/local/mastertree/host/sfp-svn/hook-scripts/check-case-insensitive.py "$1" "$2"
    /var/local/mastertree/host/sfp-svn/hook-scripts/check-mime-type.pl "$1" "$2"
    exit 0
    
     

  • Chris Tsai
    2012-10-16

    Hello Maddes,

    The server for shell sessions is not the same as the server that runs commit hooks. On the servers that run the scripts, svnnotify is in the standard /usr/bin/svnnotify location (though just using svnnotify without the full path also works)

    As for the other scripts, I'm not certain that those are installed on that server yet, but I expect the work on this ticket to root those out and get them added properly.

    Regards,
    Chris Tsai, SourceForge.net Support

     

  • Anonymous
    2012-10-16

    Originally by: maddes

    Thanks for the quick info.
    Changed my post-commit-user file accordingly.

    Will track this ticket for "check-case-insensitive.py" and "check-mime-type.pl" to arrive. The first one is critical for projects with mixed development platforms (Windows/*nixes).

    Normally these are available in /usr/share/subversion/hook-scripts/ and must be executable.

    Maddes

     

  • Anonymous
    2012-10-16

    Originally by: maddes

    The scripts are available on the code server (found via pre-commit hook) but not at their usual place.
    Normally these are in /usr/share/subversion/hook-scripts/ and must be executable.

    -rwxr-xr-x 1 root root 6527 Oct  5  2007   /usr/share/doc/subversion-1.6.17/contrib/hook-scripts/check-mime-type.pl  
    -rwxr-xr-x 1 root root 6527 Jun  1  2011   /usr/lib64/subversion/contrib/hook-scripts/check-mime-type.pl  
    -rwxr-xr-x 1 root root 4794 Oct 17  2007   /usr/share/doc/subversion-1.6.17/contrib/hook-scripts/case-insensitive.py  
    -rwxr-xr-x 1 root root 4794 Jun  1  2011   /usr/lib64/subversion/contrib/hook-scripts/case-insensitive.py
    

    Maddes

     

  • Anonymous
    2012-10-16

    Originally by: maddes

    /usr/lib64/subversion/contrib/hook-scripts/check-mime-type.pl:
    Is working for me. Still unusual place for hook script.

    /usr/lib64/subversion/contrib/hook-scripts/case-insensitive.py:
    The Python script is not working.

    Traceback (most recent call last):  
      File "/usr/lib64/subversion/contrib/hook-scripts/case-insensitive.py", line  
     64, in ?  
        from svn import repos, fs  
    ImportError: No module named svn
    
     
  • Cory Johns
    Cory Johns
    2012-10-16

    • status: open --> in-progress
    • assigned_to: Cory Johns
     
  • Cory Johns
    Cory Johns
    2012-10-17

    • status: in-progress --> code-review
    • qa: Dave Brondsema⚛
     
  • Cory Johns
    Cory Johns
    2012-10-17

    allura:cj/5076

     
  • Dave Brondsema
    Dave Brondsema
    2012-10-18

    Can we put this in forge-classic? Monkey-patching even, if needed. This is very SF-specific.

    We need to handle these two hooks also:

    /var/local/mastertree/host/sfp-svn/hook-scripts/check-case-insensitive.py "$1" "$2"
    /var/local/mastertree/host/sfp-svn/hook-scripts/check-mime-type.pl "$1" "$2"
    

    There's an extra slash in --revisionURI=http://localhost//p...

    We should check with siteops about what environment these run in so that we can confirm the paths (e.g. the new paths in this code branch aren't present for me on the hosts I expected, but neither was /usr/bin/svnnotify so I'm clearly looking on the wrong host) and confirm functionality (e.g. ImportError mentioned above).

    Should we do manual fixups for the existing repos? Not sure how many there are, but that might be easier than scripting it. Either way, we should do that in 'validation' stage or make a separate ticket.

     
  • Dave Brondsema
    Dave Brondsema
    2012-10-18

    • status: code-review --> in-progress
     
  • Dave Brondsema
    Dave Brondsema
    2012-10-19

    • labels: support, p2 --> support, p2, migration
     

  • Anonymous
    2012-10-25

    Originally by: peterbecker

    I'm experiencing this problem with the 'toscanaj' project and the check-case-insensitive.py script. How do I work around this issue? I'm reasonably happy to drop the script completely, but I don't know how.

    Here's the exact error if needed:

      svn: E165001: Commit failed (details follow):
      svn: E165001: Commit blocked by pre-commit hook (exit code 1) with output:
      /svn/p/toscanaj/code/hooks/pre-commit: line 3: /var/local/mastertree/host/sfp-svn/hook-scripts/check-case-insensitive.py: No such file or directory
    
     

  • Anonymous
    2012-10-25

    Originally by: *anonymous

    Sorry -- I just realised the issue linked in the description answers my question. Fixed.

     
  • Cory Johns
    Cory Johns
    2012-10-25

    I believe you should be able to use the shell service to access the hooks directly at /home/svn/p/<project>/<mount>/hooks.

    Unfortunately, we do not yet have the hooks installed globally, but you should be able to put them in the hooks directory and call them from there.

     
  • Cory Johns
    Cory Johns
    2012-10-25

    I added the info to this thread as well in my reply above, as well as noting that, in addition to simply commenting out the offending hook call, you can also install it directly in your project's hook directory until we resolve the issue globally.

     
  • David Burley
    David Burley
    2012-10-26

    /var/local/mastertree/host/sfp-svn/hook-scripts/sf-svn-stats-hook.py $*
    /var/local/mastertree/host/sfp-svn/hook-scripts/keepsake -p "$1"

    ^-- those should be removed from the project's hooks anyways, they are classic forge specific.

     
  • David Burley
    David Burley
    2012-10-26

    svnnotify is present in /usr/bin/svnnotify, so the path just should need changed to reference it.

     
  • David Burley
    David Burley
    2012-10-26

    I have created all the scripts in the allura SVN environment, in the same place they were before (well, symlinked to the new location). I replaced the non-needed scripts with empty ones that just return to clear up errors.

     
  • David Burley
    David Burley
    2012-10-26

    note: I haven't tested them, however...

     
  • Cory Johns
    Cory Johns
    2012-10-26

    allura:cj/5076
    forge-classic:cj/5076

    Moved hook copying / processing code to forge-classic, added missing munging, fixed a couple of small issues, and generally cleaned it up.

    The production setup is already in place, and the default value for the new config option is correct, so this should be GTG out of the box.

     
  • Cory Johns
    Cory Johns
    2012-10-26

    • status: in-progress --> code-review
     
  • Dave Brondsema
    Dave Brondsema
    2012-10-29

    Code and tests are looking good, but got an error when migrating a test project. As you can see in the pdb shell, the src & dst args have the right values. Also /git/p/testscm/part1.git/hooks/post-receive.sample actually exists when I check after this error. Before running the migration, I did run rm -rf /{hg,svn,git}/p/testscm to make sure I had a fresh clean start.

      File "/var/local/forge-classic/scripts/migrate_sfx_project.py", line 18, in main
        migrator.migrate()
      File "/home/dbrondsema/dbrondsema-7024/forge-classic/sfx/lib/migrate.py", line 220, in migrate
        self.migrate_repos()
      File "/home/dbrondsema/dbrondsema-7024/forge-classic/sfx/lib/migrate.py", line 299, in migrate_repos
        self.tool_position += SCMMigrator(self.allura_project).migrate(repos, self.tool_position)
      File "/home/dbrondsema/dbrondsema-7024/forge-classic/sfx/lib/migrate.py", line 995, in migrate
        self._post_process_repo(repo, app)
      File "/home/dbrondsema/dbrondsema-7024/forge-classic/sfx/lib/migrate.py", line 1134, in _post_process_repo
        dispatch[repo['tool_name']](repo, app)
      File "/home/dbrondsema/dbrondsema-7024/forge-classic/sfx/lib/migrate.py", line 1144, in _post_process_git_repo
        shutil.copy2(hook, target)
      File "/usr/lib64/python2.7/shutil.py", line 128, in copy2
        copyfile(src, dst)
      File "/usr/lib64/python2.7/shutil.py", line 83, in copyfile
        with open(dst, 'wb') as fdst:
    IOError: [Errno 2] No such file or directory: u'/git/p/testscm/part1.git/hooks/post-receive.sample'
    > /usr/lib64/python2.7/shutil.py(83)copyfile()
    -> with open(dst, 'wb') as fdst:
    (Pdb) a
    src = /nfs/classic/sf-git/t/te/testscm/part1/hooks/post-receive.sample
    dst = /git/p/testscm/part1.git/hooks/post-receive.sample
    
     
  • Dave Brondsema
    Dave Brondsema
    2012-10-29

    • status: code-review --> in-progress
     
1 2 > >> (Page 1 of 2)