Apache Allura™

Google code supports it too. Their docs are at http://code.google.com/p/support/wiki/PostCommitWebHooks

Github: https://developer.github.com/v3/activity/events/types/#pushevent

Bitbucket: https://confluence.atlassian.com/display/BITBUCKET/POST+hook+management

Originally by: *anonymous

Please consider that even something very minimal — containing just the repo URL, only on push events — would be enough to allow continuous integration services like Travis CI or Drone.io to work with sourceforge.

We'll need to include some controls including:

• audit log of webhooks added
• server-side logging of each time a hook fires
• thresholds (e.g. max hooks per project/repo) and rate limits

Originally by: nathanaeljones

This enables integration with CI services like drone.io and AppVeyor. Without support for webhooks, it's a bit more difficult to follow modern software development structure.

What do you think about this webhooks package from pydanny?

Closed #714. ib/4542 (It's in my fork https://forge-allura.apache.org/u/jetmind/allura/ci/master/tree/). Can't push it to main repo somehow:

$ git push origin t714_webhooks:ib/4542    
Counting objects: 258, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (83/83), done.
Writing objects: 100% (87/87), 25.78 KiB | 0 bytes/s, done.
Total 87 (delta 66), reused 0 (delta 0)
error: RPC failed; result=56, HTTP code = 0
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

To QA:

To test without any external service I've used http://httpbin.org

• add http://httpbin.org/post as hook url
• add log.info(r.content) to send_webhook task, after requests.post call

http://httpbin.org/post will simply return your request's POST data, headers, etc, so you can see what payload actually sent.

But test with something like Travis CI also :)

Also:

• make sure it works for git, svn, mercurial
• test with big pushes (hundreds of commits?)

Some optional areas for future improvements (inspired by GitHub mostly):

• thresholds (already have a ticket for it on 42cc side)
• send "ping" event to hook url after configuration, like github does. It's useful to ensure that webhook configured properly, without triggering event by repo push
• add "ignore insecure ssl" option, it's useful for testing
• support payload with application/x-www-form-urlencoded mime type
• show response/error in UI for webhooks fired (last 5 maybe?)

I'm finishing #715 now, so if you wanna review this you better wait an ~hour to review all at once :)

Closed #715. Latest code in ib/4542 (main repo, it works now)

Any ideas of services to test against? I haven't had much luck yet

• HipChat
  • No generic commit webhook. Can't re-use GitHub or Bitbucket integrations since setups requires authenticating to those sites, presumably to configure the hook for you
  • We can piggy-back on Google Code v1 api but that doesn't come through good: "Dave Brondsema committed r of : "
• Slack
  • Using their BitBucket integration: 500 error, body "No payload received from bitbucket"
  • Using their GitHub integration with manual configuration: our webhook submits successfully, but nothing shows up.
• Travis-CI
  • Only works with GitHub

We may need to more closely emulate the GitHub format of a webhook, so that Allura webhooks can work with existing services more easily.

Feedback so far. More to come, probably:

• I think we should make the payload format more like bitbucket and github. In my testing so far I tried to piggy-back on 3rd parties' existing github or bitbucket integration but couldn't get it to work. A big differnce is in using "commits" instead of "revisions" (which Google Code and this branch so far use). I'd say match as much of the github payload as we can. There's a ton in there though that we won't be able to match, but the more the better. Bitbucket's payload is a bit more reasonably small. The closer we match, the more likely it is that Allura webhooks can work out-of-the-box with more 3rd party services.
• A big feature like this certainly needs docs. I made a separate ticket for that [#7829]
• I haven't fully evaluated the way hooks are set up (entry points) vs having them be per-tool. My initial thought is that this doesn't need to be on admin sidebar, but could be less prominent in the per-tool admin links. Again, not fully thought through.. but I am thinking that when a user creates a webhook, they have to associate it with a specific tool anyway, so why not have it be a per-tool setting. Different tools could still use a common controller/template for that.
• It's weird to me to have the edit link of a webhook be a (different) URL. I think a link that just says "edit" would be better.

Need a tweak like this for better error handling:

--- Allura/allura/webhooks.py
+++ Allura/allura/webhooks.py
@@ -256,11 +256,12 @@ def log_msg(self, msg, response=None):
             self.webhook.type,
             self.webhook.hook_url,
             self.webhook.app_config.url())
-        if response:
-            message = '{} {} {}'.format(
+        if response is not None:
+            message = '{} {} {} {}'.format(
                 message,
                 response.status_code,
-                response.reason)
+                response.text,
+                response.headers)
         return message

     def send(self):

MingOneOf seems like it it might be duplicative of the OneOf class in ming.schema Perhaps MingOneOf could be removed. Or if it has some unique functionality, could you add a docstring to explain that?

If a rate limit is configured and reached, it just skips the event and logs it in the server logs. That's not a good UX for the project developers, but I'm not sure of a better option without adding a lot of complexity (like allowing a several fast events in a row, but if they persist for a long time then block; or delay and retry but somehow avoid backlogging taskd with tons of events)

"You have exceeded the maximum number of projects" should say "... of webhooks"

Closed #723. Updated ib/4542 with the following:

• Tweak error message
• Change repo-push hook payload

.

Created #726 for the following:

• Add separate 'edit' link for webhook edit
• Fix table width when webhook link is long
• Move 'Webhooks' from sidebar to app admin
• Remove or document MingOneOf
• Fix message: "You have exceeded the maximum number of projects" should say "... of webhooks"

.

Tested with Slack (GitHub with manual configuration), Flowdock (the same) and HipChat (Google Code API hook url).

Slack works.

Flowdock: submit successful, but nothing shows up. It requires to configure GitHub hook as "send everything", so I guess it checks X-GitHub-Event header to distinguish between events, and we do not provide that. And if I add this header, webhook submit fails with:

09:57:21,258 ERROR [taskd:allura.webhooks.send_webhook:54db278004687d34e52600ef:allura.webhooks] Webhook send error: repo-push https://www.flowdock.com/jari/github/messages/bd36bad4407ee69c9145df34ce9c5856?tag=9af05261-97dd-47d3-8303-5f2721562ac9&jari=778fcd79e9ffdc6594d8331a7ea85189 /p/test/git/ 500 {"message":"Internal server error"} CaseInsensitiveDict({'status': '500 Internal Server Error', 'x-request-id': '160bbd50-5aca-4d9a-acc9-1013d17b1724', 'x-xss-protection': '1; mode=block', 'x-content-type-options': 'nosniff', 'content-encoding': 'gzip', 'x-server-id': '45c836f72727be03bd8c2ae92cedbf6a75a556bc', 'vary': 'Accept-Encoding', 'content-length': '55', 'x-runtime': '0.039335', 'date': 'Wed, 11 Feb 2015 09:57:21 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=utf-8'})

.

HipChat: webhook submit fails due to incompatible payload format:

10:16:08,659 ERROR [taskd:allura.webhooks.send_webhook:54db2be604687d449b51b8b3:allura.webhooks] Webhook send error: repo-push https://api.hipchat.com/v1/webhooks/googlecode/?auth_token=155a45b4e6176b2c06e51c333b4658&room_id=1215273 /p/test/git/ 400 {"error":{"code":400,"type":"Bad Request","message":"Bad POST data format"}} CaseInsensitiveDict({'x-ratelimit-remaining': '99', 'content-length': '76', 'server': 'nginx', 'x-hipchat-jobs': 'Want to help make HipChat better? Apply at hipchat.com/jobs and mention this header.', 'connection': 'keep-alive', 'x-ratelimit-limit': '100', 'date': 'Wed, 11 Feb 2015 10:16:08 GMT', 'content-type': 'application/json', 'x-ratelimit-reset': '1423649820'})

Closed #726. Updated ib/4542.

Moving 'webhooks' link to app admin menu has allowed to remove quite a bit of not very straightforward code, including MingOneOf. :)

Nice that the admin form change made for cleaner code too :)

Yay for creating the admin_url helper. There's a lot of places that would be a little cleaner with it. Should Webhook.url() use it too?

For the payload:

• I think we should include the ref like "ref":"refs/heads/master" Slack for example renders [repo-name:branch] at the beginning of the message, and it currently comes through like [Code:] Not sure what value to put there when it's SVN, maybe don't include it for SVN.
• Again with slack, if multiple commits are pushed it shows <|3 new commits> compared to a github commit which is 3 new commits linked to the "compare" payload URL. Including that would be nice. Might as well include "before" and "after" hash too.

The _diff helper is nice. I might be a bit biased (since I wrote it) but give datadiff.tools.assert_equals a try. It'll do a diff of the data structure, aware of the semantics of dicts, lists, and how far down they are nested.

For SVN, can you change the "id" attribute to have a value like "r1" instead of "54dd2fc2d8be35744298ee3d:1".

Mercurial appears to work well (same payload recommendations as above for git).