#4542 Implement webhooks

v1.3.0
closed
General
2015-08-20
2012-07-13
No

We should have webhooks, compatible with github & bitbucket. I don't see a spec for SCM webhooks anywhere.

Related

Tickets: #7829

Discussion

1 2 > >> (Page 1 of 2)
  • Anonymous - 2014-05-29

    Originally by: *anonymous

    Please consider that even something very minimal — containing just the repo URL, only on push events — would be enough to allow continuous integration services like Travis CI or Drone.io to work with sourceforge.

     
  • Dave Brondsema

    Dave Brondsema - 2014-08-19

    We'll need to include some controls including:

    • audit log of webhooks added
    • server-side logging of each time a hook fires
    • thresholds (e.g. max hooks per project/repo) and rate limits
     
  • Anonymous - 2014-10-06

    Originally by: nathanaeljones

    This enables integration with CI services like drone.io and AppVeyor. Without support for webhooks, it's a bit more difficult to follow modern software development structure.

     
  • Dave Brondsema

    Dave Brondsema - 2015-01-12
    • labels: --> sf-current, sf-8
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-01-14
    • labels: sf-current, sf-8 --> sf-current, sf-8, 42cc
    • status: open --> in-progress
    • assigned_to: Igor Bondarenko
     
  • Heith Seewald

    Heith Seewald - 2015-01-16

    What do you think about this webhooks package from pydanny?

     
    • Igor Bondarenko

      Igor Bondarenko - 2015-01-20

      It's promising, but I don't think it is very useful for our use case in the state it is now. I skimmed through code and it seems like this package focuses more on providing different kind of senders (sync, async via redis, etc), which we probably won't need, since we using our own async jobs system. At the same time it misses some essential parts of functionality, like hmac signature of a payload.

       
  • Igor Bondarenko

    Igor Bondarenko - 2015-01-30

    Closed #714. ib/4542 (It's in my fork https://forge-allura.apache.org/u/jetmind/allura/ci/master/tree/). Can't push it to main repo somehow:

    $ git push origin t714_webhooks:ib/4542    
    Counting objects: 258, done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (83/83), done.
    Writing objects: 100% (87/87), 25.78 KiB | 0 bytes/s, done.
    Total 87 (delta 66), reused 0 (delta 0)
    error: RPC failed; result=56, HTTP code = 0
    fatal: The remote end hung up unexpectedly
    fatal: The remote end hung up unexpectedly
    Everything up-to-date
    
    To QA:

    To test without any external service I've used http://httpbin.org

    http://httpbin.org/post will simply return your request's POST data, headers, etc, so you can see what payload actually sent.

    But test with something like Travis CI also :)

    Also:

    • make sure it works for git, svn, mercurial
    • test with big pushes (hundreds of commits?)
    Some optional areas for future improvements (inspired by GitHub mostly):
    • thresholds (already have a ticket for it on 42cc side)
    • send "ping" event to hook url after configuration, like github does. It's useful to ensure that webhook configured properly, without triggering event by repo push
    • add "ignore insecure ssl" option, it's useful for testing
    • support payload with application/x-www-form-urlencoded mime type
    • show response/error in UI for webhooks fired (last 5 maybe?)
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-01-30
    • status: in-progress --> review
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-02-02

    I'm finishing #715 now, so if you wanna review this you better wait an ~hour to review all at once :)

     
  • Igor Bondarenko

    Igor Bondarenko - 2015-02-02

    Closed #715. Latest code in ib/4542 (main repo, it works now)

     
  • Dave Brondsema

    Dave Brondsema - 2015-02-04

    Any ideas of services to test against? I haven't had much luck yet

    • HipChat
      • No generic commit webhook. Can't re-use GitHub or Bitbucket integrations since setups requires authenticating to those sites, presumably to configure the hook for you
      • We can piggy-back on Google Code v1 api but that doesn't come through good: "Dave Brondsema committed r of : "
    • Slack
      • Using their BitBucket integration: 500 error, body "No payload received from bitbucket"
      • Using their GitHub integration with manual configuration: our webhook submits successfully, but nothing shows up.
    • Travis-CI
      • Only works with GitHub

    We may need to more closely emulate the GitHub format of a webhook, so that Allura webhooks can work with existing services more easily.

     
  • Dave Brondsema

    Dave Brondsema - 2015-02-09
    • status: review --> in-progress
    • Reviewer: Dave Brondsema
     
  • Dave Brondsema

    Dave Brondsema - 2015-02-09

    Feedback so far. More to come, probably:

    • I think we should make the payload format more like bitbucket and github. In my testing so far I tried to piggy-back on 3rd parties' existing github or bitbucket integration but couldn't get it to work. A big differnce is in using "commits" instead of "revisions" (which Google Code and this branch so far use). I'd say match as much of the github payload as we can. There's a ton in there though that we won't be able to match, but the more the better. Bitbucket's payload is a bit more reasonably small. The closer we match, the more likely it is that Allura webhooks can work out-of-the-box with more 3rd party services.
    • A big feature like this certainly needs docs. I made a separate ticket for that [#7829]
    • I haven't fully evaluated the way hooks are set up (entry points) vs having them be per-tool. My initial thought is that this doesn't need to be on admin sidebar, but could be less prominent in the per-tool admin links. Again, not fully thought through.. but I am thinking that when a user creates a webhook, they have to associate it with a specific tool anyway, so why not have it be a per-tool setting. Different tools could still use a common controller/template for that.
    • It's weird to me to have the edit link of a webhook be a (different) URL. I think a link that just says "edit" would be better.

    Need a tweak like this for better error handling:

    --- Allura/allura/webhooks.py
    +++ Allura/allura/webhooks.py
    @@ -256,11 +256,12 @@ def log_msg(self, msg, response=None):
                 self.webhook.type,
                 self.webhook.hook_url,
                 self.webhook.app_config.url())
    -        if response:
    -            message = '{} {} {}'.format(
    +        if response is not None:
    +            message = '{} {} {} {}'.format(
                     message,
                     response.status_code,
    -                response.reason)
    +                response.text,
    +                response.headers)
             return message
    
         def send(self):
    
     

    Related

    Tickets: #7829


    Last edit: Dave Brondsema 2015-02-09
    • Igor Bondarenko

      Igor Bondarenko - 2015-02-10

      Created #723 to change payload format and tweak error handling for now.

      I'm not sure what you mean by edit link of a webhook, could you elaborate on that?

      Regarding entry points vs. per-tool my thought process was this:

      • it seems natural to have one place where you can see and configure all available types of hooks
      • it should be extensible by third-party apps. In theory with entry points you can create a new type of webhook and assign it to existing app without modifying the app itself.

      Don't sure if it's a big deal, though

       

      Last edit: Igor Bondarenko 2015-02-10
      • Dave Brondsema

        Dave Brondsema - 2015-02-10

        Here's a screenshot of the webhook admin page. To edit one, you have to click on the webhook destination URL.

         
      • Dave Brondsema

        Dave Brondsema - 2015-02-10

        I think you can keep the entry points and Webhook model exactly how it is now. I am thinking of just changing the admin form so that instead of one page to manage all your webhooks, you go to a Webhook admin form per app. You would still be able to have 3rd-party webhooks associated with any app.

        Changing the admin pages as I described I think would actual be more natural, at least for now. We only have one type of webhook currently so a page to manage all of them isn't too useful. (We could add/restore it later). Since a user creates a webhook and associates it with a specific app, I think a per-app configuration page would be fine. Lastly, while webhooks are very cool I don't think they warrant the prominence of a left-nav menu item on the admin pages. A little more "hidden" under the tool config area would be just fine.

         
  • Dave Brondsema

    Dave Brondsema - 2015-02-10

    MingOneOf seems like it it might be duplicative of the OneOf class in ming.schema Perhaps MingOneOf could be removed. Or if it has some unique functionality, could you add a docstring to explain that?

    If a rate limit is configured and reached, it just skips the event and logs it in the server logs. That's not a good UX for the project developers, but I'm not sure of a better option without adding a lot of complexity (like allowing a several fast events in a row, but if they persist for a long time then block; or delay and retry but somehow avoid backlogging taskd with tons of events)

    "You have exceeded the maximum number of projects" should say "... of webhooks"

     
  • Igor Bondarenko

    Igor Bondarenko - 2015-02-11

    Closed #723. Updated ib/4542 with the following:

    • Tweak error message
    • Change repo-push hook payload

    .

    Created #726 for the following:

    • Add separate 'edit' link for webhook edit
    • Fix table width when webhook link is long
    • Move 'Webhooks' from sidebar to app admin
    • Remove or document MingOneOf
    • Fix message: "You have exceeded the maximum number of projects" should say "... of webhooks"

    .

    Tested with Slack (GitHub with manual configuration), Flowdock (the same) and HipChat (Google Code API hook url).

    Slack works.

    Flowdock: submit successful, but nothing shows up. It requires to configure GitHub hook as "send everything", so I guess it checks X-GitHub-Event header to distinguish between events, and we do not provide that. And if I add this header, webhook submit fails with:

    09:57:21,258 ERROR [taskd:allura.webhooks.send_webhook:54db278004687d34e52600ef:allura.webhooks] Webhook send error: repo-push https://www.flowdock.com/jari/github/messages/bd36bad4407ee69c9145df34ce9c5856?tag=9af05261-97dd-47d3-8303-5f2721562ac9&jari=778fcd79e9ffdc6594d8331a7ea85189 /p/test/git/ 500 {"message":"Internal server error"} CaseInsensitiveDict({'status': '500 Internal Server Error', 'x-request-id': '160bbd50-5aca-4d9a-acc9-1013d17b1724', 'x-xss-protection': '1; mode=block', 'x-content-type-options': 'nosniff', 'content-encoding': 'gzip', 'x-server-id': '45c836f72727be03bd8c2ae92cedbf6a75a556bc', 'vary': 'Accept-Encoding', 'content-length': '55', 'x-runtime': '0.039335', 'date': 'Wed, 11 Feb 2015 09:57:21 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=utf-8'})
    

    .

    HipChat: webhook submit fails due to incompatible payload format:

    10:16:08,659 ERROR [taskd:allura.webhooks.send_webhook:54db2be604687d449b51b8b3:allura.webhooks] Webhook send error: repo-push https://api.hipchat.com/v1/webhooks/googlecode/?auth_token=155a45b4e6176b2c06e51c333b4658&room_id=1215273 /p/test/git/ 400 {"error":{"code":400,"type":"Bad Request","message":"Bad POST data format"}} CaseInsensitiveDict({'x-ratelimit-remaining': '99', 'content-length': '76', 'server': 'nginx', 'x-hipchat-jobs': 'Want to help make HipChat better? Apply at hipchat.com/jobs and mention this header.', 'connection': 'keep-alive', 'x-ratelimit-limit': '100', 'date': 'Wed, 11 Feb 2015 10:16:08 GMT', 'content-type': 'application/json', 'x-ratelimit-reset': '1423649820'})
    
     

    Last edit: Igor Bondarenko 2015-02-11
  • Igor Bondarenko

    Igor Bondarenko - 2015-02-11
    • status: in-progress --> review
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-02-11

    Closed #726. Updated ib/4542.

    Moving 'webhooks' link to app admin menu has allowed to remove quite a bit of not very straightforward code, including MingOneOf. :)

     
  • Dave Brondsema

    Dave Brondsema - 2015-02-12

    Nice that the admin form change made for cleaner code too :)

    Yay for creating the admin_url helper. There's a lot of places that would be a little cleaner with it. Should Webhook.url() use it too?

    For the payload:

    • I think we should include the ref like "ref":"refs/heads/master" Slack for example renders [repo-name:branch] at the beginning of the message, and it currently comes through like [Code:] Not sure what value to put there when it's SVN, maybe don't include it for SVN.
    • Again with slack, if multiple commits are pushed it shows <|3 new commits> compared to a github commit which is 3 new commits linked to the "compare" payload URL. Including that would be nice. Might as well include "before" and "after" hash too.

    The _diff helper is nice. I might be a bit biased (since I wrote it) but give datadiff.tools.assert_equals a try. It'll do a diff of the data structure, aware of the semantics of dicts, lists, and how far down they are nested.

    For SVN, can you change the "id" attribute to have a value like "r1" instead of "54dd2fc2d8be35744298ee3d:1".

    Mercurial appears to work well (same payload recommendations as above for git).

     
  • Dave Brondsema

    Dave Brondsema - 2015-02-12
    • status: review --> in-progress
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-02-16
    • status: in-progress --> review
     
1 2 > >> (Page 1 of 2)

Log in to post a comment.