#7890 Neighborhood cache preventing saving admin changes

v1.3.1
closed
General
2015-06-29
2015-06-04
No

If you have neighborhood.cache.duration set in the .ini file then when you go to /p/_admin/overview (or any neighborhood) your changes will not persist. They may appear to show but if you refresh the page they won't be there.

We don't need to worry about clearing/expiring the cache (especially since multi-process multi-server deployments will all have their own memory caches) but we should actually save the changes so that when the cache naturally expires the new values are used.

Related

Tickets: #7934

Discussion

  • Igor Bondarenko

    Igor Bondarenko - 2015-06-08
    • labels: --> 42cc
    • status: open --> in-progress
    • assigned_to: Igor Bondarenko
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-06-09

    I can't reproduce this. Maybe there are some other deployment related options which affect this? Can you confirm it still persists?

     
  • Dave Brondsema

    Dave Brondsema - 2015-06-09

    I had trouble reproducing this too. I first saw it several months ago and couldn't reproduce it locally. Then I encountered it again a few days ago and I was able to reproduce it locally in an environment where neighborhood.cache.duration was set.

    Just now I was able to to reproduce in our production environment on a high-traffic neighborhood. But saving did work ok in a neighborhood that is essentially never used (and thus likely not cached in some/all the procs).

    If needed perhaps we could just add some more logging to get more info about what's happening. That is, if we can think of good logging to add :)

     
  • Igor Bondarenko

    Igor Bondarenko - 2015-06-15

    Ok, I've got some progress.

    I was able to reproduce this locally while running this script to simulate concurrent access to the same neighborhood (neighborhood.cache.duration set to 10).

    My current understanding of the problem is:

    1. Some thread misses the cache and executes NeighborhoodCache._lookup, which populates cache
    2. In the "update" thread neighborhood controller gets instantiated with a neighborhood object from the cache
    3. Since it didn't do any neighborhood lookups in the mongo, Ming's session does not contain the neighborhood object for the "update" thread
    4. When update controller exits, Ming session is flushed to disk, but neighborhood object is not, since Ming's session for current thread does not contain it.
    5. When cache expires, old version of neighborhood object is fetched from the mongo and updated values disappear.

    Some ad-hoc logging I have added confirms this.

    I was not able to fix this yet, though. If someone knows a way to add object to current thread's ming session, or to flush ming object which does not belong to a session (i.e. session(obj) == None), please let me know :)

    On 42c side, I've closed #794 and created #801 for fixing this, since I've underestimated this task initially.

     
  • Igor Bondarenko

    Igor Bondarenko - 2015-06-15
    • status: in-progress --> review
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-06-15

    Closed #801. ib/7890

    A simplest fix actually worked. I've just added a line to get neighborhood from mongo to the beginning of "update" controller.

    Notes for QA:

    • get the script
    • run it as python ./scripts/get_neighborhood.py http://<your_host>/p/wiki/Home>
    • set neighborhood.cache.duration to something small (5 - 10 seconds)

     
    On master:

    • go to /p/_admin/overview, change something and submit the form
    • make sure you see new values
    • wait for cache to expire and reload the page, make sure you see old values

     
    Checkout the branch and repeat the above. After form submission you will not see new values, but after cache expires you should see them.

     

    Last edit: Igor Bondarenko 2015-06-15
  • Dave Brondsema

    Dave Brondsema - 2015-06-16
    • labels: 42cc --> 42cc, sf-current, sf-2
    • status: review --> closed
    • Reviewer: Dave Brondsema
     
  • Dave Brondsema

    Dave Brondsema - 2015-06-16

    Nice work! I like it when the fix is simple too :)

     
  • Dave Brondsema

    Dave Brondsema - 2015-06-29
    • labels: 42cc, sf-current, sf-2 --> 42cc, sf-2
     
  • Dave Brondsema

    Dave Brondsema - 2015-08-10
    • Milestone: unreleased --> v1.3.1
     

Log in to post a comment.