#8314 @memoize on methods should still allow garbage collection

v1.12.0
closed
None
General
nobody
2019-10-04
2019-07-03
No

Our @memoize works pretty similarly to @lru_cache and doesn't do what you might expect on a method (as opposed to a simple function). It creates a global class-level cache with self as one of the args and thus the self object is referenced forever and cannot be garbage collected. https://bugs.python.org/issue19859 describes the situation

Discussion

  • Dave Brondsema

    Dave Brondsema - 2019-07-03
    • status: in-progress --> review
     
  • Dave Brondsema

    Dave Brondsema - 2019-07-03

    db/8314 (branched off of [#8311] since it adds a requirement)

    I looked into many off-the-shelf existing libraries but none worked:

    • @methodtools.lru_cache() is supposed to be exactly for this case of a method, but in my testing holds global cache still. Also has 3 more dependencies it pulls in.
    • @cachedtools.cached(cache={}) holds global cache
    • @cachedtools.cached(cache=WeakValueDictionary()) errors
    • @cachedtools.cachedmethod(lambda self: self.cache, key=partial(hashkey, 'our_method_name')) requires a self.cache (or creating one on the fly) and knowing the method name to make a properly scoped key
    • @pylru.lrudecorator() requires size param, holds global cache
    • @ring.lru() ran into UnicodeEncodeError
    • @memoized_method from https://stackoverflow.com/a/33672499/ gets interference with Ming since it rewrites the attr

    Drop-ins for py3 functools.lru_cache (which wouldn't work either):

    • @cachedtools.lru_cache() errors
    • @repoze.lru.lru_cache() requires size param, holds global cache
    • @functools32.lru_cache() holds global cache
    • @backports.functools_lru_cache holds global cache
     

    Related

    Tickets: #8311

  • Kenton Taylor

    Kenton Taylor - 2019-07-15
    • status: review --> closed
     
  • Dave Brondsema

    Dave Brondsema - 2019-10-04
    • Milestone: unreleased --> v1.12.0
     

Log in to post a comment.