Comments on: Why can’t you pickle generators in Python? A pattern for saving training state http://metaoptimize.com/blog/2009/12/22/why-cant-you-pickle-generators-in-python-workaround-pattern-for-saving-training-state/ building machine learning and natural language processing tools Mon, 30 Jan 2012 04:05:03 +0000 http://wordpress.org/?v=abc hourly 1 By: Joseph Turian http://metaoptimize.com/blog/2009/12/22/why-cant-you-pickle-generators-in-python-workaround-pattern-for-saving-training-state/comment-page-1/#comment-956 Joseph Turian Fri, 01 Oct 2010 00:39:28 +0000 http://blog.metaoptimize.com/?p=72#comment-956 <span class="topsy_trackback_comment"><span class="topsy_twitter_username"><span class="topsy_trackback_content">Why can't you pickle generators in Python? A pattern for saving training state http://bit.ly/dvUzTP</span></span> Why can’t you pickle generators in Python? A pattern for saving training state http://bit.ly/dvUzTP

]]> By: Why you cannot pickle generator http://metaoptimize.com/blog/2009/12/22/why-cant-you-pickle-generators-in-python-workaround-pattern-for-saving-training-state/comment-page-1/#comment-158 Why you cannot pickle generator Tue, 29 Dec 2009 23:07:53 +0000 http://blog.metaoptimize.com/?p=72#comment-158 [...] Turian wrote a post about regarding pickling generator on his blog. In his post, he says: However, generators become problematic when you want to persist [...] […] Turian wrote a post about regarding pickling generator on his blog. In his post, he says: However, generators become problematic when you want to persist […]

]]> By: Joseph Turian http://metaoptimize.com/blog/2009/12/22/why-cant-you-pickle-generators-in-python-workaround-pattern-for-saving-training-state/comment-page-1/#comment-145 Joseph Turian Wed, 23 Dec 2009 00:10:06 +0000 http://blog.metaoptimize.com/?p=72#comment-145 Response to criticism on <a href="http://news.ycombinator.com/item?id=1010533" rel="nofollow">hackernews</a>: <i>(1) You can only pickle generators that generate the same sequence every time they are restarted.</i> I don't know how you can persist state if you do not make this assumption. <i>(2) All the work the generator did prior to pickling must be performed again on unpickling.<i> Something faster would be to use file.tell() to get the state and file.seek() to set the state. Since the "unpickling" is not a bottleneck, I didn't optimize this. Response to criticism on hackernews:

(1) You can only pickle generators that generate the same sequence every time they are restarted.

I don’t know how you can persist state if you do not make this assumption.

(2) All the work the generator did prior to pickling must be performed again on unpickling.

Something faster would be to use file.tell() to get the state and file.seek() to set the state. Since the “unpickling” is not a bottleneck, I didn’t optimize this.

]]> By: Richard Tew http://metaoptimize.com/blog/2009/12/22/why-cant-you-pickle-generators-in-python-workaround-pattern-for-saving-training-state/comment-page-1/#comment-142 Richard Tew Tue, 22 Dec 2009 22:19:46 +0000 http://blog.metaoptimize.com/?p=72#comment-142 You can do this with Stackless, of course. http://www.disinterest.org/resource/stackless/2.6.4-docs-html/library/stackless/pickling.html You can do this with Stackless, of course.

http://www.disinterest.org/resource/stackless/2.6.4-docs-html/library/stackless/pickling.html

]]>