[Devel] Memory usage

[Mary Gardiner]

I think I may have mentioned this here before, but just a reminder...
thanks to this code in planet/__init__.py:

329         info = feedparser.parse(self.url,
330                                 etag=self.url_etag, modified=self.url_modified,
331                                 agent=self._planet.user_agent)

(which causes the entire feed to be read into memory)

446             # Create the item if necessary and update
447             if self.has_item(entry_id):
448                 item = self._items[entry_id]
449             else:
450                 item = NewsItem(self, entry_id)
451                 self._items[entry_id] = item

(which is part of copying all seemingly recent entries into an attribute
of self -- ie something which unless removed will have the lifetime of
the channel object)

and these in planet.py:

137     for feed_url in config.sections():

141         # Create a channel, configure it and subscribe it
142         channel = planet.Channel(my_planet, feed_url)

146         my_planet.subscribe(channel)

planet uses up an enormous amount of memory -- essentially it builds up
all new items in memory and then dumps them as a lump to the template
manager. For some planets, I've seen it use 40MB of memory, which is
fine on a grunty server, less good for anyone (like me) running on a UML
with very restricted memory.

This is somewhere on my own to-hack list, but bringing it to folks'
attention just in case anyone feels inspired.

-Mary