Venus: My Homepage has suddently stopped updating

[Sam Ruby]

Gordon Hodgson wrote:
> Ok, so I found taking the avatar thing out of the spidered files seemed 
> to stop the default avatar from working for some (but not all) of my 
> feeds... Don;t know why.
> 
> Am I correct in thinking the log levels make a hierarchy like this
> DEBUG
> INFO
> WARNING
> ERROR
> CRITICAL
> 
> so DEBUG gives all message levels downwards, INFO gives all except 
> DEBUG, and so on?

Yes:

http://docs.python.org/library/logging.html#logging-levels

> Gordon
> 
> On 16 Apr 2009, at 18:37, Gordon Hodgson wrote:
> 
>> On 16 Apr 2009, at 18:23, Amit Chakradeo (अमित चक्रदेव) wrote:
>>
>>> On Thu, Apr 16, 2009 at 10:13 AM, Gordon Hodgson 
>>> <feeds at greenfeed.org.uk <mailto:feeds at greenfeed.org.uk>> wrote:
>>>
>>>     What I've done is cut my config.ini in half into two files called
>>>     config1.ini and config2.ini (the scripts were terminating before
>>>     the spidering was complete)
>>>
>>>     Then I've run spider config1.ini
>>>     And then 5 minutes later  spider config 2.ini
>>>     And then 5 minutes later splice config.ini
>>>
>>>     Does this make sense?
>>>
>>> It does. But it still depends on how many feeds does each config file 
>>> has and how much time the final splice takes. 60 seconds is quite 
>>> restrictive. 
>>
>> Yes, the host has limited scripts to 60 seconds, and says that the 
>> longest recorded time he could find for me running the planet.py 
>> script is 80 seconds... now I have to run three scripts. Surely that 
>> will take more than 60 seconds of CPU time...?
>>
>>> Maybe you can just run the planet on your local machine and just 
>>> scp/FTP the generated files to the web server ?
>>
>> Not ideal. I don't like to leave my machines turned on more than I 
>> have to.
>>
>>>     I'm not completely sure about how much of the file info can be
>>>     removed (for instance, whether the avatar stuff and most of the
>>>     [planet] section needs to be in the spidered files (although I've
>>>     cut it out, and everything seems to work fine). Can anything be
>>>     cut out from the master config.ini (spliced) file? I haven;t been
>>>     able to work all this stuff out too easily from the Venus
>>>     documentation website (and I don't understand python).
>>>
>>> Take a look at the architecture 
>>> diagram: http://intertwingly.net/code/venus/docs/venus.svg
>>> <http://intertwingly.net/code/venus/docs/venus.svg>It looks like the 
>>> splice.py would need the avatar stuff etc. BTW you can leave all the 
>>> stuff in the config file, parsing config file takes minuscule amount 
>>> of time compared with rest of the things the scripts have to do...
>>
>> Yeah, that sort of makes sense to me...
>>
>> On slight tangent, is there any way the cron emails can report if a 
>> script finished early?

Unfortunately, it probably works the other way: cron will email you the 
output, so you can get a report if it runs to completion; getting a 
report if the script is terminated early would be a bit harder to 
accomplish...

>> I can only tell now, because occassionally one of the lines of the 
>> email is truncated (but sometimes it isn't).
>>
>> I've also reduced the time-out time to 10 seconds is this reasonable? 
>> I'm not even spidering a huge number of feeds... Something like 113...

time-out probably won't help if the limit is CPU seconds.

Feeds that aren't updated since the last run won't take much CPU time. 
Feeds that are updated but you don't get to due to the job getting 
killed early will be processed on the next run, presumably building up a 
backlog.

>>> --Amit
>>
>> Cheers for your help so far,
>>
>> Gordon
>> -- 
>> devel mailing list
>> devel at lists.planetplanet.org <mailto:devel at lists.planetplanet.org>
>> http://lists.planetplanet.org/mailman/listinfo/devel

- Sam Ruby