Recent problems

[Andrew Flegg <andrew@xxxxxxxx>]

Hi,

You may have noticed there've been some problems the last few days with
nil-sized replies and failed downloads.

This is because a couple of applications have been downloading XMLTV
downloads for all channels on all days; and then these apps have
become more popular than their authors expected.

The problem can be seen on this graph of system load:

    http://coto.bleb.org/mrtg/sysload.html

On around the 7th April, the load average was sitting about where it is
now: low. Very low. Around 0.01-0.1. This shows that the box has plenty
of spare capacity to do processing. The load average ramped up to around
32 on Sunday evening, after steadily increasing over the last week.

It's technically not the fault of these applications: after all
downloading the entire data set is inefficient, but not against the
rules. The fault is mine for the inefficient way in which the "bleb"-XML
is transformed into XMLTV via XSLT.

As a stop-gap measure, I've limited the number of channels available in
any bulk download to 10 - apologies for this.

What needs doing is a more clever provision of XMLTV data: I've a few
thoughts on this and will (hopefully) have time to do it this weekend.
The same process may also allow end-times on days 0-5 for the final
channels.

The thought is to have a new process outside of the framework[1] which
takes the bleb-XML and converts it, statically, to a fragment of XMLTV.
This fragment will be stored as a static file (just like the bleb-XML),
and the bulk download will provide a suitable header, concatenate the
requested fragments and then append a suitable footer.

This should make the XMLTV provision *much* more efficient.

Any thoughts? In particular, is there any value in having the XMLTV
fragments available outside of a "proper" XMLTV download?

Cheers,

Andrew

-- 
Andrew Flegg -- mailto:andrew@xxxxxxxx  |  http://www.bleb.org/