[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [TV] Source code, GPL and licenses...
OK. so I pulled and scanned the code... (tv-v2.00-20040126.tar.gz)
What I'm not seeing is the grabber source.... any suggestions?
Looking at dgc2xml.pl, I'm guessing that you have URLs for Digiguide
data for some sites.
I've also gone back through the archives and seen the 'mirroring'
I'd add some thoughts in this area (bear in mind I'm thinking primarily
PVR here - although all the Peers could act as bleb www and xml mirrors too)
Listing data changes after publication and before it's 'live'.
I think there are 3 reasonable stages:
14 days away : Fuzzy data, quite a few unknown slots. Useful advance
notice of some things.
7 days away : Pretty solid. Ready to be paper published.
1 days away : Last minute schedule changes.
I call this the +1/+7/+14 approach.
Source : master site for Diguide/XML/HTML listings data.
Network : An ordered list of Peers offering the bleb.org service
Master : www.bleb.org
Peer : www.another.site in the Network
Grab : A grab from the Source site
Scheduled Grab : Each channel has an schedule for obtaining data from
the Source Site.
Client : application (not browser) using bleb data (eg: xmltv's
Use : a Client grab or use of the service.
Service is available if any Network member fails.
Reasonable Client load balancing
Minimal load on Sources
High level Peer behaviour
A Peer regularly [on the hour + 2mins * position in the Network list]
connects to the top entry in the Network list to synchronise listings
data and code [rsync].
If the connection fails, the Network member immediately trys the next
entry in the Network list and so on until the next entry points to
itself; in that case it checks to see if a Scheduled Grab is due and
tries to Grab from the Source.
The Master immediately checks to see if a Scheduled Grab is due and
tries to Grab from the Source.
The data is stamped in order that the test for the retrieving Scheduled
Grab will fail in the future.
If any Grab fails then notification is sent.
This is a fairly simple distribution approach - less to go wrong.
Clock sync between the Network servers isn't critical (but ntp is always
Peers will never attempt a Grab unless the Master and all intervening
Peers are unreachable.
When any Network member returns it should not respond to requests until
it has synchronised with a superior (or the Sources in the case of the
There's one obvious timing issue: we need to understand how long a full
set of Grabs will take allowing for parallelism and maybe the +1/+7/+14
Worst case is a Master going down just before a Grab set is due to
complete and the next Peer starting the Grab very late and causing eager
clients to essentially miss a days update. Clients could notice this and
compensate the next day.
Note that Grabs are retried after an hour. Potential issue if a grab
takes over an hour! That would require some locking. If the grabbing
server failed whilst the lock was held then the assuming Peer should
ignore the lock.
High level Client behaviour
Initially Clients pull the Network list from the Master; thereafter they
select a machine from the Network at random and use that. If a
connection times out then they try a different Peer. The Network list is
updated each time the Client uses the bleb service.
Load balancing is done simply by the client.
Temporary Peer failure is hardly noticed.
Out of service Peers can be removed by the network and will cause minor
inconvenience until the Use following their removal (nothing stops the
client from blacklisting a Peer). In the event of the Master going down
for good (for whatever reason) then the next entry in the Network can
issue a new Network list.
Service Uses should probably be scheduled to allow time for Sources to
be consulted - maybe 3am onwards?
[ I wrote this and it's true so I'll leave it in but a review tells me
it's overkill: ] From a 'security' point of view - any Peer can
permanently hijack clients using it by providing a bad Network file. If
we care then we could digitally sign the file and arrange for trusted
admins to have access to the signature. The clients would then refuse to
honour a bad Network file.
This is just a start - I've no doubt made mistakes :)