07 May 2006 I've spent some more time on my Skype to Google Maps prototype. The main challenge has been finding ways to make it more responsive. And the big problem is that Google Maps slows up dramatically when plotting more than about 50 points. So here's the basic setup
1. Use the Skype API via Delphi to extract all the contacts and their profile information. 2. Upload this to a PHP page whihc stores it in a mysql database. 3. Produce a php page that produces an XML file suitable for the Google API AJAX input based on the SW and NE corners of a square. It takes swlat, swlong, nelat and nelong as parameters. Does a Mysql search for entries with lat/long within that square and returns the results. 4. A Google Maps page that does the basics and then gets its data from the php search based on the current map viewport. There's about 500 points in the database. This is preparation for when I've got 10,000 points. Even with 500 I can't just display them all at once because the initial paint is way too slow. So I need to refresh the points when the map changes. So the first question is when to refresh the data. If you refresh on every move using the moveend event, you get a wait for new data even if you only move a few pixels. Plus, popping up an info window will auto-move the map if it's close to the edge. So what I do now is plot for a space that is 4 times as big (a border of half a map width in each direction. NSEW). If at the end of the moveend, the viewport is still within this space don't bother to refresh. I've then got a toggle using zoomend to force a refresh when the user changes the zoom level. Now at lower zooms you can still get more than 50 points needing plotting. I need a way of bunching the points together when they're close. I did a search for people doing this and found this page. Now he suggests doing the work in the database to bunch points and then reading it out depending on the zoom level. This probably produces the neatest results but it's a lot of extra work. I wanted an algorithm that would bunch automatically. So in the backend XML page, I do the basic query and get the results into an array. I then iterate through the array in a two layer loop looking for points that are within 1/10th of the map width and 1/10th of the map height. When I find one, I discard the point I'm exploring and alter the type of the matching point to "bunch". I then compute the point half way between them and change the matching point's lat/long to this mid point location. This effectively divides the map into 100 squares and then aggregates points so that there's no more than one in any square. If there are two or more, they're replaced with one "bunch" point. Now we have a maximum of 100 points returned. Finally I read through the array and generate the XML whihc looks like this. <marker lat="51.2091125" lng="-0.78744615625" type="bunch"> There's a bunch of 7 near here. Click to zoom in and see them. </marker> <marker lat="51.1289" lng="-0.015761" type="point"> Some HTML for the infowindow </marker> The first is a bunch point, the second is a real point. The contents of <marker></marker> is the html to be displayed in the infowindow. So now we've got the map displayed with it's points. And it only refreshes the points when you move half a map away or zoom. And we've limited the number of points plotted by bunching them. I use a standard icon for the main points and a custom icon for "Bunch" points created by recoloring one of Googles push pins. The last stage is to handle events for the UI. For plain points, I just use Marker.click to popup the infowindow using the html I passed over in the XML. I've also added a Map.click which checks if a marker has been clicked if it hasn't, then the user clicked in space between points and I close any open infowindow. For bunch points, I've also added a marker.mouseover and marker.mouseout which popup the infowindow explaining what a bunch point is. Finally, there's a bunch marker.click which zooms in, and then centres the map on the bunch point. This last caused me some problems. zoom also does a move and then the centring does a move. I had to be quite careful about sequence and added an extra flag so that the old points do actually get cleared before the new points get drawn. The next problem is when 2 or more points have exactly the same location. Perhaps because the geocoder was limited or because people live in the same town and that's all the address I've got. What I did was to add some small randomness (1/100th of map width) to spread the points out a bit. So there we have it. At the cost of some php processing in memory I can now handle large numbers of points with a fairly simple and self explanatory bunching algorithm. It seems to work well at small to medium zoom levels but still needs some fine tuning for country level maps. Now here's a complete aside. The UK postcode data is proprietary, commercial and expensive. There are free databases of the first part of the code to lat/long but not to any deeper level. So how about we build a database of the full postcode to lat/long. We then get people to enter their first part, draw a Google map based on that and then get them to locate their actual location on the map. We can then read out the lat/long and store it in the database. Over time we'd build a free and open dataset of full postcode to lat/long. We then offer a REST API to geocode the full postcode. Anyone up for this? email me at julian_bond at voidstar.com The usual problem applies. How do we give enough immediate value back to get people to actually do this and enter the data? And how do we clean the data and make sure it's reasonably accurate. And finally, the Skype Map mashup is an example of the limited security model around this data. Someone authorises me to see their profile. I can then get their address, geocode it and plot it and then present it to the world with no further authorisation from owner. Now I don't see how Skype can prevent this as long as they allow me access to the data. Left this on Doc's weblog.
The key phrase is "a government-managed regulatory habitat". He replied with some agreement and an argument that basically said "It's different in America." At whihc point too much coffee led me to this. > Also impossible. The telcos, like the copyright giants, have > successfully positioned their holdings as "simple property". They > built the infrastructure. It's their property. Why shoud government > come in and tell them how they can and cannot make money with that > property? That's anti-business, anti-market, etc. Won't fly. Seems my Anti-US sentiments are getting the better of me again :) It also seems that I keep on having these conversations where I point out an alternative and the other side is unwilling to even consider it because "That's just the way it is". Now I'm not specifically accusing Doc here of closed minded thinking but something much more general. I do think a lot of people and in particular USians on the web seem to be unable to see past the status quo and to think that there might be an alternative. It doesn't have to be this way. It really doesn't. The existing way of things may have huge momentum but it can and does change. Whatever. As a Brit I really should stop bothering with US Net Neutrality issues. If the typical US broadband customer gets screwed by their monopoly Telco ISP will it actually affect me at all, at all? Won't it just mean that the centre of innovation will move a bit more out of the USA? Spring and early summer are finally upon us because the swifts have arrived. yay!
05 May 2006 The engineering work yesterday took longer than expected due to a collection of small unforeseen problems. However we are up and running in our new rack.
We still have some work to do today (Sunday April 30) which may mean some short outages. Please bear with us, if Ecademy is out of action today. [from: JB Ecademy] [ 05-May-06 2:25pm ] 28 Apr 2006 popurls.com | popular urls to the latest web buzz
Where is the internet's attention focused today? [from: del.icio.us] 27 Apr 2006 25 Apr 2006 Dave W had an entry yesterday about BT and needs.
We need some BT help on the publishing end. Blog software has got quite good at one click publishing of rich media files and then automatically adding them to the RSS/Atom feeds. We need to make BitTorrent publishing as easy and simple. There's work to be done here to integrate the sort of approach at http://www.blogtorrent.com/ into mainstream blog software. Critical, I think, is to:- - Hide the complexity of creating the torrent - Having the web end be a BT client and automatically seed so there's always a full seed available - Having the client end stay online and guarantee 100% sharing from everyone who downloads. Wordpress developers! Are you listening? How about Drupal? BTW. Did we solve the problem of constructing a podcast feed that contains both plain and BT enclosures of the same file? 24 Apr 2006 Skype Journal: Two phreaks experiment with Skype contact integration :
Skype is relatively immune from SPIM because a lot of Skype users have their privacy settings set to only receive chat and voice from people in their contact list. I was asked to do a Skype API program to blast the same message to all of a person's contacts. It's quite easy to do and I did a proof of concept that takes a message from a form sends it and then closes each message window that pops up. you can then deal with the multiple popping windows as people reply at your leisure. But I didn't give it to the person who asked because I didn't want to be on the receiving end. Apart from scouring the web for addresses, the other approach is to build an API prog that people want to use (like say put winamp listening to into your mood). And then report their contact's profiles up to a database. This is getting close to a viral technique for gathering profile data. It's not necessarily evil but it's pushing the boundaries. Home Taping is Killing Music - Wikipedia, the free encyclopedia
Yay! My image made it into Wikipedia as an example of a parodied version. 23 Apr 2006 Enjoy. The next stage in the proof of concept is now done.
Notice the small ads top right. They pay good money. Still chicken feed but better than AdSense.
But what have I done? It looks to me like the benefit to the advertiser is taking advantage of my pagerank. Which means I'm distorting Google. Have I become Evil? I've currently got 534 contacts. Of these 257 have got good enough address detail to be able to get a single location in Google Maps. Skype is reporting that 233 of those contacts were last online this year. The intersection of the two is 67 people online this year and with address detail.
Now I've been collecting Skype contacts since the very early days so I may have a disproportionate number of old and dead entries. But a rule of thumb suggests that maybe 25% of someone's contact list can successfully be mapped. Which then raises the question. Why don't people fill in their profiles on Skype? And when they do, why don't they fill them in accurately? There's 117 entries online this year but with blank country. 22 Apr 2006 With the Skype API it's possible to grab profile information from all your contacts. You could then do something like post it on a google map such as this one. I can't decide if this is a really bad abuse of privacy or really neat.
I did a little more work on this last night and this morning. here's the pieces of the puzzle so far. - From the API you can get everything on your contact's profiles except email address. - I can use Google unofficially or some other geocoder service to convert the country+province+city into a lat/long - I can store all these and then read them out onto a Google Map. - If I can find the magic USP, and get lots of people to run a local Skype AddIn program, I can aggregate everybody's contact list together and effectively build a Skype white pages outside Skype. The privacy downside to this is that any one person only needs to authorise one user of the add in to see their profile detail and the world would be able to see it via my aggregator. As an aside I'm still looking for a truly global Geocoder that can take a fairly arbitrary address string and turn it into a Lat/Long. Google Maps is very good at this, but curiously you get different results on maps.google.com from maps.google.co.uk. A search for "ware, herts, uk" works in one but not the other. While "ware, hertfordshire, uk" works in both. Google doesn't like you scraping their html to get the results out and they don't offer a geocoder API service (yet). I found another one that uses the USA (CIA?) world city database http://worldkit.org/geocoder/rest/ but it's less resilient to bad or partial addresses. The problem is that even in countries that are fairly formalised end users get very confused about how to write an address. In the UK, there's quite a lot of large metropolita areas with no county. So for instance people desribe themselves as being in London, London, UK. The existing mapping services have a hard time with this. And don't get me started on all the map and geocoder services that are USA only. Like Yahoo! WSJ: eBay looking for allies against Google :: AO : eBay should go into competition with AdSense. They've got a huge inventory of listings. They could charge people who post on eBay a premium to have their listing added to the Ad Inventory. They then sell Ads to publishers/Bloggers in the same way as the AdSense program and use Paypal to pay them.
Look at the virtuous circle here. - Every Ad is also an Ad for eBay. - eBay listers get wider exposure. - publishers get real Ads instead of "Get iPods on eBay". - Paypal gets another boost. The trick is to solve the problem of getting the context right. Perhaps they should actually team up with Technorati and use tags[1] to solve the context problem. [1] When is eBay going to transition from fixed categories to tags? 14 Apr 2006 11 Apr 2006 I've now got Ecademy marketplace listings feeding automatically into Google Base, Google Blogsearch, and now EdgeIO.
EdgeIO was all pretty easy to work with as we already use Tags extensively in Marketplace, generate RSS and ping the major Ping Servers. I just added a "Listing" tag to the RSS and added EdgeIO to the Extended Ping list. I've added a couple of tags for the location, but I've hit the usual problem of USA based services not handling UK addresses very well. What I should be able to do is the same as for Google and just use an arbitrary address string and let a geocoder service figure it out. The main problem is that we have a very good postcode scheme in the UK that is accurate to a few hundred metres, but it's different from USA Zip Codes. 08 Apr 2006 The planned maintenance work for this evening has been re-scheduled. We will let you know nearer the time when we intend to complete the work.
There may be a short outage of 5 minutes or so around 6pm tonight. (Sat April 8th) [from: JB Ecademy] [ 08-Apr-06 6:41pm ] 07 Apr 2006 Google Talkabout: Pictures, themes, and more!
Wow! Googletalk has got pictures. More Wow! You can theme the background of chats! Come on guys WTF are you doing? How are you going to overhaul Skype like this? How about putting in some real missing function rather than screwing around. Chat Themes? Jeez! Meanwhile MSN Live Messenger V.8 Beta is out. Doesn't look significantly different from V7.0. Hidden in there is one significant feature. You can now send a chat to someone who's offline. This is perhaps the biggest reason I like Skype so much. Skypejournal is looking for the person with the most buddies in Skype. I know a couple with more than 1000. makes MSN's 150 limit look a bit sick. Jyve have launched the Beta of Jyve Pro. This is a Skype add-in that lets you charge for your time on a Skype call. Very interesting possibilities here. Interesting question on the Ecademy MacOSX forum. Can anyone suggest an alternative to MSN Messenger for Video Conferencing. I have used ichat but it means everyone I know has to change to aol or be on .mac Note that Skype for Mac doesn't yet have video. Still. Maybe users of Macinteltoshs will need to run Parallel and have XP running in a window just so they can have the latest greatest IM system running. 05 Apr 2006 To add to bOingbOing's ear cleaning saga, perhaps they'd like this image that I'm fairly sure I scanned from an old paper copy of their magazine.
[ 05-Apr-06 8:55am ] Who Designed This Crap? The Great Ipod Scam | MobilityGuru
No comment needed. Except to say that the comments were entertaining. |
The Blog