I've spent some more time on my Skype to Google Maps prototype. The main challenge has been finding ways to make it more responsive. And the big problem is that Google Maps slows up dramatically when plotting more than about 50 points. So here's the basic setup

1. Use the Skype API via Delphi to extract all the contacts and their profile information.
2. Upload this to a PHP page whihc stores it in a mysql database.
3. Produce a php page that produces an XML file suitable for the Google API AJAX input based on the SW and NE corners of a square. It takes swlat, swlong, nelat and nelong as parameters. Does a Mysql search for entries with lat/long within that square and returns the results.
4. A Google Maps page that does the basics and then gets its data from the php search based on the current map viewport.

There's about 500 points in the database. This is preparation for when I've got 10,000 points. Even with 500 I can't just display them all at once because the initial paint is way too slow. So I need to refresh the points when the map changes.

So the first question is when to refresh the data. If you refresh on every move using the moveend event, you get a wait for new data even if you only move a few pixels. Plus, popping up an info window will auto-move the map if it's close to the edge. So what I do now is plot for a space that is 4 times as big (a border of half a map width in each direction. NSEW). If at the end of the moveend, the viewport is still within this space don't bother to refresh. I've then got a toggle using zoomend to force a refresh when the user changes the zoom level.

Now at lower zooms you can still get more than 50 points needing plotting. I need a way of bunching the points together when they're close. I did a search for people doing this and found this page. Now he suggests doing the work in the database to bunch points and then reading it out depending on the zoom level. This probably produces the neatest results but it's a lot of extra work. I wanted an algorithm that would bunch automatically.

So in the backend XML page, I do the basic query and get the results into an array. I then iterate through the array in a two layer loop looking for points that are within 1/10th of the map width and 1/10th of the map height. When I find one, I discard the point I'm exploring and alter the type of the matching point to "bunch". I then compute the point half way between them and change the matching point's lat/long to this mid point location. This effectively divides the map into 100 squares and then aggregates points so that there's no more than one in any square. If there are two or more, they're replaced with one "bunch" point. Now we have a maximum of 100 points returned. Finally I read through the array and generate the XML whihc looks like this.

<marker lat="51.2091125" lng="-0.78744615625" type="bunch">
There's a bunch of 7 near here.
Click to zoom in and see them.
</marker>

<marker lat="51.1289" lng="-0.015761" type="point">
Some HTML for the infowindow
</marker>

The first is a bunch point, the second is a real point. The contents of <marker></marker> is the html to be displayed in the infowindow.

So now we've got the map displayed with it's points. And it only refreshes the points when you move half a map away or zoom. And we've limited the number of points plotted by bunching them.

I use a standard icon for the main points and a custom icon for "Bunch" points created by recoloring one of Googles push pins.

The last stage is to handle events for the UI. For plain points, I just use Marker.click to popup the infowindow using the html I passed over in the XML. I've also added a Map.click which checks if a marker has been clicked if it hasn't, then the user clicked in space between points and I close any open infowindow. For bunch points, I've also added a marker.mouseover and marker.mouseout which popup the infowindow explaining what a bunch point is. Finally, there's a bunch marker.click which zooms in, and then centres the map on the bunch point. This last caused me some problems. zoom also does a move and then the centring does a move. I had to be quite careful about sequence and added an extra flag so that the old points do actually get cleared before the new points get drawn.

The next problem is when 2 or more points have exactly the same location. Perhaps because the geocoder was limited or because people live in the same town and that's all the address I've got. What I did was to add some small randomness (1/100th of map width) to spread the points out a bit.

So there we have it. At the cost of some php processing in memory I can now handle large numbers of points with a fairly simple and self explanatory bunching algorithm. It seems to work well at small to medium zoom levels but still needs some fine tuning for country level maps.

Now here's a complete aside. The UK postcode data is proprietary, commercial and expensive. There are free databases of the first part of the code to lat/long but not to any deeper level. So how about we build a database of the full postcode to lat/long. We then get people to enter their first part, draw a Google map based on that and then get them to locate their actual location on the map. We can then read out the lat/long and store it in the database. Over time we'd build a free and open dataset of full postcode to lat/long. We then offer a REST API to geocode the full postcode. Anyone up for this? email me at julian_bond at voidstar.com The usual problem applies. How do we give enough immediate value back to get people to actually do this and enter the data? And how do we clean the data and make sure it's reasonably accurate.

And finally, the Skype Map mashup is an example of the limited security model around this data. Someone authorises me to see their profile. I can then get their address, geocode it and plot it and then present it to the world with no further authorisation from owner. Now I don't see how Skype can prevent this as long as they allow me access to the data.

Left this on Doc's weblog.

The key phrase is "a government-managed regulatory habitat".

Maybe what is needed is more legislation, not less. And not controls over what the Telcos and cable companies do with net neutrality but controls to force them to open up their monopoly to competing 3rd parties. eg

- Force them to sell wholesale bandwidth to 3rd party ISPs - Force them to sell space in their switching centres to 3rd party ISPs to unbundle the local loop.

When direct competitors are selling net neutral broadband, how will the Telcos be able to offer hobbled broadband?

The problem here is a common one to all utilities that have a monopoly hold over a single connection on the last mile. Phones, electricity, gas, water, sewage, cable TV, etc. The government has to control this for the common good. But the government can do this by granting a monopoly with controls or by creating an artificial market with competition. If they do the second, then they have to fine tune it to reflect changes in technology and possibilities. If they don't, it just falls back into the first approach. So we have the absurd situation of AT&T being broken up into the Baby Bells only for them to then merge back into a new "AT&T".

Before calling for Net Neutrality legislation take a long hard look at the UK experience and what the UK has done with BT. It's not perfect, but it almost works. BT had a similar government managed monopoly to the US Telcos. They've been forced to sell wholesale broadband and LLU. Now we have a genuinely competitive market that is forcing up broadband functionality and forcing down broadband cost.

And finally, the Cable companies have had it easy for too long. It's now time to treat them the same as Telcos. They got lots of tax credits for laying cable. It's now time to treat that cable as an unfortunate monopoly over a last mile and force them to open it up to 3rd parties.

He replied with some agreement and an argument that basically said "It's different in America." At whihc point too much coffee led me to this.

> Also impossible. The telcos, like the copyright giants, have > successfully positioned their holdings as "simple property". They > built the infrastructure. It's their property. Why shoud government > come in and tell them how they can and cannot make money with that > property? That's anti-business, anti-market, etc. Won't fly.

I know you know this. And I know it is different in the USA and maybe impossible.

But "anti-business, anti-market" is doublespeak. There is no market when the last mile is controlled in a government-managed regulatory habitat unless the government also creates an artificial market. The government has created a regulated monopoly. Now it needs to create a regulated market.

Looking from the outside at the USA it feels like we see this constantly. Endless bleating about pro-business, pro-market, pro-capitalist, anti-socialist, freedom of this that and the other, when reality is a very highly regulated, very highly government controlled environment where the government is for sale to the highest bidder (sorry, lobbier). If it wasn't for the last bit about everybody working for their shareholders (government included) it would be socialism. It's not capitalism. There isn't really a word for it. Unless it's Pigopolism. ;-)

Completely free. Just as long as you stay between the white lines on the freeway. And if you have enough money you can tell the government where to put the white lines.

Sorry. I seem to have gone off on one there...

Seems my Anti-US sentiments are getting the better of me again :) It also seems that I keep on having these conversations where I point out an alternative and the other side is unwilling to even consider it because "That's just the way it is". Now I'm not specifically accusing Doc here of closed minded thinking but something much more general. I do think a lot of people and in particular USians on the web seem to be unable to see past the status quo and to think that there might be an alternative. It doesn't have to be this way. It really doesn't. The existing way of things may have huge momentum but it can and does change.

Whatever. As a Brit I really should stop bothering with US Net Neutrality issues. If the typical US broadband customer gets screwed by their monopoly Telco ISP will it actually affect me at all, at all? Won't it just mean that the centre of innovation will move a bit more out of the USA?

Spring and early summer are finally upon us because the swifts have arrived. yay!

The engineering work yesterday took longer than expected due to a collection of small unforeseen problems. However we are up and running in our new rack.

We still have some work to do today (Sunday April 30) which may mean some short outages. Please bear with us, if Ecademy is out of action today. [from: JB Ecademy]

popurls.com | popular urls to the latest web buzz
Where is the internet's attention focused today? [from: del.icio.us]

Dave W had an entry yesterday about BT and needs.

We need some BT help on the publishing end. Blog software has got quite good at one click publishing of rich media files and then automatically adding them to the RSS/Atom feeds. We need to make BitTorrent publishing as easy and simple. There's work to be done here to integrate the sort of approach at http://www.blogtorrent.com/ into mainstream blog software. Critical, I think, is to:-

- Hide the complexity of creating the torrent
- Having the web end be a BT client and automatically seed so there's always a full seed available
- Having the client end stay online and guarantee 100% sharing from everyone who downloads.

Wordpress developers! Are you listening? How about Drupal?

BTW. Did we solve the problem of constructing a podcast feed that contains both plain and BT enclosures of the same file?

Skype Journal: Two phreaks experiment with Skype contact integration :

Skype is relatively immune from SPIM because a lot of Skype users have their privacy settings set to only receive chat and voice from people in their contact list.

I was asked to do a Skype API program to blast the same message to all of a person's contacts. It's quite easy to do and I did a proof of concept that takes a message from a form sends it and then closes each message window that pops up. you can then deal with the multiple popping windows as people reply at your leisure. But I didn't give it to the person who asked because I didn't want to be on the receiving end.

Apart from scouring the web for addresses, the other approach is to build an API prog that people want to use (like say put winamp listening to into your mood). And then report their contact's profiles up to a database. This is getting close to a viral technique for gathering profile data. It's not necessarily evil but it's pushing the boundaries.

Home Taping is Killing Music - Wikipedia, the free encyclopedia

Yay! My image made it into Wikipedia as an example of a parodied version.

Enjoy. The next stage in the proof of concept is now done.

1. Extract Profile information using Skype API, Delphi and the Skype4Com Activex


2. Upload the data to a Mysql dbms by calling a php page



3. Convert "City, Province, Country" to Lat/Long using The Worldkit Geocoder and some (ahem!) by hand with Google Maps.


4. Display the Map and fill in the points with the Google Maps API. This calls a PHP script to get entries from the DBMS based on the lat.long rectangle of the current map.


5. Any Questions? julian_bond at voidstar.com or skype:julian.bond?chat

Notice the small ads top right. They pay good money. Still chicken feed but better than AdSense.

But what have I done?

It looks to me like the benefit to the advertiser is taking advantage of my pagerank. Which means I'm distorting Google.

Have I become Evil?

I've currently got 534 contacts. Of these 257 have got good enough address detail to be able to get a single location in Google Maps. Skype is reporting that 233 of those contacts were last online this year. The intersection of the two is 67 people online this year and with address detail.

Now I've been collecting Skype contacts since the very early days so I may have a disproportionate number of old and dead entries. But a rule of thumb suggests that maybe 25% of someone's contact list can successfully be mapped.

Which then raises the question. Why don't people fill in their profiles on Skype? And when they do, why don't they fill them in accurately? There's 117 entries online this year but with blank country.

With the Skype API it's possible to grab profile information from all your contacts. You could then do something like post it on a google map such as this one. I can't decide if this is a really bad abuse of privacy or really neat.

I did a little more work on this last night and this morning. here's the pieces of the puzzle so far.

- From the API you can get everything on your contact's profiles except email address.
- I can use Google unofficially or some other geocoder service to convert the country+province+city into a lat/long
- I can store all these and then read them out onto a Google Map.
- If I can find the magic USP, and get lots of people to run a local Skype AddIn program, I can aggregate everybody's contact list together and effectively build a Skype white pages outside Skype.

The privacy downside to this is that any one person only needs to authorise one user of the add in to see their profile detail and the world would be able to see it via my aggregator.

As an aside I'm still looking for a truly global Geocoder that can take a fairly arbitrary address string and turn it into a Lat/Long. Google Maps is very good at this, but curiously you get different results on maps.google.com from maps.google.co.uk. A search for "ware, herts, uk" works in one but not the other. While "ware, hertfordshire, uk" works in both. Google doesn't like you scraping their html to get the results out and they don't offer a geocoder API service (yet). I found another one that uses the USA (CIA?) world city database http://worldkit.org/geocoder/rest/ but it's less resilient to bad or partial addresses. The problem is that even in countries that are fairly formalised end users get very confused about how to write an address. In the UK, there's quite a lot of large metropolita areas with no county. So for instance people desribe themselves as being in London, London, UK. The existing mapping services have a hard time with this. And don't get me started on all the map and geocoder services that are USA only. Like Yahoo!

WSJ: eBay looking for allies against Google :: AO : eBay should go into competition with AdSense. They've got a huge inventory of listings. They could charge people who post on eBay a premium to have their listing added to the Ad Inventory. They then sell Ads to publishers/Bloggers in the same way as the AdSense program and use Paypal to pay them.

Look at the virtuous circle here.
- Every Ad is also an Ad for eBay.
- eBay listers get wider exposure.
- publishers get real Ads instead of "Get iPods on eBay".
- Paypal gets another boost.
The trick is to solve the problem of getting the context right. Perhaps they should actually team up with Technorati and use tags[1] to solve the context problem.

[1] When is eBay going to transition from fixed categories to tags?

eConsultant : Web 2.0 Directory, Technical Tips and Lists
[from: del.icio.us]

I've now got Ecademy marketplace listings feeding automatically into Google Base, Google Blogsearch, and now EdgeIO.

EdgeIO was all pretty easy to work with as we already use Tags extensively in Marketplace, generate RSS and ping the major Ping Servers. I just added a "Listing" tag to the RSS and added EdgeIO to the Extended Ping list. I've added a couple of tags for the location, but I've hit the usual problem of USA based services not handling UK addresses very well. What I should be able to do is the same as for Google and just use an arbitrary address string and let a geocoder service figure it out. The main problem is that we have a very good postcode scheme in the UK that is accurate to a few hundred metres, but it's different from USA Zip Codes.

The planned maintenance work for this evening has been re-scheduled. We will let you know nearer the time when we intend to complete the work.

There may be a short outage of 5 minutes or so around 6pm tonight. (Sat April 8th) [from: JB Ecademy]

Google Talkabout: Pictures, themes, and more!

Wow! Googletalk has got pictures. More Wow! You can theme the background of chats!

Come on guys WTF are you doing? How are you going to overhaul Skype like this? How about putting in some real missing function rather than screwing around. Chat Themes? Jeez!

Meanwhile MSN Live Messenger V.8 Beta is out. Doesn't look significantly different from V7.0. Hidden in there is one significant feature. You can now send a chat to someone who's offline. This is perhaps the biggest reason I like Skype so much.

Skypejournal is looking for the person with the most buddies in Skype. I know a couple with more than 1000. makes MSN's 150 limit look a bit sick.

Jyve have launched the Beta of Jyve Pro. This is a Skype add-in that lets you charge for your time on a Skype call. Very interesting possibilities here.

Interesting question on the Ecademy MacOSX forum.
Can anyone suggest an alternative to MSN Messenger for Video Conferencing. I have used ichat but it means everyone I know has to change to aol or be on .mac Note that Skype for Mac doesn't yet have video. Still. Maybe users of Macinteltoshs will need to run Parallel and have XP running in a window just so they can have the latest greatest IM system running.

To add to bOingbOing's ear cleaning saga, perhaps they'd like this image that I'm fairly sure I scanned from an old paper copy of their magazine.

Who Designed This Crap? The Great Ipod Scam | MobilityGuru

No comment needed. Except to say that the comments were entertaining.