Archiving sites

When we rebuilt the openly local map we knew that there were hundreds of hyperlocal sites, Facebook pages and Twitter accounts out there that weren’t on the map.  And that the map database contained hundreds of sites that had ceased publishing as natural wastage since 2009 in a fast moving sector where people are testing a myriad of business models.  Based on our recent work we judge that there are between 1,500 and 2,500 active hyperlocal sites in the UK of which we have 511 mapped.

Through some desk based exercises in the last couple of months, Finding Scottish Hyperlocals, Devon & Somerset – putting about 100 hyperlocal sites on the map and the Wakefield experiment we have added 209 sites to the map – not ‘new’ sites, just sites that had been around for ages but hadn’t got on the map. Our exercise covered barely 10% of the population. So based on this exercise we are cautiously confident that there are at least 1,000 other sites out there waiting to be added.

We have known for a long time that the OL map had lots of dead wood in it – sites that started and then shut up shop for some reason as part of natural wastage since 2009 – for instance the Daily Mail’s ‘Local People’ sites that ‘restructured’ in 2013 removing 122 sites from the database. So I’ve spent the past week reviewing all the sites on local Web List and archiving sites.

When we took the map over from Openly Local we imported 716 sites.  We added in around 209 in our desk based exercises.  This led to a grand total of 925 sites.

Every one of the 925 sites has been checked to see if it still exists and is occasionally updated, i.e is active somehow.  From this process the current state of the database is:

  • 925 sites at start of exercise
  • 372 sites no longer active (Archived but findable through search)
  • 42 sites that seem to have become spam or malware factories as their domains have been taken over – we have unpublished these to protect the unwary
  • 511 active hyperlocals displayed on map

This leaves 511 active hyperlocal sites on the map in the UK and ROI (that we know of).  These changes are natural wastage over five years that we have been overdue in catching up with.
Will and I have made a decision to not delete any sites in the list that are no longer active, but to move them into Archive status. This means that while they don’t appear on the map or lists of sites, they are still searchable. In the future if they do become active again we can just change the status of the site back to live.

The criteria I used when reviewing the sites was very roughly this:


Dead sites – Does the site exist yes or no, if the site returns a 404 or server doesn’t respond then I archive it.

Changed sites – If the site exists then I check is it the original site, if it isn’t then I’ll unpublish it.

Spam/Link Bait I have found that some domains have expired and been hoovered up, these sites are now something completely different, some are legitimate sites for new things but a vast majority of them are spam / link bait sites. Will and I will need to discuss what is the best way forward with these, I’d hate to just delete them so maybe we just need to remove the URL so people can’t click through. If you have a view on what we should do, then let us know.

Active sites – if it is the original site, I check to see if it has been updated in roughly the last 3 months, I take a varying view on this depending on the type of site:

News-y sites  So if it is a site that looks news led i.e. latest stories on the front page, or one that I know used to publish every day or week I look to see when the last post was published. If it has been updated recently I leave it, if it hasn’t then I move it to archive status.

Community information – If the site is more a community information site than a news site and hasn’t been updated then I leave it.

One thing that appeared during this exercise is that custom sites with domains seemed to stop publishing and die off more than sites that used the available free platforms like  Blogger & WordPress. Maybe the hassle of having to keep the site updated technically as well as with content makes it less fun? More analysis would be needed to draw any real conclusion from this.

All of this isn’t fool proof, so if I have got your site wrong or you think we should be doing it differently then get in touch with us.

Finding Scottish hyperlocals

Living and working in rural Aberdeenshire, where I run my own hyperlocal site, I am the official Scottish correspondent for Talk About Local. Following on from Will’s post about filling in the white space around Wakefield on the Local Web List map I thought I’d share my experiences adding in sites from Scotland.

Way back in the history of Talk About Local, Will found a great site that listed lots of community websites across Scotland, the fantastically named Sadly now the site seems to be more of a business directory and the list of local sites is nowhere to be seen.

Fortunately the wayback machine has a copy of the local sites page

I am using this list as a starting point to find local sites to add to the map. The sites listed on here will, if they are still live, have a decent history and archive of content. I’ve added around 50 sites using this method, when we took the data from Openly Local there were less than 20 sites listed for Scotland, I now have 64.

The task isn’t an easy one and can be mind numbingly boring at times, working through the list you have to click the link which takes you to the archive page, then you need to go to the current site, if it still exists, a great number don’t.

You come across sites which look like they have great potential but when you go to the live URL, it is ‘reserved for future use’, then you find a gem like

Once you have found the live site you need to review it, I use some basic criteria

  • Is the site still in the spirit of a hyperlocal site? There are a lot of early hyperlocal sites that have changed to glossy corporate sites or affiliate link directories sites. I discount these sites, unless I can find any hyperlocal information.
  • Is the site dead? If it is dead but not been hoovered up by someone and still has the original content on there then I add it for posterity.

there are no real short cuts for us to add sites to the list, we have to fill the form in for each site we find, this ensures that the data is accurate and valuable to the public and our academic colleagues. We do have to make some assumptions about the radius the site covers and the description of the area covered.

Sometimes when you are looking at these sites, you get drawn in to them, there is something on the site that is just too interesting to not read, then you end up clicking links and before you know it you have lost an hour. I lost more time than I should looking around what is one of my all time favourite sites and its virtual museum.

I also found this link on which may be of interest to anyone doing research in to rural networks

I am now focused on just reviewing each site and not getting drawn in to the links. Once I have checked each site listed on the archive page I can go back and revisit the sites I have added and look for other local sites that are linked from them to add to the list.

I’ve not yet found a hyperlocal site for Dunfermline, home of the funders of Local Web List, Carnegie UK Trust, if anyone can help with this or any other Scottish hyperlocal sites that are not yet on the map, feel free to jump in and add them for us, you shouldn’t be able to add any duplicates.


How to approach white spaces in the #hyperlocal map – Wakefield experiment

Carnegie UK Trust is supporting Talk About Local in improving the map of UK hyperlocal sites.  Despite having hundreds of entries, the map when we rehosted it had some curious white spaces – it can’t be right that Devon only had five sites listed nor that towns such as Wakefield had none, nor Scotland about eight.  So Mike and I at Talk About Local have been trying several methods of filling in blanks to help the hyperlocal community link up and to help those interested in finding a local site.

We’ve had some success which we are writing up and I’ll kick off with a desk based exercise/experiment to see what we could find for Wakefield, prompted by a Leeds hyperlocal lunch.

I have no hyperlocal contacts in Wakefield so started with some random Googling for Wakefield blogs, Wakefield civic news etc – ideally I am looking for sites that link to others locally with a civic feel.  Then ricocheted around web sites or reports about the local civic society, rotary, local history, genealogy, lists of Yorkshire lifestyle bloggers, a Wakefield hashtag and other groups until  I start to turn up web media with a hyperlocal feel.

In this case Unique Wakefield – a lovely service three years old now that promotes local independent traders.  It’s not a hyperlocal news-type site but is distinctive and genuinely local.  Their website is fine but as with most modern web operations the Facebook page and Twitter feeds contain real gems.

I start to google around things that pop up on Unique Wakefield and follow some links from Twitter etc. This then gives me other things to Google and I stumble across a news report about Horbury Village where someone is clearly trying to promote what goes on there.  This leads to a Horbury Village website which is very much a traditional parish site – from there though I get to the Facebook group and Twitter feeds and also an unexpected Twitter hashtag for Horbury.   The social media are rich.  Again this is a common pattern in 2015 – a trad website very much for reference that shows up in Google but new updates and action on Facebook and Twitter.

Looking at the map to work out where Horbury is I see a large parks complex to the North East and Google for Thornes Park hoping to find a ‘Friends of’ website, which turns up but as a bonus covers several local parks.   But I find another entry in google for Thornes Park on a much older platform which was nominated for an award for e-government in 2006.  Back tracking to the parent site takes me to Wakefield District Community Online – a platform apparently provided by the council. The full site list operating on this CMS seems to be over 100 local sites.  This will no doubt lead me off to record others to go on the map.  I have also mailed the local library to see if they have a list of links, which is common.

About 45 minutes work enables me to start to fill in a blank spot on the map and I still need to put a couple of hours more into Wakefield alone.  It’s very labour intensive and is hard to distil out as a method.  I was pleased that when I talked this through in a hyperlocal lunch in Leeds, no one there had come across the sites I had mapped.  Emma Bearman observed that this could be automated somehow – maybe running a gazetteer of place names against Twitter , looking for repetitions and hashtags to give a list for manual inspection.  I agree but am not sure how to go about it yet.  I shall write up shortly my experience with Devon and Somerset where I was able to add dozens of local parish sites.