On the Commons

Put a bird on it

Our collection of entirely free images on Flickr just got 3,500 high-resolution additions. Thanks, Python!

Everything we’ve uploaded to that account is yours to use in whatever way you want. These are images that have no copyright, donor restrictions, or anything else that might get in the way. They’ve also been uploaded at the highest resolution we have.

You can download the images, use them online, print them, turn them into lovely tea towels or linoleum.

(We’d love it if you linked back to us when you do, you don’t have to.)

The new uploads came from our free download pool, which is full of – you guessed it – images you can download for free. So why did I spend far too long writing terrible code to pipe them all up to Flickr?

First, there’s a ton of people on Flickr who are never going to come to our site. Why not get our stuff right in front of their faces?

Secondly, our interface for downloading these images isn’t great. You need to log in with RealMe and take the image you want through our whole image ordering process. On Flickr, zooming in to see details or downloading the size you want is far easier.

Lastly, I wanted to show that sharing our open content is possible at a large scale. I’m hoping this is going to make it easier for us to open up and share more from the collections.

What’s next?

We’ll be releasing more images, at the collection level, as they’re checked and cleared. That’ll probably mean repeatedly needing to upload dozens (or thousands?) of images in a go.

When that happens I’ll have a much cleaner script that is easier to use, and more reliable. From start to finish, it should:

• Go over the list of images we’ve picked out for inclusion
• Check if an image has already been uploaded, discard it if so
• Get the list of ID numbers we’re working with and send it off to NDHA so we can get the high-res tiffs
• Rename the files with their DigitalNZ identifiers
• Smoothly handle authentication with Flickr
• Grab all the metadata
• Sort images into their collections
• Upload!
• Mark each upload as done so we can run it in batches

Have a look at the code on GitHub

Read on for technical muckery!

Bleep bloop

Moving the files and all their info had several steps:

• Identify the free downloads
• Check if any were already on Flickr, and see what was on Flickr but not in the Free Downloads
• Get the source files
• Get their metadata
• Upload them with all that metadata

And to avoid getting lethal clicking finger strain or dying of boredom, do it all automatically 3,500 times over.

2 APIs and a bunch of messy Python

To write this code I picked Python, because it’s the language I’m most familiar with, and it’s well supported with helpful modules and advice around the web.

Step one was building a list to tell me what’s actually in the free download pool. Happily, natlib.govt.nz runs off the DigitalNZ API, making that information findable via API.

import requests
dnz\_api\_key = My DigitalNZ API key
def create\_free\_download\_list():
facet = "and[atl\_free\_download]=True"
result\_count\_url = "http://api.digitalnz.org/v3/records.json?api\_key=%s&%s&per\_page=0" % (dnz\_api\_key, facet)
results\_count\_json = requests.get(result\_count\_url).json()
count = results\_count\_json['search']['result\_count']
pages = count / 100 + 2
with open('freedownloads.txt', 'w') as f:
for page in range(1, pages):
search\_url = "http://api.digitalnz.org/v3/records.json?api\_key=%s&%s&per\_page=100&page=%d" % (dnz\_api\_key, facet, page)
search\_json = requests.get(search\_url).json()
for result in search\_json['search']['results']:
dnz\_id = result['id']
f.write('%s\n' % dnz\_id)

(My query was a simple lookup of items within a specific ‘collection’. For more complex stuff, you might like to use Chris McDowall’s pydnz wrapper.)

I now had a list of DigitalNZ identifiers, and needed to check if any of them were already up on Flickr. I knew that the existing uploads included descriptions that linked back to their source record. If I found a match, I could strike it from the list.

This was a two-step process. First, I pulled down the description data for each item, using Flickr’s own API:

def get\_photo\_info(i):
all\_ids = get\_pool\_ids()
request\_url = build\_flickr\_request(str(i))
data = untangle.parse(request\_url)
for description in data.rsp.photo.description:
photo\_description = description.cdata

Then, I processed the image’s description to look for a url, using a regular expression. If it was a natlib.govt.nz url, I checked the identifier against my list.

def get\_photo\_info(i):
natlib\_id = find\_natlib\_id(photo\_description)
...
def find\_natlib\_id(description):
try:
natlib\_url = re.search('(?Phttps?://[^"]+)', description).group("url")
if '/records/' in natlib\_url:
natlib\_id = natlib\_url[-8:]
return natlib\_id
else:
return False
except:
return False

From there I worked out if the ID was already in my list of images in the pool and wrote it to the appropriate place.

def get\_photo\_info(i):
...
natlib\_id = find\_natlib\_id(photo\_description)
if natlib\_id:
test\_photo(natlib\_id, all\_ids)
else:
with open('nonatlibrecord.txt', 'a') as f:
f.write("""
https://www.flickr.com/photos/nationallibrarynz\_commons/%r
""" % int(i))
f.close()
time.sleep(0.5)
def test\_photo(natlib\_id, all\_ids):
if int(natlib\_id) in all\_ids:
print "YAY"
with open('imagesinboth.txt', 'a') as f:
f.write("""
http://natlib.govt.nz/records/%r
""" % int(natlib\_id))
f.close()
ll\_ids.remove(int(natlib\_id))
else:
with open('flickrcommons.txt', 'a') as f:
f.write("""
http://natlib.govt.nz/records/%r
""" % int(natlib\_id))
f.close()

In the end, I had four lists:

• In both places (Already done! Yay!)
• In free downloads but not Flickr Commons (Let’s get this uploaded!)
• In Flickr Commons but not free downloads (These need adding to the free downloads pool!)
• Unknown (Take a closer look – it’s either a detail or missing a link to the source!)

My main concern was the second list, the 3,500 images that needed uploading with their metadata.

Let’s get the images

Our digitised images are much bigger than the versions you see on this site, and those high-res versions are what I wanted to share.

The versions held by the National Digital Heritage Archive are very large, very detailed tiffs, exactly right for pushing out to Flickr.

This is when I realised that the DigitalNZ identifiers I’d used in my list of files weren’t going to be any help. Those identifiers are applied after the NDHA have already taken the items in and preserved them (which is also after the cataloguing librarians have added their reference numbers). They have their own identification schema: Intellectual Entities, or IEs.

To get the image files, I needed to tell the NDHA what IE numbers I wanted. That information lives in DNZ record metadata, in the dc\_identifier field. A quick call to the DNZ API, asking for a specific field:

dnz\_api\_key = My DigitalNZ API key
def get\_IE\_number(dnz\_id):
api\_url = 'http://api.digitalnz.org.nz/v3/records/%s.json?api\_key=%s&fields=dc\_identifier' % (str(dnz\_id), dnz\_api\_key)
response = requests.get(api\_url).json()
for i in response['record']['dc\_identifier']:
if 'ndha' in i:
id\_string = i
IE\_number = id\_string[7:]
return IE\_number
else:
return None

And after a few thousand iterations I could ask for the right files.

(As an aside, the NDHA usually delivers access copies as jp2000 files. Before asking for the tiffs, I grabbed some jp2s and turned them into jpgs using Imagemagick. However, it turns out jp2s don’t retain the metadata tiffs keep. That could turn out to be valuable information for a user one day, so I went back for the tiffs.)

Authentication is a pain, let someone else do it

That might be bad advice. I’m new to this.

At this point I hit a brick wall of getting my head around Flickr’s API. First, because I was actually looking at out of date documentation. Then, on finding the updated documentation, I had no idea how to implement their authentication requirements.

I got lucky: there are many Python wrappers for Flickr, giving you access to the functions of the API without having to think as hard about it. I used python-flickr-api and was up and running right away.

(Another aside: along with the DNZ API key, we’ve now got a Flickr API key and a Flickr secret key. Keeping that stuff in your script’s not a great idea, so I stored them as environment variables. Here’s some info on setting environment variables in Mac OS Yosemite, which can be called in a Python script like so: dnz\_api\_key = os.environ['DNZ\_API\_KEY'])

Finally, the time had come to upload some images.

Metadata is better data

I wanted to make sure the images kept their context: basic catalogue information and a link back to the source.

Within the uploading process I looked this information up with another DigitalNZ query:

def get\_metadata(dnz\_id):
parameters = ['title','description']
api\_url = 'http://api.digitalnz.org/v3/records/%s.json?api\_key=%s' % (str(dnz\_id), dnz\_api\_key)
response = requests.get(api\_url).json()
title = response['record']['title']
description = response['record']['description']
return {'title': title, 'description': description}

And scraping the record’s page for the existing, nicely formatted citation. I extracted this from the page with BeautifulSoup:

import requests
from bs4 import BeautifulSoup
def add\_citation(description, dnz\_id):
natlib\_url = 'http://natlib.govt.nz/records/%s' % str(dnz\_id)
natlib\_page = requests.get(natlib\_url)
natlib\_html = natlib\_page.text
soup = BeautifulSoup(natlib\_html)
citation = soup.find("div", {"class": "usage"})
citation = citation.span.text
description = '%s \n\n %s' % (description, citation)
return description

Those functions are used in the course of actually uploading each item:

def upload(dnz\_id, IE\_number):
try:
photo\_path = ('files/%s.tiff' % IE\_number)
metadata = get\_metadata(dnz\_id)
title = metadata['title']
description = add\_citation(metadata['description'], dnz\_id)
flickr\_api.upload(photo\_file = photo\_path, title = title, description = description)
print '%s uploaded!' % title
status = open('files/%s.status' % IE\_number, 'a')
status.close
return True
except:
print "Unexpected error!", sys.exc\_info()[0]
raise
failure = open('files/%s.failure' % IE\_number, 'a')
failure.close

And bam: 100 gigs of images make it out into the wider world!

Now, go explore, download, and turn these fantastic pictures into whatever you want.

Thanks!

I got a bunch of help from:

• Chris McDowall
• Michael Lascarides
• Dan Charles
• Cynthia Wu
• Greig Roulston
• Jay Buzenberg