It's been a while since I started my little adventure of building an image recognition pipeline that would reliably identify animal and plant species from photos. I'm pretty happy with what the results so far, and these days I'm even successfully scratching my own itch with it. Which is how I came up with the idea in the first place: I suck at identifying birds, butterflies, and plants, but I want to learn about them. And knowing a species' name is really the gateway to learning more about it.
So here's a painless way of identifying these species. Say you want to identify one bird that you see. You take a photo of it and send it to @id_birds on Twitter using Twitter's own image upload. After a minute or two, you'll get the automatic reply, with hopefully the right species name. @id_birds can with a remarkably high accuracy identify yours between the 256 different birds that it knows.
There's two other bots that are also available to use for free: @id_butterflies knows around 50 butterfly species, and @id_wildflowers around 200 wildflowers. All three bots know species occurring natively in Europe only, at least for now.
"But does it actually work?" I hear you say. And the answer is: it works pretty well, especially if you're sending it good quality photos where the animal or plant is clearly visible, is not too blurry, has good white balance etc.
Seeing is believing though, so to prove that it works well, let's try and identify the ten birds that are in The Wildlife Trust's river bird spotter sheet.
Image credits: Grey heron (c) Amy Lewis / Moorhen and mallard (c) Steve Waterhouse / Kingfisher (c) Margaret Holland / Mute swan (c) Neil Aldridge / Dipper and grey wagtail (c) Tom Marshall / Coot (c) David Longshaw / Great crested grebe (c) Don Sutherland / Tufted duck (c) Bob Coyle
Let's start with the first bird, a grey heron. To the left we have me addressing the Twitter bot and sending the image. To the right is @id_birds' reply. And it's correct! Ardea cinerea is indeed the scientific name for the grey heron:
@id_birds hey can you ID me this bird from @wildlifetrust's river birds spotter sheet? pic.twitter.com/z5JnE3WIb3
— Daniel Nouri (@dnouri) September 13, 2014@dnouri
1. Ardea cinerea http://t.co/gGuu8ombMs
2. Anthropoides virgo http://t.co/exqiChamSg
— Bird auto-ID (@id_birds) September 13, 2014Notice how @id_birds will always respond with the two most likely guesses. The second guess here was the demoiselle crane (Anthropoides virgo), which looks sort of similar.
The next bird on the river bird spotter sheet is the common moorhen (Gallinula chloropus). Let's try to ID it:
@id_birds how about this one? pic.twitter.com/9csoNaAaJw
— Daniel Nouri (@dnouri) September 13, 2014@dnouri
1. Anas platyrhynchos http://t.co/9xoDyXJGMJ
2. Gallinula chloropus http://t.co/ISQ0N7m2Lu
— Bird auto-ID (@id_birds) September 13, 2014And already we have @id_birds' first mis-classification. It thinks that this is a mallard (Anas platyrhynchos). Only the second guess was correct. Thus so far we're counting one correct, one error.
The next bird we want to identify is incidentally a mallard:
@id_birdspic.twitter.com/x9h89dKkkb
— Mushroom auto-id (@id_mushrooms) September 13, 2014@id_mushrooms
1. Anas platyrhynchos http://t.co/9xoDyXJGMJ
2. Coracias garrulus http://t.co/MKJTOTfeOv
— Bird auto-ID (@id_birds) September 13, 2014And the response is correct. Notice how the second guess (Coracias garrulus) seems a little weird. I guess it's the similarities in color here. We don't count the second guesses here though, so it's two correct, one error.
What about the kingfisher? That's a bird with some pretty unique appearance among birds native to Europe. So it's probably one of the easier birds to identify:
@id_birdspic.twitter.com/NNy1t6izLQ
— Butterfly auto-ID (@id_butterflies) September 13, 2014@id_butterflies
1. Alcedo atthis http://t.co/Q1fUKAlYSt
2. Hirundo rustica http://t.co/mtVqrMC8oG
— Bird auto-ID (@id_birds) September 13, 2014And sure enough, @id_birds knows what it is. The latin name for the kingfisher is Alcedo atthis.
The fifth bird on the spotter sheet is the mute swan (Cygnus olor). Everyone probably knows this one, but does @id_birds?
@id_birds and this? pic.twitter.com/RiFk3pP38B
— Butterfly auto-ID (@id_butterflies) September 13, 2014@id_butterflies
1. Cygnus olor http://t.co/jz7DLdqVW2
2. Cygnus cygnus http://t.co/wx8MSFv3sy
— Bird auto-ID (@id_birds) September 13, 2014Yes, the answer is correct. Note that the second guess is also a swan, namely the whooper swan, quite similar in appearance. A good job!
So out of the first five birds we got four correct, not bad. But let's see about the rest of the birds from the spotter sheet.
I'll post the responses to the remaining five birds without further comment. But do note that of the remaining ones, only the coot (Fulica atra) is mis-classified, that's the third one below:
@id_birds I found this one in @wildlifetrust's river birds spotter sheet. Do you know what it is? pic.twitter.com/R9Qf5s2sSf
— Wildflower auto-ID (@id_wildflowers) September 13, 2014@id_wildflowers
1. Cinclus cinclus http://t.co/u2W2R8NYZX
2. Corvus corone http://t.co/1JjU7HtcSj
— Bird auto-ID (@id_birds) September 13, 2014@id_birds Here's another cute bird from that same spotter sheet. Can you ID it? pic.twitter.com/onxOeALtaL
— Daniel Nouri (@dnouri) September 13, 2014@dnouri
1. Motacilla cinerea http://t.co/RhZyl5ltwy
2. Motacilla flava http://t.co/TWG5YgTxBQ
— Bird auto-ID (@id_birds) September 13, 2014@id_birdspic.twitter.com/qYVqQGKa4I
— Wildflower auto-ID (@id_wildflowers) September 13, 2014@id_wildflowers
1. Gallinula chloropus http://t.co/ISQ0N7m2Lu
2. Podiceps nigricollis http://t.co/WgflyFJyCU
— Bird auto-ID (@id_birds) September 13, 2014@id_birds Can you ID this as Podiceps cristatus? pic.twitter.com/jQwS5cjuEi
— Wildflower auto-ID (@id_wildflowers) September 13, 2014@id_wildflowers
1. Podiceps cristatus http://t.co/JzrmePWuf9
2. Branta ruficollis http://t.co/X6fCvqF0xo
— Bird auto-ID (@id_birds) September 13, 2014@id_birds here's the last one: a tufted duck (Aythya fuligula) pic.twitter.com/S8VwHd6rry
— Butterfly auto-ID (@id_butterflies) September 13, 2014@id_butterflies
1. Aythya fuligula http://t.co/n4O4tTIwtv
2. Aythya marila http://t.co/H9g0mXy6Eu
— Bird auto-ID (@id_birds) September 13, 2014So all in all we have two mis-classifications out of ten. That's 80% accuracy, which is pretty remarkable when you consider that differences between these birds can sometimes be minuscule (grey wagtail versus yellow wagtail, anyone?). As mentioned, @id_birds knows 256 bird species. Which means that if you made guesses at random, then you'd have 0.4% accuracy.
I'll write about how the computer vision algorithm works under the hood another time. I call the stack that I've developed and that's running behind @id_birds and the other bots PhotoID. So far PhotoID has proven to have very competitive performance and be remarkably flexible, that is, I can use it as a basis for different kinds of classification problems without too much adaptation work.
Let's conclude this post with two other IDs, one from @id_butterflies, the other from @id_wildflowers:
#WansteadFlats It's an upside down world sometimes. Maybe a Meadow Brown? @id_butterflies@wildlife_ukpic.twitter.com/yDnkTUTMO8
— TheNatureOfTheFlats (@RoseStephensArt) August 22, 2014@id_butterflies
1. Maniola jurtina http://t.co/4YGYG6WQHc
2. Cupido minimus http://t.co/2hbLKCYzpF
— Butterfly auto-ID (@id_butterflies) August 22, 2014Can @id_wildflowers identify this wildflower photographed by @LornaCrowcroft?
Wondering what it is? See the reply. pic.twitter.com/WI31e7aN8x
— Wildflower auto-ID (@id_wildflowers) September 4, 2014@id_wildflowers
1. Solanum nigrum http://t.co/kvMbrC7KB4
2. Cyclamen hederifolium http://t.co/mcZHVmRecq
— Wildflower auto-ID (@id_wildflowers) September 4, 2014Happy ID'ing!
(Plug: Feel free to contact me through my company if you have a challenging image recognition problem that you want me to help solve.)