Take a look at the photograph below – it’s just an ordinary picture of two people outside a building, right? One of them appears to be handing something to the other.

Now take a closer look. Not everything in this picture is as it seems.

Two men exchanging some documents outside? Not necessarily. Credit: James O’Brien and Hany Farid.

The tell-tale signs may not jump out at you, but to Hany Farid, the image is littered with evidence – one of the reflections in the window is misaligned and the shadows do not line up.

This photograph is a fake. One of the men was not there at all.

Research suggests that regardless of what you might think about your own abilities to spot a hoax, most of us are pretty bad at it. Farid, however, looks at photographs in a different way to most people. As a leading expert in digital forensics and image analysis, he scrutinises them for the almost imperceptible signs that suggest an image has been manipulated.

One trick he has picked up over time is to check the points of light in people’s eyes. “If you have two people standing next to each other in a photograph, then we will often see the reflection of the light source (such as the Sun or a camera flash) in their eyes,” he explains. “The location, size, and colour of this reflection tells us about the location, size, and colour of the light source. If these properties of the light source are not consistent, then the photo may be a composite.”

Another giveaway is the colour of people’s ears. “If the Sun is behind me, my ears will look red from the front because you’ll see the blood,” he says. “If the light is coming from the front, you won’t see the red in the ear.”

But Farid also has some more scientific tools at his disposal. As the chair of computer science at Dartmouth College, he has been studying how to spot photographs that have been manipulated for decades.

Some other types of fake are easier to spot, like this Time magazine cover that was found displayed in the US President's golf clubs. Credit: Washington Post/Twitter.

Take shadow, for example. If you draw a line from the edge of a shadow in a photograph, to a point on the object that is casting the shadow, you can trace that further to reveal where the light in an image is coming from. If you map out several points on a shadow, the lines should intersect.

If a photo has been tampered with, the shadows of some objects in the image may not match the light sources in the rest of the picture, says Farid. He has shown it is possible with this method to identify images that have had objects or people added after they were taken.

Similarly, reflections like those in the image at the start of this article are also a giveaway. Again, by tracing a line from the person or object creating the reflection and their mirror image, they should all converge at a single point somewhere behind the reflective surface. If they do not, then something has been doctored.

Tracing the reflections in an image can make it easy to spot fakes if they don’t line up. Credit: Washington Post/Twitter.

In today’s world, fake images have implications for everything from politics to medicine.

“There isn’t an election that goes by where you don’t see fake photographs in one form or another,” says Farid. “Images will be manipulated to make a candidate look better. They might create these crowds to add diversity so the candidate doesn’t look like a racist, or use composites to show their opponents in a negative light.”

For example, when a picture was “unearthed” during the 2004 presidential election showing then candidate and Vietnam war veteran John Kerry sitting next to Jane Fonda at an anti-war rally in 1970, at least one prominent newspaper referred to the image, which spread across the internet. It later turned out to be a composite made from two different images.

This photo of politician John Kerry was designed to mislead. Credit: Fourandsix.com.

This photo of Barack Obama circulated in 2008, but is fake - the clock is a clue, a reference to a Hillary Clinton TV ad about picking up the phone at 3:00. Credit: Snopes.

Faked pictures are clearly not a new phenomenon, and indeed BBC Future has reported on their ubiquity before. In 2012, for example, we published a guide to fake images of Hurricane Sandy which were doing the rounds at the time. You may well have seen the dramatic, but fake, images of supercell storm clouds swirling above the Statue of Liberty. BBC Future writer Rose Eveleth has also reported on the way that fake images affect memories.

Doctored images emerged as Hurricane Sandy hit New York in 2012.

In fact, doctored images have been used since the early days of photography. Even a famous portrait of US President Abraham Lincoln is widely regarded as a composite, with the president’s head added onto the body of another politician (see below). The ubiquity of digital cameras and photo editing software has made the issue more troublesome than ever.

Photo from the Library of Congress.

Even governments are not above releasing manipulated images – Iran famously released a picture of a missile test in 2008 where one that had likely failed to launch had been made to look as if it had worked with some creative copy and paste (see below). When photographs from places like North Korea, Iraq and Syria are used to help governments make crucial security decisions, their veracity must be verified wherever possible, warns Farid.

Iran’s show of military might in 2008 was doctored to remove a launcher which failed to fire – and replaced with a fourth projectile. Courtesy: Fourandsix.com.

The Defense Advanced Research Projects Agency (Darpa), which develops technology for the US military, is attempting to create a tool that will automatically detect the manipulation of images and videos and assess their integrity. Farid is working as a researcher on one of the Darpa teams along with Kevin Conner, who co-founded image analysis firm Fourandsix with Farid in 2011. They have licensed their tool izitru, which looks at how a file is packaged and helps determine if an image came directly from the camera, to the agency.

“The challenge is that the technology is not at the stage where you can feed in any random image and get back some unequivocal answer,” says Conner, who worked at Adobe for 16 years and spent much of that time working on Photoshop. “It may not be possible to really get to that point, but that is essentially what Darpa is trying to do.”

By themselves, though, humans are exceptionally poor at identifying untrustworthy images. A recent study from Stanford University, for example, demonstrated that students from middle school to college struggle to analyse whether things they read online are credible. In one exercise, students were shown a photograph of what were supposedly “Fukushima Nuclear Flowers,” similar to the image below, posted to a website without credentials. Of the 170 high school students who saw the image, less than 20 percent successfully questioned the source of the photo.

This effect is not due to radiation, it's actually a natural process called fasciation. Credit: Candiru/Flickr/PD.

Even if we are sceptical of the source of an image, we are still bad at eye-balling inconsistencies. In one study conducted at the Federal University of Rio Grande do Sul in Brazil, for example, participants were presented with a series of photos and asked if they had been manipulated. While some of them were not altered in any way, more than half had been spliced (meaning that they were a composite of multiple photos), had areas that had been erased, or contained areas that were copied and pasted from the same image. Participants were only able to spot the fakes about 47 percent of the time.

Victor Schetinger, a doctoral candidate who worked on the study, says friends and colleagues regularly ask him about the legitimacy of photos. “All my research has informed me to conclude that I can’t tell (them) by examining a photo visually,” he says. “A lot of things can happen with an image that can leave artefacts and create something very visually striking. For example, maybe the image was saturated, maybe there was a strange flash, maybe a piece of dust on the lens. People will suspect it’s fake, but you shouldn’t assume that’s the case.”

If you’re curious about your ability to spot the difference between real images and those that have been Photoshopped, Adobe has put together a little online quiz. When I took it, I guessed correctly 15 out of 25 times, or about 60 percent of the time.

“More often than not, people think that the real images are fake and that things that are fake are real,” says Farid. “And their confidence is very high. So people are both ignorant and confident, which is the worst combination.”

The solution is to turn to computers to spot the inconsistencies that humans can miss. Photographic forensics uses a battery of techniques and algorithms to identify fake photos, many of which examine whether images fit with the laws of physics. While it may never be possible to authenticate a photograph to 100 percent confidence, forensics specialists can test photos using a number of techniques.

Let’s look at a famous photograph that has had conspiracy theorists whispering for decades.

This picture of Lee Harvey Oswald has often been claimed by conspiracy theorists to be a fake. Credit: Warren Commission/Wikipedia Commons.

The photo above is of Lee Harvey Oswald, the former US Marine who assassinated President John F Kennedy in 1963. According to authorities, the photo was taken in Oswald’s backyard and sent to his friend in April 1963. Investigators used it as evidence of Oswald’s guilt after matching markings from the rifle in the image to the gun found in the Texas School Book Depository in Dallas, Texas, after the assassination. Questions over the authenticity of the photo have fueled conspiracy theories that Oswald was framed for the assassination by the government, or criminal groups, particularly because Oswald himself denied the photo was real and was killed by a gunman before he could stand trial.

Conspiracy theorists have pointed to a few features in the picture as “evidence” of tampering – the shadows, particularly those on Oswald’s face, appear to some as if they are cast from a different light source than the shadows of other objects in the photo. Oswald’s chin looks broader than in his mugshot while his stance supposedly looks odd given the weight of the gun while others dispute the length of the gun itself in the image.

Farid and his colleagues examined the photo in a series of papers published in 2009, 2010 and 2015. In their analyses, researchers built 3D models of the scene and of Oswald based on his mugshot, his known height and weight, and the weight of the gun. They found that the shadows in the scene were consistent with a single light source, with the shadows on his face accounting for the appearance of a broader chin than that in his mug shot.

They also found that his posture was plausible given his centre of mass and the way he was carrying the gun, and estimated that the length of the rifle in the photo, after accounting for perspective, was 40.186in (101.2cm) long, less than an inch shorter than the length reported by the manufacturer. In all, the researchers couldn’t find any evidence of photo tampering.

Hany Farid and his team showed that Lee Harvey Oswald’s stance in the photograph was entirely plausible and so conclude the photo was genuine. Credit: Hany Farid.

“So this is a good example of the failure of our visual system to reason correctly,” says Farid. “You can’t really fault it because at first glance, some aspects of the photo do look weird. This is an interesting example where the forensic science can show that things that people are pointing to are not actually inconsistent with reality – they are all perfectly physically plausible.”

Other methods of authentication have nothing to do with the content of an image, but rather how its data file is packaged by the software that encodes it. When an image comes off a phone or camera, for example, it is often packaged as a jpeg file, which uses a type of lossy compression. Ordinarily there can be a lot of data in a photograph, so to reduce the size of digital files, some of the information is discarded (hence “lossy”) based on certain algorithms that make the file smaller. In addition, there is metadata associated with the image, with information about when the image was taken, what camera was used, how the thumbnail should look, and even the location where it was captured.

“There’s no such thing as a single jpeg format,” explains Farid. “Every camera compresses by different amounts. An iPhone compresses a lot more than a high-end SLR, for example. Even a point-and-shoot camera has different quality settings, and the way they create the thumbnails or store the metadata is a little different. All those things get embedded into a file.”

Law enforcement agencies often use this to help verify whether a picture has been altered since it was downloaded from the camera. “When you look into the packaging of a jpeg using code – the order in which all the bits of information are ordered are very specific and are very different for Photoshop versus an iPhone versus Panasonic or Nikon,” explains Farid. “So we can look at the packaging of a file and say this has gone through Photoshop because there are these tell-tale signs.”

It is still possible, however, for a photograph to be manipulated and the data reordered to look authentic, but it’s difficult to do. Farid compares it to trying to repackage an item like a new computer after you have taken it out of the box it came in.

“The way the package is wrapped with the Styrofoam, etc,” he explains. “If you try to take it all apart and put it back together, it would be very difficult to get it to look the exact same way. Hiding [file manipulation] is the digital equivalent of that. You can do it if you try really hard but it’s almost impossible, and you nearly always make mistakes.”

By tracing a line from a shadow in an image back to a light source, it is possible to see if something has been added to a picture. Credit: Hany Farid.

Despite this, Farid emphasises that forensic techniques do not guarantee fake photos can be spotted. They are, however, creating an arms race between hoaxers and those working in the forensic community.

“They raise the bar for the level of difficulty and skill required to create a compelling fake,” says Farid. “My hope is that when you run a photo through 20 or so different forensic techniques, and every single one, from the packaging to the shadows to the colour to the noise is completely consistent, it is more likely that the photo is real.”

So what can the rest of us do to spot those pesky fake images circulating on the internet? While we might not be able to use the full range of forensic techniques at Farid’s disposal to spot inconsistencies in images, there are other ways we can examine photos more critically. Reverse image searches (which can be done at sites like tineye.com or Google Images) are a good way to find out if a specific image has already been revealed as a fake. Reputable sites like snopes.com also vet viral images.

Farid also suggests looking at the source of the image. “Photos published on mainstream and reputable news sites like the New York Times have a high likelihood of being real as compared to photos published on unknown media sites, blogs, or Facebook,” he says.

Yet even supposedly reliable news organisations can get duped by a good photo. Often the best course of action is to ask yourself if a photo is simply too good to be true.

“A healthy amount of scepticism is always required when consuming digital images,” urges Farid. “But don’t let that scepticism overwhelm you as it is just as easy to think that a real photo is fake, as the other way around.”