The process, which involves taking moulds of the teeth similar to those a dentist might take, used focus variation microscopy to digitally capture details of the tooth surfaces, zooming in to an area just 1/7th of a mm in width, around the same as that of a human hair.
These 3D data allowed the researchers to distinguish between different diet by comparing the roughness and shape of the tooth surface on a tiny scale. This offers a new method to analyse fishes diet based on the fossil record.
"We had a specific question: is it possible to use the microscopic wear patterns on fish teeth to work out what they eat." Explains Mark Purnell, an author on the paper, based at the University of Leicester.
"It's interesting as the shape of teeth is often used to infer the diet of fossil species - but we've seen with living species that they don't always eat what their teeth seem to be best adapted for."
Judging the diet of a species by tooth shape can be misleading.
"We saw this in this work, where the Sheepshead seabream and the Atlantic Wolffish have quite similar teeth, which look like they are most suitable for a shell-crushing predator," continues Purnell.
"However the Sheepshead seabream, like it's mammalian namesake, can have a diet that involves a significant amount of plant material."
One of the populations of Sheepshead seabream the researchers analysed had such a plant-based diet, and they were able to identify difference in texture between populations with different diets.
The researchers also looked at different populations of cichlid fish, and where able to distinguish between wild fish with a shell-crushing diet (including snails) and pond raised fish fed a more controlled diet.
"The tooth textures pick all this up," adds Purnell. "And the range of species this technique seems to work for was also pleasantly surprising."
Whilst this technique has been used before on land animals, the application to other fish species and sea mammals will allow researchers to investigate how diet controlled the evolution of aquatic creatures in the distant past.
The results are published in the journal Surface Topography: Metrology and Properties.