Classification of multiple emotional states from facial expressions in head-fixed mice using a deep learning-based image analysis

1.

Darwin C. The expression of the emotions in man and animals. D Appleton, New York. 1873.

2.

Ekman P. Emotion in the human face: Guide-lines for research and an integration of findings. Pergamon Press, New York. 1972.

3.

Hart TC, Hart PS. Genetic studies of craniofacial anomalies: clinical implications and applications.

Orthod Craniofac Res. 2009; 12(3):212–20. https://doi.org/10.1111/j.1601-6343.2009.01455.x PMID:

19627523

4.

Grabowski K, Rynkiewicz A, Lassalle A, Baron-Cohen S, Schuller B, Cummins N, et al. Emotional

expression in psychiatric conditions: New technology for clinicians. Psychiatry Clin Neurosci. 2019; 73

(2):50–62. https://doi.org/10.1111/pcn.12799 PMID: 30565801

5.

Gurovich Y, Hanani Y, Bar O, Nadav G, Fleischer N, Gelbman D, et al. Identifying facial phenotypes of

genetic disorders using deep learning. Nat Med. 2019; 25(1):60–4. https://doi.org/10.1038/s41591-0180279-0 PMID: 30617323

6.

Pell MD, Leonard CL. Facial expression decoding in early Parkinson’s disease. Brain Res Cogn Brain

Res. 2005; 23(2–3):327–40. https://doi.org/10.1016/j.cogbrainres.2004.11.004 PMID: 15820640

7.

Parr LA, Waller BM, Fugate J. Emotional communication in primates: implications for neurobiology.

Curr Opin Neurobiol. 2005; 15(6):716–20. https://doi.org/10.1016/j.conb.2005.10.017 PMID: 16274983

8.

Boneh-Shitrit T, Feighelstein M, Bremhorst A, Amir S, Distelfeld T, Dassa Y, et al. Explainable automated recognition of emotional states from canine facial expressions: the case of positive anticipation

and frustration. Sci Rep. 2022; 12(1):22611. https://doi.org/10.1038/s41598-022-27079-w PMID:

36585439

9.

Langford DJ, Bailey AL, Chanda ML, Clarke SE, Drummond TE, Echols S, et al. Coding of facial expressions of pain in the laboratory mouse. Nat Methods. 2010; 7(6):447–9. https://doi.org/10.1038/nmeth.

1455 PMID: 20453868

10.

Finlayson K, Lampe JF, Hintze S, Wurbel H, Melotti L. Facial Indicators of Positive Emotions in Rats.

PLoS One. 2016; 11(11):e0166446. https://doi.org/10.1371/journal.pone.0166446 PMID: 27902721

11.

Dolensek N, Gehrlach DA, Klein AS, Gogolla N. Facial expressions of emotion states and their neuronal

correlates in mice. Science. 2020; 368(6486):89–94. https://doi.org/10.1126/science.aaz9468 PMID:

32241948

12.

Li W, Nakano T, Mizutani K, Kawatani M, Matsubara T, Danjo T, et al. Primary motor cortex drives

expressive facial movements related to reward processing in mice. bioRxiv. 2022:2022.10.28.514159.

13.

Tuttle AH, Molinaro MJ, Jethwa JF, Sotocinal SG, Prieto JC, Styner MA, et al. A deep neural network to

assess spontaneous pain from mouse facial expressions. Mol Pain. 2018; 14:1744806918763658.

https://doi.org/10.1177/1744806918763658 PMID: 29546805

14.

Loken LS, Wessberg J, Morrison I, McGlone F, Olausson H. Coding of pleasant touch by unmyelinated

afferents in humans. Nat Neurosci. 2009; 12(5):547–8. https://doi.org/10.1038/nn.2312 PMID:

19363489

15.

Pawling R, Cannon PR, McGlone FP, Walker SC. C-tactile afferent stimulating touch carries a positive

affective value. PLoS One. 2017; 12(3):e0173457. https://doi.org/10.1371/journal.pone.0173457

PMID: 28282451

16.

Liu Q, Vrontou S, Rice FL, Zylka MJ, Dong X, Anderson DJ. Molecular genetic visualization of a rare

subset of unmyelinated sensory neurons that may detect gentle touch. Nat Neurosci. 2007; 10(8):946–

8. https://doi.org/10.1038/nn1937 PMID: 17618277

PLOS ONE | https://doi.org/10.1371/journal.pone.0288930 July 20, 2023

12 / 14

PLOS ONE

Deep learning-powered classifier for mouse facial expressions

17.

Vrontou S, Wong AM, Rau KK, Koerber HR, Anderson DJ. Genetic identification of C fibres that detect

massage-like stroking of hairy skin in vivo. Nature. 2013; 493(7434):669–73. https://doi.org/10.1038/

nature11810 PMID: 23364746

18.

Huzard D, Martin M, Maingret F, Chemin J, Jeanneteau F, Mery PF, et al. The impact of C-tactile lowthreshold mechanoreceptors on affective touch and social interactions in mice. Sci Adv. 2022; 8(26):

eabo7566. https://doi.org/10.1126/sciadv.abo7566 PMID: 35767616

19.

Elias LJ, Succi IK, Schaffler MD, Foster W, Gradwell MA, Bohic M, et al. Touch neurons underlying

dopaminergic pleasurable touch and sexual receptivity. Cell. 2023; 186(3):577–90 e16. https://doi.org/

10.1016/j.cell.2022.12.034 PMID: 36693373

20.

Le Moene O, Larsson M. A New Tool for Quantifying Mouse Facial Expressions. eNeuro. 2023; 10(2).

https://doi.org/10.1523/ENEURO.0349-22.2022 PMID: 36759187

21.

Kuo CM, Lai SH, Sarkis M. A Compact Deep Learning Model for Robust Facial Expression Recognition.

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2018.

22.

Schofield D, Nagrani A, Zisserman A, Hayashi M, Matsuzawa T, Biro D, et al. Chimpanzee face recognition from videos in the wild using deep learning. Sci Adv. 2019; 5(9):eaaw0736. https://doi.org/10.1126/

sciadv.aaw0736 PMID: 31517043

23.

Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. arXiv. 2017.

24.

Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual Explanations

from Deep Networks via Gradient-Based Localization. IEEE International Conference on Computer

Vision. 2017:618–26.

25.

Guo ZV, Hires SA, Li N, O’Connor DH, Komiyama T, Ophir E, et al. Procedures for behavioral experiments in head-fixed mice. PLoS One. 2014; 9(2):e88678. https://doi.org/10.1371/journal.pone.0088678

PMID: 24520413

26.

Augustinaite S, Kuhn B. Chronic Cranial Window for Imaging Cortical Activity in Head-Fixed Mice.

STAR Protoc. 2020; 1(3):100194. https://doi.org/10.1016/j.xpro.2020.100194 PMID: 33377088

27.

Komiyama T, Sato TR, O’Connor DH, Zhang YX, Huber D, Hooks BM, et al. Learning-related fine-scale

specificity imaged in motor cortex circuits of behaving mice. Nature. 2010; 464(7292):1182–6. https://

doi.org/10.1038/nature08897 PMID: 20376005

28.

Ino D, Tanaka Y, Hibino H, Nishiyama M. A fluorescent sensor for real-time measurement of extracellular oxytocin dynamics in the brain. Nat Methods. 2022; 19(10):1286–94. https://doi.org/10.1038/

s41592-022-01597-x PMID: 36138174

29.

Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. Imagenet: A large-scale hierarchical image database.

IEEE conference on computer vision and pattern recognition. 2009:248–55.

30.

Schindelin J, Arganda-Carreras I, Frise E, Kaynig V, Longair M, Pietzsch T, et al. Fiji: an open-source

platform for biological-image analysis. Nat Methods. 2012; 9(7):676–82. https://doi.org/10.1038/nmeth.

2019 PMID: 22743772

31.

Ebbesen CL, Froemke RC. Body language signals for rodent social communication. Curr Opin Neurobiol. 2021; 68:91–106. https://doi.org/10.1016/j.conb.2021.01.008 PMID: 33582455

32.

O’Connor DH, Clack NG, Huber D, Komiyama T, Myers EW, Svoboda K. Vibrissa-based object localization in head-fixed mice. J Neurosci. 2010; 30(5):1947–67. https://doi.org/10.1523/JNEUROSCI.

3762-09.2010 PMID: 20130203

33.

Hinchcliffe JK, Mendl M, Robinson ESJ. Rat 50 kHz calls reflect graded tickling-induced positive emotion. Curr Biol. 2020; 30(18):R1034–R5. https://doi.org/10.1016/j.cub.2020.08.038 PMID: 32961155

34.

Jun JJ, Steinmetz NA, Siegle JH, Denman DJ, Bauza M, Barbarits B, et al. Fully integrated silicon

probes for high-density recording of neural activity. Nature. 2017; 551(7679):232–6. https://doi.org/10.

1038/nature24636 PMID: 29120427

35.

Steinmetz NA, Aydin C, Lebedeva A, Okun M, Pachitariu M, Bauza M, et al. Neuropixels 2.0: A miniaturized high-density probe for stable, long-term brain recordings. Science. 2021; 372(6539). https://doi.

org/10.1126/science.abf4588 PMID: 33859006

36.

Ghosh KK, Burns LD, Cocker ED, Nimmerjahn A, Ziv Y, Gamal AE, et al. Miniaturized integration of a

fluorescence microscope. Nat Methods. 2011; 8(10):871–8. https://doi.org/10.1038/nmeth.1694 PMID:

21909102

37.

Matsumiya LC, Sorge RE, Sotocinal SG, Tabaka JM, Wieskopf JS, Zaloum A, et al. Using the Mouse

Grimace Scale to reevaluate the efficacy of postoperative analgesics in laboratory mice. J Am Assoc

Lab Anim Sci. 2012; 51(1):42–9. PMID: 22330867

38.

Crawley JN, Belknap JK, Collins A, Crabbe JC, Frankel W, Henderson N, et al. Behavioral phenotypes

of inbred mouse strains: implications and recommendations for molecular studies. Psychopharmacology (Berl). 1997; 132(2):107–24. https://doi.org/10.1007/s002130050327 PMID: 9266608

PLOS ONE | https://doi.org/10.1371/journal.pone.0288930 July 20, 2023

13 / 14

PLOS ONE

Deep learning-powered classifier for mouse facial expressions

39.

Yin L, Wei X, Sun Y, Wang J, Rosato MJ. A 3D facial expression database for facial behavior research.

7th International Conference on Automatic Face and Gesture Recognition. 2006.

40.

Simola N, Granon S. Ultrasonic vocalizations as a tool in studying emotional states in rodent models of

social behavior and brain disease. Neuropharmacology. 2019; 159:107420. https://doi.org/10.1016/j.

neuropharm.2018.11.008 PMID: 30445100

41.

Coffey KR, Marx RE, Neumaier JF. DeepSqueak: a deep learning-based system for detection and analysis of ultrasonic vocalizations. Neuropsychopharmacology. 2019; 44(5):859–68. https://doi.org/10.

1038/s41386-018-0303-6 PMID: 30610191

42.

Krizhevsky A, Sutskever I, Hinton GE. ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the 25th International Conference on Neural Information Processing Systems.

2012;1:1097–105.

43.

Alzubaidi L, Zhang J, Humaidi AJ, Al-Dujaili A, Duan Y, Al-Shamma O, et al. Review of deep learning:

concepts, CNN architectures, challenges, applications, future directions. J Big Data. 2021; 8(1):53.

https://doi.org/10.1186/s40537-021-00444-8 PMID: 33816053

44.

Mathis A, Mamidanna P, Cury KM, Abe T, Murthy VN, Mathis MW, et al. DeepLabCut: markerless pose

estimation of user-defined body parts with deep learning. Nat Neurosci. 2018; 21(9):1281–9. https://doi.

org/10.1038/s41593-018-0209-y PMID: 30127430

45.

Sangiamo DT, Warren MR, Neunuebel JP. Ultrasonic signals associated with different types of social

behavior of mice. Nat Neurosci. 2020; 23(3):411–22. https://doi.org/10.1038/s41593-020-0584-z PMID:

32066980

46.

Mathis MW, Mathis A. Deep learning tools for the measurement of animal behavior in neuroscience.

Curr Opin Neurobiol. 2020; 60:1–11. https://doi.org/10.1016/j.conb.2019.10.008 PMID: 31791006

PLOS ONE | https://doi.org/10.1371/journal.pone.0288930 July 20, 2023

14 / 14

...