Ultrathin Gallium Nitride Quantum-Disk-in-Nanowire-Enabled Reconfigurable Bioinspired Sensor for High-Accuracy Human Action Recognition
Corresponding Author: Haiding Sun
Nano-Micro Letters,
Vol. 18 (2026), Article Number: 54
Abstract
Human action recognition (HAR) is crucial for the development of efficient computer vision, where bioinspired neuromorphic perception visual systems have emerged as a vital solution to address transmission bottlenecks across sensor-processor interfaces. However, the absence of interactions among versatile biomimicking functionalities within a single device, which was developed for specific vision tasks, restricts the computational capacity, practicality, and scalability of in-sensor vision computing. Here, we propose a bioinspired vision sensor composed of a GaN/AlN-based ultrathin quantum-disks-in-nanowires (QD-NWs) array to mimic not only Parvo cells for high-contrast vision and Magno cells for dynamic vision in the human retina but also the synergistic activity between the two cells for in-sensor vision computing. By simply tuning the applied bias voltage on each QD-NW-array-based pixel, we achieve two biosimilar photoresponse characteristics with slow and fast reactions to light stimuli that enhance the in-sensor image quality and HAR efficiency, respectively. Strikingly, the interplay and synergistic interaction of the two photoresponse modes within a single device markedly increased the HAR recognition accuracy from 51.4% to 81.4% owing to the integrated artificial vision system. The demonstration of an intelligent vision sensor offers a promising device platform for the development of highly efficient HAR systems and future smart optoelectronics.
Highlights:
1 A novel GaN/AlN-based ultrathin quantum-disks-in-nanowires sensor was fabricated, demonstrating voltage bias tunable response characteristics to light stimuli.
2 Image enhancement functionality and a robust reservoir computing system were demonstrated based on the voltage tunable long-term and short-term persistent photocurrent respectively.
3 Furthermore, a high-performance artificial vision system with the two integrated functions was demonstrated, achieving a remarkable improvement in human action recognition.
Keywords
Download Citation
Endnote/Zotero/Mendeley (RIS)BibTeX
- P. Antonik, N. Marsal, D. Brunner, D. Rontani, Human action recognition with a large-scale brain-inspired photonic computer. Nat. Mach. Intell. 1(11), 530–537 (2019). https://doi.org/10.1038/s42256-019-0110-8
- E. Picco, P. Antonik, S. Massar, High speed human action recognition using a photonic reservoir computer. Neural Netw. 165, 662–675 (2023). https://doi.org/10.1016/j.neunet.2023.06.014
- J. Chen, Z. Zhou, B.J. Kim, Y. Zhou, Z. Wang et al., Optoelectronic graded neurons for bioinspired in-sensor motion perception. Nat. Nanotechnol. 18(8), 882–888 (2023). https://doi.org/10.1038/s41565-023-01379-2
- M. Kim, X. Jiang, K. Lauter, E. Ismayilzada, S. Shams, Secure human action recognition by encrypted neural network inference. Nat. Commun. 13(1), 4799 (2022). https://doi.org/10.1038/s41467-022-32168-5
- M. Lu, Y. Hu, X. Lu, Driver action recognition using deformable and dilated faster R-CNN with optimized region proposals. Appl. Intell. 50(4), 1100–1111 (2020). https://doi.org/10.1007/s10489-019-01603-4
- W. Hu, D. Xie, Z. Fu, W. Zeng, S. Maybank, Semantic-based surveillance video retrieval. IEEE Trans. Image Process. 16(4), 1168–1181 (2007). https://doi.org/10.1109/TIP.2006.891352l
- I. Rodomagoulakis, N. Kardaris, V. Pitsikalis, E. Mavroudi, A. Katsamanis et al., Multimodal human action recognition in assistive human–robot interaction. Presented at 2016 IEEE international conference on acoustics speech signal process (ICASSP), pp 2702–2706 (2016). https://doi.org/10.1109/ICASSP.2016.7472168
- M. Vrigkas, C. Nikou, I.A. Kakadiaris, A review of human activity recognition methods. Front. Robot. AI 2, 28 (2015). https://doi.org/10.3389/frobt.2015.00028
- A. Elgammal, R. Duraiswami, D. Harwood, L.S. Davis, Background and foreground modeling using nonparametric kernel density estimation for visual surveillance. Proc. IEEE 90(7), 1151–1163 (2002). https://doi.org/10.1109/JPROC.2002.801448
- Y. Zhou, J. Fu, Z. Chen, F. Zhuge, Y. Wang et al., Computational event-driven vision sensors for in-sensor spiking neural networks. Nat. Electron. 6(11), 870–878 (2023). https://doi.org/10.1038/s41928-023-01055-2
- C. Li, Z. Wang, M. Rao, D. Belkin, W. Song et al., Long short-term memory networks in memristor crossbar arrays. Nat. Mach. Intell. 1(1), 49–57 (2019). https://doi.org/10.1038/s42256-018-0001-4
- J. Schmidhuber, D. Wierstra, F.J. Gomez, Presented at international joint conference on artificial intelligence, Evolino: hybrid neuroevolution/optimal linear search for sequence learning (Morgan Kaufmann, San Francisco, 2005). https://api.semanticscholar.org/CorpusID:6183435
- W. Bao, J. Yue, Y. Rao, A deep learning framework for financial time series using stacked autoencoders and long-short term memory. PLoS ONE 12(7), e0180944 (2017). https://doi.org/10.1371/journal.pone.0180944
- Z. Sun, Q. Ke, H. Rahmani, M. Bennamoun, G. Wang et al., Human action recognition from various data modalities: a review. IEEE Trans. Pattern Anal. Mach. Intell. 45(3), 3200–3225 (2023). https://doi.org/10.1109/TPAMI.2022.3183112
- Y.M. Song, Y. Xie, V. Malyarchuk, J. Xiao, I. Jung et al., Digital cameras with designs inspired by the arthropod eye. Nature 497(7447), 95–99 (2013). https://doi.org/10.1038/nature12083
- H.C. Ko, M.P. Stoykovich, J. Song, V. Malyarchuk, W.M. Choi et al., A hemispherical electronic eye camera based on compressible silicon optoelectronics. Nature 454(7205), 748–753 (2008). https://doi.org/10.1038/nature07113
- K.-H. Jeong, J. Kim, L.P. Lee, Biologically inspired artificial compound eyes. Science 312(5773), 557–561 (2006). https://doi.org/10.1126/science.1123053
- Y. Kim, A. Chortos, W. Xu, Y. Liu, J.Y. Oh et al., A bioinspired flexible organic artificial afferent nerve. Science 360(6392), 998–1003 (2018). https://doi.org/10.1126/science.aao0098
- A.G. Leventhal, R.W. Rodieck, B. Dreher, Retinal ganglion cell classes in the old world monkey: morphology and central projections. Science 213(4512), 1139–1142 (1981). https://doi.org/10.1126/science.7268423
- M. Livingstone, D. Hubel, Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science 240(4853), 740–749 (1988). https://doi.org/10.1126/science.3283936
- R. Shapley, E. Kaplan, R. Soodak, Spatial summation and contrast sensitivity of X and Y cells in the lateral geniculate nucleus of the macaque. Nature 292(5823), 543–545 (1981). https://doi.org/10.1038/292543a0
- D. Wang, X. Liu, Y. Kang, X. Wang, Y. Wu et al., Bidirectional photocurrent in p–n heterojunction nanowires. Nat. Electron. 4(9), 645–652 (2021). https://doi.org/10.1038/s41928-021-00640-7
- H. Yu, R. Wang, M.H. Memon, Y. Luo, S. Xiao et al., Highly responsive switchable broadband DUV-NIR photodetector and tunable emitter enabled by uniform and vertically grown III–V nanowire on silicon substrate for integrated photonics. Small 20(10), 2307458 (2024). https://doi.org/10.1002/smll.202307458
- W. Chen, D. Wang, W. Wang, Y. Kang, X. Liu et al., Manipulating surface band bending of III-nitride nanowires with ambipolar charge-transfer characteristics: a pathway toward advanced photoswitching logic gates and encrypted optical communication. Adv. Mater. 36(1), 2307779 (2024). https://doi.org/10.1002/adma.202307779
- T. Kawashima, H. Yoshikawa, S. Adachi, S. Fuke, K. Ohtsuka, Optical properties of hexagonal GaN. J. Appl. Phys. 82(7), 3528–3535 (1997). https://doi.org/10.1063/1.365671
- Z. Zhang, M. Kushimoto, T. Sakai, N. Sugiyama, L.J. Schowalter et al., Design and characterization of a low-optical-loss UV-C laser diode. Jpn. J. Appl. Phys. 59(9), 094001 (2020). https://doi.org/10.35848/1347-4065/abaac6
- T. Baden, T. Euler, P. Berens, Understanding the retinal basis of vision across species. Nat. Rev. Neurosci. 21(1), 5–20 (2020). https://doi.org/10.1038/s41583-019-0242-1
- L. Li, S. Fang, W. Chen, Y. Li, M.F. Vafadar et al., Facile semiconductor p-n homojunction nanowires with strategic p-type doping engineering combined with surface reconstruction for biosensing applications. Nano-Micro Lett. 16(1), 192 (2024). https://doi.org/10.1007/s40820-024-01394-5
- Y. Kang, D. Wang, A. Wang, W. Chen, B. Liu et al., Light-induced adaptive structural evolution in gallium nitride nanowire/nickel hydroxide symbiotic system in photoelectrochemical environment. Adv. Funct. Mater. 34(7), 2311223 (2024). https://doi.org/10.1002/adfm.202311223
- Y. Kang, D. Wang, Y. Gao, S. Guo, K. Hu et al., Achieving record-high photoelectrochemical photoresponse characteristics by employing Co3O4 nanoclusters as hole charging layer for underwater optical communication. ACS Nano 17(4), 3901–3912 (2023). https://doi.org/10.1021/acsnano.2c12175
- Y. Wu, Y. Wang, K. Sun, Z. Mi, Molecular beam epitaxy and characterization of AlGaN nanowire ultraviolet light emitting diodes on Al coated Si (0 0 1) substrate. J. Cryst. Growth 507, 65–69 (2019). https://doi.org/10.1016/j.jcrysgro.2018.10.028
- S. Cheng, Z. Wu, B. Langelier, X. Kong, T. Coenen et al., Nanoscale structural and emission properties within “Russian doll”-type InGaN/AlGaN quantum wells. Adv. Opt. Mater. 8(17), 2000481 (2020). https://doi.org/10.1002/adom.202000481
- A.D. Andreev, E.P. O’Reilly, Theory of the electronic structure of GaN/AlN hexagonal quantum dots. Phys. Rev. B 62(23), 15851–15870 (2000). https://doi.org/10.1103/physrevb.62.15851
- B. Sheng, F. Bertram, G. Schmidt, P. Veit, M. Müller et al., Cathodoluminescence nano-characterization of individual GaN/AlN quantum disks embedded in nanowires. Appl. Phys. Lett. 117(13), 133106 (2020). https://doi.org/10.1063/5.0024110
- R.C. Atkinson, R.M. Shiffrin, Human memory: a proposed system and its control processes. in Psychology of Learning and Motivation (Elsevier, 1968). https://doi.org/10.1016/s0079-7421(08)60422-3
- Z. Gao, X. Ju, H. Zhang, X. Liu, H. Chen et al., InP quantum dots tailored oxide thin film phototransistor for bioinspired visual adaptation. Adv. Funct. Mater. 33(52), 2305959 (2023). https://doi.org/10.1002/adfm.202305959
- X. Liu, D. Wang, W. Chen, Y. Kang, S. Fang et al., Optoelectronic synapses with chemical-electric behaviors in gallium nitride semiconductors for biorealistic neuromorphic functionality. Nat. Commun. 15(1), 7671 (2024). https://doi.org/10.1038/s41467-024-51194-z
- Z. Zhang, X. Zhao, X. Zhang, X. Hou, X. Ma et al., In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array. Nat. Commun. 13(1), 6590 (2022). https://doi.org/10.1038/s41467-022-34230-8
- X. Ji, B.D. Paulsen, G.K.K. Chik, R. Wu, Y. Yin et al., Mimicking associative learning using an ion-trapping non-volatile synaptic organic electrochemical transistor. Nat. Commun. 12(1), 2480 (2021). https://doi.org/10.1038/s41467-021-22680-5
- Y. Ren, X. Bu, M. Wang, Y. Gong, J. Wang et al., Synaptic plasticity in self-powered artificial striate cortex for binocular orientation selectivity. Nat. Commun. 13(1), 5585 (2022). https://doi.org/10.1038/s41467-022-33393-8
- T.-J. Lee, K.-R. Yun, S.-K. Kim, J.-H. Kim, J. Jin et al., Realization of an artificial visual nervous system using an integrated optoelectronic device array. Adv. Mater. 33(51), 2105485 (2021). https://doi.org/10.1002/adma.202105485
- V. Ayzenberg, M. Behrmann, Development of visual object recognition. Nat. Rev. Psychol. 3(2), 73–90 (2024). https://doi.org/10.1038/s44159-023-00266-w
- L. Gorelick, M. Blank, E. Shechtman, M. Irani, R. Basri, Actions as space-time shapes. IEEE Trans. Pattern Anal. Mach. Intell. 29(12), 2247–2253 (2007). https://doi.org/10.1109/TPAMI.2007.70711
References
P. Antonik, N. Marsal, D. Brunner, D. Rontani, Human action recognition with a large-scale brain-inspired photonic computer. Nat. Mach. Intell. 1(11), 530–537 (2019). https://doi.org/10.1038/s42256-019-0110-8
E. Picco, P. Antonik, S. Massar, High speed human action recognition using a photonic reservoir computer. Neural Netw. 165, 662–675 (2023). https://doi.org/10.1016/j.neunet.2023.06.014
J. Chen, Z. Zhou, B.J. Kim, Y. Zhou, Z. Wang et al., Optoelectronic graded neurons for bioinspired in-sensor motion perception. Nat. Nanotechnol. 18(8), 882–888 (2023). https://doi.org/10.1038/s41565-023-01379-2
M. Kim, X. Jiang, K. Lauter, E. Ismayilzada, S. Shams, Secure human action recognition by encrypted neural network inference. Nat. Commun. 13(1), 4799 (2022). https://doi.org/10.1038/s41467-022-32168-5
M. Lu, Y. Hu, X. Lu, Driver action recognition using deformable and dilated faster R-CNN with optimized region proposals. Appl. Intell. 50(4), 1100–1111 (2020). https://doi.org/10.1007/s10489-019-01603-4
W. Hu, D. Xie, Z. Fu, W. Zeng, S. Maybank, Semantic-based surveillance video retrieval. IEEE Trans. Image Process. 16(4), 1168–1181 (2007). https://doi.org/10.1109/TIP.2006.891352l
I. Rodomagoulakis, N. Kardaris, V. Pitsikalis, E. Mavroudi, A. Katsamanis et al., Multimodal human action recognition in assistive human–robot interaction. Presented at 2016 IEEE international conference on acoustics speech signal process (ICASSP), pp 2702–2706 (2016). https://doi.org/10.1109/ICASSP.2016.7472168
M. Vrigkas, C. Nikou, I.A. Kakadiaris, A review of human activity recognition methods. Front. Robot. AI 2, 28 (2015). https://doi.org/10.3389/frobt.2015.00028
A. Elgammal, R. Duraiswami, D. Harwood, L.S. Davis, Background and foreground modeling using nonparametric kernel density estimation for visual surveillance. Proc. IEEE 90(7), 1151–1163 (2002). https://doi.org/10.1109/JPROC.2002.801448
Y. Zhou, J. Fu, Z. Chen, F. Zhuge, Y. Wang et al., Computational event-driven vision sensors for in-sensor spiking neural networks. Nat. Electron. 6(11), 870–878 (2023). https://doi.org/10.1038/s41928-023-01055-2
C. Li, Z. Wang, M. Rao, D. Belkin, W. Song et al., Long short-term memory networks in memristor crossbar arrays. Nat. Mach. Intell. 1(1), 49–57 (2019). https://doi.org/10.1038/s42256-018-0001-4
J. Schmidhuber, D. Wierstra, F.J. Gomez, Presented at international joint conference on artificial intelligence, Evolino: hybrid neuroevolution/optimal linear search for sequence learning (Morgan Kaufmann, San Francisco, 2005). https://api.semanticscholar.org/CorpusID:6183435
W. Bao, J. Yue, Y. Rao, A deep learning framework for financial time series using stacked autoencoders and long-short term memory. PLoS ONE 12(7), e0180944 (2017). https://doi.org/10.1371/journal.pone.0180944
Z. Sun, Q. Ke, H. Rahmani, M. Bennamoun, G. Wang et al., Human action recognition from various data modalities: a review. IEEE Trans. Pattern Anal. Mach. Intell. 45(3), 3200–3225 (2023). https://doi.org/10.1109/TPAMI.2022.3183112
Y.M. Song, Y. Xie, V. Malyarchuk, J. Xiao, I. Jung et al., Digital cameras with designs inspired by the arthropod eye. Nature 497(7447), 95–99 (2013). https://doi.org/10.1038/nature12083
H.C. Ko, M.P. Stoykovich, J. Song, V. Malyarchuk, W.M. Choi et al., A hemispherical electronic eye camera based on compressible silicon optoelectronics. Nature 454(7205), 748–753 (2008). https://doi.org/10.1038/nature07113
K.-H. Jeong, J. Kim, L.P. Lee, Biologically inspired artificial compound eyes. Science 312(5773), 557–561 (2006). https://doi.org/10.1126/science.1123053
Y. Kim, A. Chortos, W. Xu, Y. Liu, J.Y. Oh et al., A bioinspired flexible organic artificial afferent nerve. Science 360(6392), 998–1003 (2018). https://doi.org/10.1126/science.aao0098
A.G. Leventhal, R.W. Rodieck, B. Dreher, Retinal ganglion cell classes in the old world monkey: morphology and central projections. Science 213(4512), 1139–1142 (1981). https://doi.org/10.1126/science.7268423
M. Livingstone, D. Hubel, Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science 240(4853), 740–749 (1988). https://doi.org/10.1126/science.3283936
R. Shapley, E. Kaplan, R. Soodak, Spatial summation and contrast sensitivity of X and Y cells in the lateral geniculate nucleus of the macaque. Nature 292(5823), 543–545 (1981). https://doi.org/10.1038/292543a0
D. Wang, X. Liu, Y. Kang, X. Wang, Y. Wu et al., Bidirectional photocurrent in p–n heterojunction nanowires. Nat. Electron. 4(9), 645–652 (2021). https://doi.org/10.1038/s41928-021-00640-7
H. Yu, R. Wang, M.H. Memon, Y. Luo, S. Xiao et al., Highly responsive switchable broadband DUV-NIR photodetector and tunable emitter enabled by uniform and vertically grown III–V nanowire on silicon substrate for integrated photonics. Small 20(10), 2307458 (2024). https://doi.org/10.1002/smll.202307458
W. Chen, D. Wang, W. Wang, Y. Kang, X. Liu et al., Manipulating surface band bending of III-nitride nanowires with ambipolar charge-transfer characteristics: a pathway toward advanced photoswitching logic gates and encrypted optical communication. Adv. Mater. 36(1), 2307779 (2024). https://doi.org/10.1002/adma.202307779
T. Kawashima, H. Yoshikawa, S. Adachi, S. Fuke, K. Ohtsuka, Optical properties of hexagonal GaN. J. Appl. Phys. 82(7), 3528–3535 (1997). https://doi.org/10.1063/1.365671
Z. Zhang, M. Kushimoto, T. Sakai, N. Sugiyama, L.J. Schowalter et al., Design and characterization of a low-optical-loss UV-C laser diode. Jpn. J. Appl. Phys. 59(9), 094001 (2020). https://doi.org/10.35848/1347-4065/abaac6
T. Baden, T. Euler, P. Berens, Understanding the retinal basis of vision across species. Nat. Rev. Neurosci. 21(1), 5–20 (2020). https://doi.org/10.1038/s41583-019-0242-1
L. Li, S. Fang, W. Chen, Y. Li, M.F. Vafadar et al., Facile semiconductor p-n homojunction nanowires with strategic p-type doping engineering combined with surface reconstruction for biosensing applications. Nano-Micro Lett. 16(1), 192 (2024). https://doi.org/10.1007/s40820-024-01394-5
Y. Kang, D. Wang, A. Wang, W. Chen, B. Liu et al., Light-induced adaptive structural evolution in gallium nitride nanowire/nickel hydroxide symbiotic system in photoelectrochemical environment. Adv. Funct. Mater. 34(7), 2311223 (2024). https://doi.org/10.1002/adfm.202311223
Y. Kang, D. Wang, Y. Gao, S. Guo, K. Hu et al., Achieving record-high photoelectrochemical photoresponse characteristics by employing Co3O4 nanoclusters as hole charging layer for underwater optical communication. ACS Nano 17(4), 3901–3912 (2023). https://doi.org/10.1021/acsnano.2c12175
Y. Wu, Y. Wang, K. Sun, Z. Mi, Molecular beam epitaxy and characterization of AlGaN nanowire ultraviolet light emitting diodes on Al coated Si (0 0 1) substrate. J. Cryst. Growth 507, 65–69 (2019). https://doi.org/10.1016/j.jcrysgro.2018.10.028
S. Cheng, Z. Wu, B. Langelier, X. Kong, T. Coenen et al., Nanoscale structural and emission properties within “Russian doll”-type InGaN/AlGaN quantum wells. Adv. Opt. Mater. 8(17), 2000481 (2020). https://doi.org/10.1002/adom.202000481
A.D. Andreev, E.P. O’Reilly, Theory of the electronic structure of GaN/AlN hexagonal quantum dots. Phys. Rev. B 62(23), 15851–15870 (2000). https://doi.org/10.1103/physrevb.62.15851
B. Sheng, F. Bertram, G. Schmidt, P. Veit, M. Müller et al., Cathodoluminescence nano-characterization of individual GaN/AlN quantum disks embedded in nanowires. Appl. Phys. Lett. 117(13), 133106 (2020). https://doi.org/10.1063/5.0024110
R.C. Atkinson, R.M. Shiffrin, Human memory: a proposed system and its control processes. in Psychology of Learning and Motivation (Elsevier, 1968). https://doi.org/10.1016/s0079-7421(08)60422-3
Z. Gao, X. Ju, H. Zhang, X. Liu, H. Chen et al., InP quantum dots tailored oxide thin film phototransistor for bioinspired visual adaptation. Adv. Funct. Mater. 33(52), 2305959 (2023). https://doi.org/10.1002/adfm.202305959
X. Liu, D. Wang, W. Chen, Y. Kang, S. Fang et al., Optoelectronic synapses with chemical-electric behaviors in gallium nitride semiconductors for biorealistic neuromorphic functionality. Nat. Commun. 15(1), 7671 (2024). https://doi.org/10.1038/s41467-024-51194-z
Z. Zhang, X. Zhao, X. Zhang, X. Hou, X. Ma et al., In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array. Nat. Commun. 13(1), 6590 (2022). https://doi.org/10.1038/s41467-022-34230-8
X. Ji, B.D. Paulsen, G.K.K. Chik, R. Wu, Y. Yin et al., Mimicking associative learning using an ion-trapping non-volatile synaptic organic electrochemical transistor. Nat. Commun. 12(1), 2480 (2021). https://doi.org/10.1038/s41467-021-22680-5
Y. Ren, X. Bu, M. Wang, Y. Gong, J. Wang et al., Synaptic plasticity in self-powered artificial striate cortex for binocular orientation selectivity. Nat. Commun. 13(1), 5585 (2022). https://doi.org/10.1038/s41467-022-33393-8
T.-J. Lee, K.-R. Yun, S.-K. Kim, J.-H. Kim, J. Jin et al., Realization of an artificial visual nervous system using an integrated optoelectronic device array. Adv. Mater. 33(51), 2105485 (2021). https://doi.org/10.1002/adma.202105485
V. Ayzenberg, M. Behrmann, Development of visual object recognition. Nat. Rev. Psychol. 3(2), 73–90 (2024). https://doi.org/10.1038/s44159-023-00266-w
L. Gorelick, M. Blank, E. Shechtman, M. Irani, R. Basri, Actions as space-time shapes. IEEE Trans. Pattern Anal. Mach. Intell. 29(12), 2247–2253 (2007). https://doi.org/10.1109/TPAMI.2007.70711