AI, Deep Learning, and Machine Learning in the Dynamic World of Spectroscopy

Highlighting artificial intelligence and data analysis applications © Gophotograph - stock.adobe.com

The Future of Chemometrics—Data-Driven Measurements and Instruments for Chemistry

In an Analytically Speaking episode entitled, “The Future of Chemometrics—Data-Driven Measurements and Instruments for Chemistry,” podcast host Jerry Workman talks to talks to Karl Booksh a professor in the Department of Chemistry and Biochemistry at the University of Delaware, Newark, to discuss a National Science Foundation (NSF) workshop organized with Barry Lavine entitled, “Data-Driven Measurements and Instruments for Chemistry” (1).We spoke to Booksh about his organizing this NSF workshop to explore research on the development of portable chemical sensors for environmental, biomedical, and industrial process monitoring and the type of data analysis requirements for these sensors. Booksh’s own research is predicated on the belief that it is better to build small chemical sensors capable of reliable measurements in the field or in the process than to collect samples for future laboratory analysis (1).

Artificial Intelligence in Analytical Spectroscopy, Part I: Basic Concepts and Discussion

A two-part Chemometrics in Spectroscopy series on artificial intelligence (AI), and its subfield machine learning (ML), was published in February and June 2023 (2,3). We noted that AI and ML are major buzz-terms in today’s technology world. In this two-part series, we begin to take a look under the hood and behind the scenes to see what AI is and its applicability to analytical chemistry and spectroscopy for future discussion and elaboration. What are the benefits and limitations, as well as the praises and detractions, of AI? How does AI relate to chemometrics?

As with any modeling technique, it might serve us well to remember Robert A. Heinlein’s reference: “There ain’t no such thing as a free lunch (TANSTAAFL)”, and supercalibratestatisticexpedientalgorithmadnauseous, a play-on word inspired by the wisdom of Mary Poppins, reminding us about the hype of new algorithms. The term was coined to mock the then-current practice of trying all-possible combinations of available algorithms and test statistics to automate the process of developing calibration models for near-infrared (NIR) analysis. It took a long time, but eventually, the NIR community recognized the futility of that methodology (2). Remember, if you are going to make mistakes, automation allows you to make them more quickly and repetitively. One must be cautioned when using powerful and automated regression-based methods to remember that modeling algorithms are only able to build an accurate predictive model based on the information they are presented. As Mark Twain once said, “There are three kinds of lies: lies, damn lies, and statistics” (2). And as Rasmus Bro, a leader in the field of chemometrics, has said:

“As most people are aware, there is currently a hype on machine learning and artificial intelligence (AI), and that is all fine and good. Part of the hype is that these methods are sometimes oversold, and so then people are relieved that now they don’t have to think about things, saying, ‘I don’t have to think about experimental design, sampling error, analytical quality, etc.’ And when that happens, those projects typically fail, because these new and excellent methods don’t actually replace the responsibility for you to actually know what you are doing” (2).

Artificial Intelligence in Analytical Spectroscopy, Part II: Examples in Spectroscopy

In this second part of the two-part Chemometrics in Spectroscopy column series on artificial intelligence (AI), and its subfield machine learning (ML), we presented the variety of chemometric algorithms used to compare AI, ML, and chemometrics. These algorithms included those used for classification, regression, clustering, ensemble learning, signal processing, and component analysis. Now, in Part II, we discuss the applications of AI to electronic and vibrational spectroscopy. We also touch on some applications of deep learning (DL), which is a subfield of machine learning where more complex artificial neural networks (ANNs) with more hidden layers are used (3). This column article includes a number of selected references that discuss the application of AI in analytical chemistry and in molecular spectroscopy. We give a few early and late examples of AI and ML as applied to different vibrational spectroscopy methods, such as Raman, infrared (FT-IR), near-infrared (NIR), and ultraviolet–visible (UV-vis) spectroscopic techniques. This article is intended only as a sampling of the numerous research manuscripts addressing this subject (3).

An Interview with AI About Its Potential Role in Vibrational and Atomic Spectroscopy

In our next AI piece, we interviewed an AI program (ChatGPT) for Spectroscopy asking questions about AI and its role in various applications for vibrational and atomic spectroscopy, including data analysis. For vibrational spectroscopy we asked about Raman, Fourier transform infrared (FT-IR), near-infrared (NIR), ultraviolet-visible (UV-vis), terahertz (THz), and nuclear magnetic resonance (NMR). For atomic spectroscopy we inquired about inductively coupled plasma-atomic emission spectroscopy (ICP-AES), inductively coupled plasma-mass spectrometry (ICP-MS), laser-induced breakdown spectroscopy (LIBS), X-ray fluorescence (XRF), and atomic absorption spectroscopy (AAS). We asked 20 general questions for this virtual interview, keeping in mind that our readers want to know all they can about how to use AI for their analytical chemistry needs. We also included a set of published references and further reading materials for those wishing to look more deeply into the subject of AI and spectroscopy (4).

We note that a formal definition of AI does not always indicate the use of neural networks. Neural networks are one of the many techniques used in AI, but they are not the only ones. AI encompasses a wide range of techniques and approaches, including rule-based systems, decision trees, genetic algorithms, and others (2,3). While neural networks have gained prominence in recent years due to their ability to handle complex data and perform well on a variety of tasks, AI as a field is much broader and includes a range of techniques that do not rely on neural networks (4).

New Near-Infrared Machine Learning Technique Identifies Dangerous Blood for Transfusion Safety

In a news article, we described a team of researchers from Gannan Normal University and the Central Blood Station of Ganzhou City in Jiangxi Province, China, who conducted a study using the GaiaSorter "Gaia" hyperspectral sorter to analyze human blood plasma samples. This advanced technology extracts 254 spectral bands of NIR plasma hyperspectral images, from 900 nm to 1700 nm, allowing for precise analysis of the blood's spectral characteristics (5). The study, titled "Detection of Chylous Plasma Based on Machine Learning and Hyperspectral Techniques," was led by Yafei Liu and his colleagues and published in the journal Applied Spectroscopy. The researchers used four different machine learning algorithms to classify the NIR hyperspectral plasma images: decision tree, Gaussian Naive Bayes (GaussianNB), perceptron, and stochastic gradient descent (5).

AI-Based Neural Networks Revolutionize Infrared Spectra Analysis

A researcher from Lomonosov Moscow State University has developed a convolutional neural network (CNN) model for FT-IR spectra recognition. This AI-based system is capable of classifying 17 functional groups and 72 coupling oscillations with remarkable accuracy, providing a significant boost to material analysis in fields like organic chemistry, materials science, and biology (6). In this innovative study, deep learning was employed to streamline the analysis of FT-IR spectra. This technique, which is crucial for identifying chemical compounds and assessing their structures, is traditionally labor-intensive and requires a high level of experience and expertise. By leveraging CNNs, Daniil S. Koshelev, has developed a model that simplifies this process, allowing for faster and more accurate analysis. This research has been published in the journal Applied Spectroscopy (6).

Deep Learning Advances Gas Quantification Analysis in Near-Infrared Dual-Comb Spectroscopy

Researchers from Tsinghua University and Beihang University in Beijing have developed a deep learning-based data processing framework that significantly improves the accuracy of dual-comb absorption spectroscopy (DCAS) in gas quantification analysis. By using a U-net model for etalon removal and a modified U-net combined with traditional methods for baseline extraction, their framework achieves high-fidelity absorbance spectra, even in challenging conditions with complex baselines and etalon effects (7). Dual-comb absorption spectroscopy (DCAS) is a powerful technique for gas absorption analysis, providing broad spectral coverage and high resolution. However, baseline distortions and etalon effects can complicate the interpretation of DCAS results. To address these challenges, researchers Chao Huang, Tianyou Zhang, Xiangchen Kong, Yan Li, and Haoyun Wei from Tsinghua University and Beihang University in Beijing, China have developed a deep learning framework that overcomes these issues, enabling more accurate and reliable gas quantification analysis. This paper was published in the journal Applied Spectroscopy (7).

Light and AI Unite: Raman Breakthrough in Noninvasive Lung Cancer Detection

Harun Hano, Charles H. Lawrie, and Beatriz Suarez and colleagues from the Department of Physics at the University of the Basque Country (UPV/EHU), in Spain; and the IKERBASQUE─Basque Foundation for Science in Spain have published a research paper in the journal ACS Omega describing the use of Raman spectroscopy with specialized data treatment for the diagnosis of lung cancer (8). Lung cancer is the leading cause of cancer-related deaths worldwide, emphasizing the urgent need for reliable and efficient diagnostic methods. Conventional approaches often involve invasive procedures and can be time-consuming, costly, and involve the risk of infection, resulting in potential delay of effective treatment. The current study explores the potential of Raman spectroscopy as a promising noninvasive technique for lung cancer detection (8).

Are We There Yet? Is There Such a Thing as an Expert Calibration System for Vibrational Spectroscopy?

In another Chemometrics in Spectroscopy column we presented a historical perspective on the development of an expert calibration system (ECS) for spectroscopic-based process analytical chemistry would be a significant advancement aimed at automating the creation of high-quality calibration models for standard zero-order and first-order calibrations as well as multidimensional imaging applications (9). The concept of ECS would seek to reduce the reliance on users’ extensive knowledge of chemometrics all the while leveraging their domain knowledge and understanding of specific sample physical and chemical properties. By providing automated tools and guidance, an ECS would aim to streamline the calibration process, improve calibration transfer, enhance operator efficiency, and improve the overall consistency and reliability of analytical results produced using advanced chemometrics and machine earning techniques. In this article, we resurrect the discussion from nearly three decades ago on the potential to automate calibration work (9).

Non-Linear Memory-Based Learning Advances Soil Property Prediction Using vis-NIR Spectral Data

Researchers from Zhejiang University have developed a new non-linear memory-based learning (N-MBL) model that enhances the prediction accuracy of soil properties using visible near-infrared (vis-NIR) spectroscopy. By comparing N-MBL with traditional machine learning and local modeling methods, the study reveals its superior performance, particularly in predicting soil organic matter and total nitrogen (10). In a world where over 700 million people suffer from hunger, according to the Food and Agriculture Organization (FAO), the need for efficient agricultural productivity and soil quality monitoring has never been more urgent. Traditional methods of analyzing soil properties through laboratory testing are often expensive and time-consuming, making them impractical for large-scale applications. Visible near-infrared (vis-NIR) spectroscopy has emerged as a rapid and non-destructive alternative, offering the potential to revolutionize soil analysis. However, to fully harness this technology, robust and accurate predictive models are essential. In a new study, researchers from Zhejiang University in China have developed a non-linear memory-based learning (N-MBL) model that significantly improves the prediction of soil properties from vis-NIR spectral data (10).

AI-Powered Spectroscopy Faces Hurdles in Rapid Food Analysis

A recent study reveals on the challenges and limitations of AI-driven spectroscopy methods for rapid food analysis. Despite the promise of these technologies, issues like small sample sizes, misuse of advanced modeling techniques, and validation problems hinder their effectiveness. The authors suggest guidelines for improving accuracy and reliability in both research and industrial settings. The food industry has seen a surge in the use of rapid, non-destructive analytical methods powered by artificial intelligence (AI) and spectroscopy (11). These methods, which include vibrational spectroscopy and other sensor-based technologies, offer the potential for quick, accurate assessments of food quality and authenticity. However, a new study by Wenyang Jia, Konstantia Georgouli, Jesus Martinez-Del Rincon, and Anastasios Koidis highlights significant challenges in the implementation of these technologies. The research, conducted across institutions like the Institute for Global Food Security at Queen’s University Belfast and the Lawrence Livermore National Laboratory in the USA, was published in the journal Foods and provides a critical overview of the current state of AI-driven food analysis (11).

Machine Learning Used for Meteorite Classification to Unlock Asteroid Composition Mysteries

A team of researchers has developed a new ML method to classify asteroid spectra by analyzing meteorite spectroscopic data. Using logistic regression, the model accurately grouped meteorites into eight categories, helping to better understand the distribution of asteroid composition in the asteroid belt. The study, published in the journal Icarus, opens new avenues for predicting asteroid composition using spectroscopy (12).

Asteroids hold vital clues about the formation and history of our Solar System, but our understanding of their composition has been limited due to the rarity of asteroid sample-return missions. While remote sensing and asteroid imagery offer some insights, detailed interpretation of asteroid surface compositions remains challenging. A new study led by researchers from the Planetary Science Institute, Mount Holyoke College, and the University of Massachusetts Amherst, proposes an innovative solution: using ML to analyze asteroid spectra based on existing meteorite data (12).

Revolutionizing Analytical Chemistry: The AI Breakthrough

AI is reshaping analytical chemistry by enhancing data analysis and optimizing experimental methods. One published study explores AI's advancements, challenges, and future directions for data analysis, emphasizing its transformative potential and the need for ethical considerations. AI has emerged as a transformative force across various scientific fields. In analytical chemistry, AI is revolutionizing the approach to complex data analysis and the development of innovative methods. AI's capabilities in interpreting large data volumes and automating analyses enhance efficiency, accuracy, and reliability, making it a game-changer in this field (13). AI is being applied for rapid spectroscopic food analysis, surface-enhanced Raman spectroscopy (SERS), metabolite profiling using NMR spectroscopy, and many other applications. A recent review by Rafael Cardoso Rial of the Federal Institute of Mato Grosso do Sul in Brazil explores this topic with 134 references in the journal Talanta (13).

Automating Advanced Chemometric Methods for Data Processing

Here in the Analytically Speaking podcast series, Episode #9, podcast host Jerry Workman speaks to guest Rasmus Bro, who is a full professor at the University of Copenhagen and one of the foremost active living data analytics and chemometrics experts. We spoke to Rasmus about the world of data analysis used for spectroscopy and other analytical methods. Over the years he has worked on many aspects of chemometrics, developing numerous algorithms and methods such as fuzzy logic, deep learning, analysis of variance, and tensor modeling. He has received multiple awards in chemometrics and in the analytical sciences, and is the second-most-cited scientist within the field of chemometrics with nearly 37,000 citations and an h-index of 78 (Google Scholar). Most of the algorithms and data sets he has worked on have been made publicly available on the internet. Here we discuss his research on the development and automation of several chemometrics methods for use with any spectroscopic technique (14).

Clarifying the Meaning of Chemometrics, Artificial Intelligence (AI), Machine Learning (ML), and Neural Networks (NNs)

In our December 1, 2024 Analytically Speaking podcast, host Jerry Workman speaks with guest Barry M. Wise, Founder and President of Eigenvector Research, Inc. about the meaning of the terms chemometrics, AI, ML, and neural networks (NNs) within the context of analytical chemistry and process analysis. This informative discussion places a perspective on the new AI techniques from a podcast guest who has been at the heart of chemometrics and software development for decades (15).

References

(1) Analytically Speaking Podcast, Ep. 3: The Future of Chemometrics—Data-Driven Measurements and Instruments for Chemistry. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/ep-3-the-future-of-chemometrics-data-driven-measurements-and-instruments-for-chemistry (accessed 2024-11-22).

(2) Workman, Jr., J.; Mark, H. Artificial Intelligence in Analytical Spectroscopy, Part I: Basic Concepts and Discussion. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/artificial-intelligence-in-analytical-spectroscopy-part-i-basic-concepts-and-discussion (accessed 2024-11-22).

(3) Workman, Jr., J.; Mark, H. Artificial Intelligence in Analytical Spectroscopy, Part II: Examples in Spectroscopy. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/artificial-intelligence-in-analytical-spectroscopy-part-ii-examples-in-spectroscopy

(4) Workman, Jr., J. An Interview with AI About Its Potential Role in Vibrational and Atomic Spectroscopy. Spectroscopyonline.com. Available at: Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/an-interview-with-ai-about-its-potential-role-in-vibrational-and-atomic-spectroscopy (accessed 2024-11-22).

(5) Workman, Jr., J. New Near-Infrared Machine Learning Technique Identifies Dangerous Blood for Transfusion Safety. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/new-near-infrared-machine-learning-technique-identifies-dangerous-blood-for-transfusion-safety

(accessed 2024-11-22).

(6) Workman, Jr., J. AI-Based Neural Networks Revolutionize Infrared Spectra Analysis. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/ai-based-neural-networks-revolutionize-infrared-spectra-analysis (accessed 2024-11-22).

(7) Workman, Jr., J. Deep Learning Advances Gas Quantification Analysis in Near-Infrared Dual-Comb Spectroscopy. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/deep-learning-advances-gas-quantification-analysis-in-near-infrared-dual-comb-spectroscopy (accessed 2024-11-22).

(8) Workman, Jr., J. Light and AI Unite: Raman Breakthrough in Noninvasive Lung Cancer Detection. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/light-and-ai-unite-raman-breakthrough-in-noninvasive-lung-cancer-detection (accessed 2024-11-22).

(9) Workman, Jr., J.; Mark, H. Are We There Yet? Is There Such a Thing as an Expert Calibration System for Vibrational Spectroscopy? Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/are-we-there-yet-is-there-such-a-thing-as-an-expert-calibration-system-for-vibrational-spectroscopy- (accessed 2024-11-22).

(10) Workman, Jr., J. Non-Linear Memory-Based Learning Advances Soil Property Prediction Using vis-NIR Spectral Data. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/non-linear-memory-based-learning-advances-soil-property-prediction-using-vis-nir-spectral-data (accessed 2024-11-22).

(11) Workman, Jr., J. AI-Powered Spectroscopy Faces Hurdles in Rapid Food Analysis. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/ai-powered-spectroscopy-faces-hurdles-in-rapid-food-analysis (accessed 2024-11-22).

(12) Workman, Jr., J. Machine Learning Used for Meteorite Classification to Unlock Asteroid Composition Mysteries. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/machine-learning-used-for-meteorite-classification-to-unlock-asteroid-composition-mysteries (accessed 2024-11-22).

(13) Workman, Jr., J. Revolutionizing Analytical Chemistry: The AI Breakthrough. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/revolutionizing-analytical-chemistry-the-ai-breakthrough (accessed 2024-11-22).

(14) Analytically Speaking Podcast, Episode #9: Automating Advanced Chemometric Methods for Data Processing. Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/view/ep-9-automating-advanced-chemometric-methods-for-data-processing (accessed 2024-11-26).

(15) Analytically Speaking Podcast, Episode #31: Clarifying the Meaning of Chemometrics, Artificial Intelligence (AI), Machine Learning (ML), and Neural Networks (NNs). Spectroscopyonline.com. Available at: https://www.spectroscopyonline.com/analytically-speaking-podcast (accessed 2024-11-22).