Artificial General Intelligence

commentaires · 41 Vues

Artificial general intelligence (AGI) is a type of expert system (AI) that matches or exceeds human cognitive abilities across a wide variety of cognitive jobs.

Artificial general intelligence (AGI) is a type of expert system (AI) that matches or goes beyond human cognitive capabilities throughout a wide variety of cognitive tasks. This contrasts with narrow AI, which is limited to specific jobs. [1] Artificial superintelligence (ASI), on the other hand, describes AGI that significantly goes beyond human cognitive capabilities. AGI is considered one of the meanings of strong AI.


Creating AGI is a primary objective of AI research study and of business such as OpenAI [2] and Meta. [3] A 2020 survey identified 72 active AGI research and advancement projects across 37 countries. [4]

The timeline for accomplishing AGI remains a topic of continuous dispute amongst scientists and specialists. As of 2023, some argue that it might be possible in years or years; others preserve it may take a century or longer; a minority believe it may never ever be accomplished; and another minority claims that it is already here. [5] [6] Notable AI scientist Geoffrey Hinton has expressed issues about the quick progress towards AGI, recommending it could be attained earlier than many expect. [7]

There is argument on the precise definition of AGI and concerning whether contemporary big language models (LLMs) such as GPT-4 are early kinds of AGI. [8] AGI is a common subject in sci-fi and futures studies. [9] [10]

Contention exists over whether AGI represents an existential threat. [11] [12] [13] Many professionals on AI have actually specified that reducing the threat of human termination presented by AGI needs to be a global concern. [14] [15] Others discover the advancement of AGI to be too remote to provide such a danger. [16] [17]

Terminology


AGI is also referred to as strong AI, [18] [19] full AI, [20] human-level AI, [5] human-level intelligent AI, or higgledy-piggledy.xyz basic intelligent action. [21]

Some academic sources book the term "strong AI" for computer system programs that experience sentience or awareness. [a] In contrast, weak AI (or narrow AI) is able to resolve one particular issue however does not have basic cognitive abilities. [22] [19] Some academic sources use "weak AI" to refer more broadly to any programs that neither experience awareness nor have a mind in the exact same sense as people. [a]

Related ideas include artificial superintelligence and transformative AI. An artificial superintelligence (ASI) is a hypothetical type of AGI that is a lot more generally smart than people, [23] while the notion of transformative AI connects to AI having a big influence on society, for example, similar to the agricultural or commercial revolution. [24]

A framework for rocksoff.org categorizing AGI in levels was proposed in 2023 by Google DeepMind scientists. They specify 5 levels of AGI: emerging, competent, specialist, virtuoso, and superhuman. For example, a qualified AGI is defined as an AI that outshines 50% of skilled grownups in a large range of non-physical tasks, and a superhuman AGI (i.e. an artificial superintelligence) is likewise specified but with a limit of 100%. They consider large language models like ChatGPT or LLaMA 2 to be instances of emerging AGI. [25]

Characteristics


Various popular meanings of intelligence have been proposed. One of the leading propositions is the Turing test. However, there are other widely known meanings, junkerhq.net and some scientists disagree with the more popular methods. [b]

Intelligence characteristics


Researchers generally hold that intelligence is needed to do all of the following: [27]

factor, usage technique, solve puzzles, and make judgments under uncertainty
represent understanding, consisting of good sense understanding
plan
discover
- communicate in natural language
- if essential, integrate these abilities in conclusion of any given objective


Many interdisciplinary methods (e.g. cognitive science, computational intelligence, and decision making) consider extra qualities such as imagination (the capability to form novel mental images and ideas) [28] and autonomy. [29]

Computer-based systems that exhibit a lot of these abilities exist (e.g. see computational imagination, automated reasoning, decision support system, robotic, evolutionary computation, intelligent representative). There is debate about whether contemporary AI systems possess them to a sufficient degree.


Physical traits


Other capabilities are thought about desirable in smart systems, as they might affect intelligence or aid in its expression. These consist of: [30]

- the ability to sense (e.g. see, hear, and so on), and
- the capability to act (e.g. relocation and manipulate items, modification place to check out, and so on).


This includes the capability to discover and react to hazard. [31]

Although the capability to sense (e.g. see, hear, etc) and the capability to act (e.g. move and manipulate objects, change location to explore, etc) can be desirable for some smart systems, [30] these physical capabilities are not strictly required for an entity to qualify as AGI-particularly under the thesis that big language designs (LLMs) might already be or become AGI. Even from a less optimistic point of view on LLMs, there is no company requirement for an AGI to have a human-like kind; being a silicon-based computational system suffices, supplied it can process input (language) from the external world in location of human senses. This analysis lines up with the understanding that AGI has actually never ever been proscribed a particular physical embodiment and therefore does not demand a capacity for mobility or conventional "eyes and ears". [32]

Tests for human-level AGI


Several tests implied to confirm human-level AGI have actually been thought about, consisting of: [33] [34]

The concept of the test is that the machine needs to try and pretend to be a male, by responding to concerns put to it, and it will only pass if the pretence is reasonably persuading. A substantial portion of a jury, who need to not be expert about makers, should be taken in by the pretence. [37]

AI-complete issues


A problem is informally called "AI-complete" or "AI-hard" if it is thought that in order to solve it, one would need to execute AGI, since the solution is beyond the abilities of a purpose-specific algorithm. [47]

There are lots of problems that have been conjectured to require general intelligence to resolve in addition to people. Examples include computer system vision, natural language understanding, and handling unexpected scenarios while fixing any real-world problem. [48] Even a specific task like translation requires a maker to check out and compose in both languages, follow the author's argument (factor), understand the context (knowledge), and consistently replicate the author's initial intent (social intelligence). All of these problems need to be solved simultaneously in order to reach human-level maker performance.


However, much of these tasks can now be carried out by modern-day big language models. According to Stanford University's 2024 AI index, AI has reached human-level performance on many benchmarks for checking out understanding and visual reasoning. [49]

History


Classical AI


Modern AI research study began in the mid-1950s. [50] The very first generation of AI scientists were encouraged that synthetic basic intelligence was possible and that it would exist in simply a few years. [51] AI pioneer Herbert A. Simon composed in 1965: "devices will be capable, within twenty years, of doing any work a guy can do." [52]

Their forecasts were the motivation for Stanley Kubrick and Arthur C. Clarke's character HAL 9000, who embodied what AI researchers believed they might develop by the year 2001. AI leader Marvin Minsky was a consultant [53] on the task of making HAL 9000 as practical as possible according to the consensus predictions of the time. He said in 1967, "Within a generation ... the problem of producing 'expert system' will significantly be resolved". [54]

Several classical AI tasks, such as Doug Lenat's Cyc project (that began in 1984), and Allen Newell's Soar project, were directed at AGI.


However, in the early 1970s, it ended up being apparent that scientists had actually grossly underestimated the difficulty of the job. Funding agencies ended up being hesitant of AGI and put researchers under increasing pressure to produce helpful "applied AI". [c] In the early 1980s, Japan's Fifth Generation Computer Project restored interest in AGI, setting out a ten-year timeline that consisted of AGI objectives like "carry on a casual discussion". [58] In action to this and the success of expert systems, both market and government pumped money into the field. [56] [59] However, confidence in AI amazingly collapsed in the late 1980s, and the goals of the Fifth Generation Computer Project were never ever fulfilled. [60] For the 2nd time in twenty years, AI researchers who forecasted the imminent accomplishment of AGI had been misinterpreted. By the 1990s, AI scientists had a track record for making vain pledges. They ended up being hesitant to make predictions at all [d] and prevented mention of "human level" expert system for worry of being labeled "wild-eyed dreamer [s]. [62]

Narrow AI research study


In the 1990s and early 21st century, mainstream AI attained business success and scholastic respectability by focusing on specific sub-problems where AI can produce proven results and business applications, such as speech acknowledgment and suggestion algorithms. [63] These "applied AI" systems are now used thoroughly throughout the innovation market, and research in this vein is heavily funded in both academic community and industry. Since 2018 [upgrade], development in this field was considered an emerging trend, and a mature phase was anticipated to be reached in more than 10 years. [64]

At the millenium, many traditional AI scientists [65] hoped that strong AI could be developed by integrating programs that resolve various sub-problems. Hans Moravec composed in 1988:


I am confident that this bottom-up path to expert system will one day fulfill the traditional top-down route majority method, ready to offer the real-world proficiency and the commonsense understanding that has actually been so frustratingly evasive in reasoning programs. Fully smart devices will result when the metaphorical golden spike is driven joining the two efforts. [65]

However, even at the time, this was disputed. For instance, Stevan Harnad of Princeton University concluded his 1990 paper on the symbol grounding hypothesis by mentioning:


The expectation has typically been voiced that "top-down" (symbolic) approaches to modeling cognition will somehow meet "bottom-up" (sensory) approaches somewhere in between. If the grounding considerations in this paper stand, then this expectation is hopelessly modular and there is really only one viable path from sense to signs: from the ground up. A free-floating symbolic level like the software level of a computer will never ever be reached by this path (or vice versa) - nor is it clear why we must even attempt to reach such a level, since it looks as if arriving would simply amount to uprooting our signs from their intrinsic significances (therefore simply minimizing ourselves to the functional equivalent of a programmable computer system). [66]

Modern synthetic basic intelligence research study


The term "synthetic basic intelligence" was used as early as 1997, by Mark Gubrud [67] in a conversation of the ramifications of totally automated military production and operations. A mathematical formalism of AGI was proposed by Marcus Hutter in 2000. Named AIXI, the proposed AGI agent increases "the capability to please objectives in a vast array of environments". [68] This kind of AGI, defined by the capability to increase a mathematical definition of intelligence instead of exhibit human-like behaviour, [69] was likewise called universal synthetic intelligence. [70]

The term AGI was re-introduced and promoted by Shane Legg and Ben Goertzel around 2002. [71] AGI research activity in 2006 was explained by Pei Wang and Ben Goertzel [72] as "producing publications and initial results". The very first summertime school in AGI was arranged in Xiamen, China in 2009 [73] by the Xiamen university's Artificial Brain Laboratory and OpenCog. The very first university course was given up 2010 [74] and 2011 [75] at Plovdiv University, Bulgaria by Todor Arnaudov. MIT presented a course on AGI in 2018, arranged by Lex Fridman and featuring a number of visitor speakers.


As of 2023 [upgrade], a small number of computer system researchers are active in AGI research, and numerous contribute to a series of AGI conferences. However, increasingly more researchers have an interest in open-ended knowing, [76] [77] which is the concept of enabling AI to continually learn and innovate like humans do.


Feasibility


Since 2023, the development and prospective achievement of AGI stays a topic of intense argument within the AI community. While traditional consensus held that AGI was a remote objective, current advancements have led some scientists and industry figures to claim that early types of AGI may currently exist. [78] AI pioneer Herbert A. Simon speculated in 1965 that "devices will be capable, within twenty years, of doing any work a male can do". This forecast stopped working to come real. Microsoft co-founder Paul Allen believed that such intelligence is not likely in the 21st century due to the fact that it would need "unforeseeable and fundamentally unforeseeable advancements" and a "scientifically deep understanding of cognition". [79] Writing in The Guardian, roboticist Alan Winfield declared the gulf in between modern computing and human-level expert system is as large as the gulf in between current area flight and useful faster-than-light spaceflight. [80]

An additional obstacle is the lack of clearness in defining what intelligence involves. Does it need consciousness? Must it display the capability to set goals along with pursue them? Is it purely a matter of scale such that if design sizes increase sufficiently, intelligence will emerge? Are centers such as planning, thinking, and causal understanding needed? Does intelligence need clearly duplicating the brain and its specific faculties? Does it require feelings? [81]

Most AI researchers think strong AI can be attained in the future, however some thinkers, like Hubert Dreyfus and Roger Penrose, deny the possibility of accomplishing strong AI. [82] [83] John McCarthy is amongst those who believe human-level AI will be achieved, however that the present level of progress is such that a date can not accurately be predicted. [84] AI specialists' views on the feasibility of AGI wax and wane. Four polls carried out in 2012 and 2013 recommended that the average quote among professionals for when they would be 50% positive AGI would get here was 2040 to 2050, depending upon the poll, with the mean being 2081. Of the professionals, 16.5% addressed with "never" when asked the exact same concern however with a 90% self-confidence instead. [85] [86] Further present AGI development considerations can be found above Tests for validating human-level AGI.


A report by Stuart Armstrong and Kaj Sotala of the Machine Intelligence Research Institute found that "over [a] 60-year time frame there is a strong predisposition towards anticipating the arrival of human-level AI as between 15 and 25 years from the time the forecast was made". They evaluated 95 predictions made in between 1950 and 2012 on when human-level AI will come about. [87]

In 2023, Microsoft researchers published a detailed evaluation of GPT-4. They concluded: "Given the breadth and depth of GPT-4's abilities, we believe that it might reasonably be considered as an early (yet still incomplete) version of a synthetic general intelligence (AGI) system." [88] Another study in 2023 reported that GPT-4 exceeds 99% of people on the Torrance tests of imaginative thinking. [89] [90]

Blaise Agüera y Arcas and Peter Norvig composed in 2023 that a substantial level of general intelligence has actually already been attained with frontier models. They composed that unwillingness to this view comes from 4 main reasons: a "healthy skepticism about metrics for AGI", an "ideological dedication to alternative AI theories or methods", a "commitment to human (or biological) exceptionalism", or a "concern about the financial implications of AGI". [91]

2023 also marked the introduction of large multimodal designs (big language designs capable of processing or creating multiple modalities such as text, audio, and images). [92]

In 2024, OpenAI launched o1-preview, the very first of a series of models that "spend more time thinking before they respond". According to Mira Murati, this capability to think before responding represents a brand-new, extra paradigm. It improves model outputs by investing more computing power when producing the answer, whereas the design scaling paradigm improves outputs by increasing the design size, training data and training calculate power. [93] [94]

An OpenAI staff member, Vahid Kazemi, claimed in 2024 that the company had actually attained AGI, stating, "In my opinion, we have actually already achieved AGI and it's much more clear with O1." Kazemi clarified that while the AI is not yet "better than any human at any task", it is "better than the majority of humans at most tasks." He likewise addressed criticisms that big language designs (LLMs) merely follow predefined patterns, comparing their learning procedure to the clinical technique of observing, hypothesizing, and verifying. These declarations have stimulated dispute, as they rely on a broad and unconventional definition of AGI-traditionally understood as AI that matches human intelligence throughout all domains. Critics argue that, while OpenAI's designs show exceptional versatility, they might not totally meet this standard. Notably, Kazemi's comments came quickly after OpenAI removed "AGI" from the regards to its partnership with Microsoft, triggering speculation about the company's strategic intents. [95]

Timescales


Progress in expert system has historically gone through durations of fast progress separated by durations when progress appeared to stop. [82] Ending each hiatus were fundamental advances in hardware, software or both to produce area for additional progress. [82] [98] [99] For example, the hardware readily available in the twentieth century was not enough to implement deep learning, which needs large numbers of GPU-enabled CPUs. [100]

In the introduction to his 2006 book, [101] Goertzel states that price quotes of the time required before a really versatile AGI is built vary from 10 years to over a century. As of 2007 [upgrade], the agreement in the AGI research study community seemed to be that the timeline discussed by Ray Kurzweil in 2005 in The Singularity is Near [102] (i.e. between 2015 and 2045) was plausible. [103] Mainstream AI researchers have actually offered a wide variety of opinions on whether progress will be this rapid. A 2012 meta-analysis of 95 such opinions found a predisposition towards forecasting that the onset of AGI would take place within 16-26 years for modern and historic forecasts alike. That paper has actually been criticized for how it categorized viewpoints as expert or non-expert. [104]

In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton established a neural network called AlexNet, which won the ImageNet competitors with a top-5 test mistake rate of 15.3%, substantially much better than the second-best entry's rate of 26.3% (the conventional approach utilized a weighted sum of scores from various pre-defined classifiers). [105] AlexNet was considered the initial ground-breaker of the present deep learning wave. [105]

In 2017, researchers Feng Liu, Yong Shi, and Ying Liu conducted intelligence tests on publicly offered and freely available weak AI such as Google AI, Apple's Siri, and others. At the optimum, these AIs reached an IQ worth of about 47, which corresponds around to a six-year-old kid in first grade. A grownup comes to about 100 usually. Similar tests were brought out in 2014, with the IQ score reaching an optimum worth of 27. [106] [107]

In 2020, OpenAI established GPT-3, a language design efficient in carrying out numerous diverse tasks without particular training. According to Gary Grossman in a VentureBeat short article, while there is consensus that GPT-3 is not an example of AGI, it is considered by some to be too advanced to be classified as a narrow AI system. [108]

In the exact same year, Jason Rohrer used his GPT-3 account to establish a chatbot, and offered a chatbot-developing platform called "Project December". OpenAI requested for modifications to the chatbot to comply with their safety standards; Rohrer detached Project December from the GPT-3 API. [109]

In 2022, DeepMind established Gato, a "general-purpose" system efficient in carrying out more than 600 various tasks. [110]

In 2023, Microsoft Research released a research study on an early variation of OpenAI's GPT-4, contending that it displayed more basic intelligence than previous AI designs and showed human-level efficiency in jobs spanning numerous domains, such as mathematics, coding, and law. This research stimulated an argument on whether GPT-4 might be considered an early, incomplete version of synthetic basic intelligence, emphasizing the need for additional expedition and assessment of such systems. [111]

In 2023, the AI researcher Geoffrey Hinton specified that: [112]

The idea that this stuff might really get smarter than individuals - a few people thought that, [...] But the majority of people believed it was method off. And I believed it was method off. I believed it was 30 to 50 years or even longer away. Obviously, I no longer think that.


In May 2023, Demis Hassabis likewise stated that "The development in the last few years has actually been quite incredible", and that he sees no reason it would decrease, expecting AGI within a decade and even a few years. [113] In March 2024, Nvidia's CEO, Jensen Huang, mentioned his expectation that within 5 years, AI would can passing any test a minimum of along with people. [114] In June 2024, the AI researcher Leopold Aschenbrenner, a previous OpenAI employee, approximated AGI by 2027 to be "strikingly plausible". [115]

Whole brain emulation


While the advancement of transformer models like in ChatGPT is considered the most appealing path to AGI, [116] [117] entire brain emulation can act as an alternative technique. With entire brain simulation, a brain model is developed by scanning and mapping a biological brain in information, and then copying and replicating it on a computer system or another computational device. The simulation design must be adequately devoted to the original, so that it acts in almost the very same method as the initial brain. [118] Whole brain emulation is a kind of brain simulation that is gone over in computational neuroscience and neuroinformatics, and for medical research study functions. It has actually been gone over in expert system research [103] as a method to strong AI. Neuroimaging innovations that could provide the essential in-depth understanding are improving quickly, and futurist Ray Kurzweil in the book The Singularity Is Near [102] anticipates that a map of adequate quality will end up being offered on a similar timescale to the computing power required to replicate it.


Early approximates


For low-level brain simulation, a very powerful cluster of computers or GPUs would be needed, offered the huge amount of synapses within the human brain. Each of the 1011 (one hundred billion) neurons has on typical 7,000 synaptic connections (synapses) to other neurons. The brain of a three-year-old kid has about 1015 synapses (1 quadrillion). This number decreases with age, supporting by adulthood. Estimates differ for an adult, ranging from 1014 to 5 × 1014 synapses (100 to 500 trillion). [120] A quote of the brain's processing power, based upon a basic switch design for neuron activity, is around 1014 (100 trillion) synaptic updates per second (SUPS). [121]

In 1997, Kurzweil looked at various quotes for the hardware needed to equal the human brain and embraced a figure of 1016 calculations per second (cps). [e] (For contrast, if a "computation" was equivalent to one "floating-point operation" - a measure utilized to rate current supercomputers - then 1016 "computations" would be equivalent to 10 petaFLOPS, achieved in 2011, while 1018 was attained in 2022.) He used this figure to predict the necessary hardware would be available at some point between 2015 and 2025, if the exponential growth in computer power at the time of composing continued.


Current research


The Human Brain Project, an EU-funded effort active from 2013 to 2023, has actually established an especially in-depth and publicly available atlas of the human brain. [124] In 2023, researchers from Duke University performed a high-resolution scan of a mouse brain.


Criticisms of simulation-based techniques


The artificial neuron design presumed by Kurzweil and used in many present synthetic neural network applications is basic compared to biological nerve cells. A brain simulation would likely have to catch the in-depth cellular behaviour of biological neurons, currently comprehended only in broad summary. The overhead presented by complete modeling of the biological, chemical, and physical information of neural behaviour (especially on a molecular scale) would require computational powers several orders of magnitude larger than Kurzweil's quote. In addition, the estimates do not account for glial cells, which are known to play a role in cognitive procedures. [125]

A basic criticism of the simulated brain method stems from embodied cognition theory which asserts that human personification is an essential aspect of human intelligence and is essential to ground meaning. [126] [127] If this theory is correct, any totally functional brain design will require to encompass more than simply the neurons (e.g., a robotic body). Goertzel [103] proposes virtual personification (like in metaverses like Second Life) as a choice, however it is unknown whether this would suffice.


Philosophical viewpoint


"Strong AI" as defined in philosophy


In 1980, theorist John Searle coined the term "strong AI" as part of his Chinese space argument. [128] He proposed a difference in between 2 hypotheses about expert system: [f]

Strong AI hypothesis: An expert system system can have "a mind" and "consciousness".
Weak AI hypothesis: A synthetic intelligence system can (only) imitate it believes and has a mind and awareness.


The very first one he called "strong" due to the fact that it makes a more powerful statement: it assumes something unique has taken place to the machine that goes beyond those abilities that we can check. The behaviour of a "weak AI" machine would be precisely similar to a "strong AI" machine, however the latter would likewise have subjective conscious experience. This use is likewise common in academic AI research study and textbooks. [129]

In contrast to Searle and traditional AI, some futurists such as Ray Kurzweil use the term "strong AI" to imply "human level synthetic basic intelligence". [102] This is not the exact same as Searle's strong AI, unless it is presumed that awareness is essential for human-level AGI. Academic thinkers such as Searle do not believe that holds true, and to most expert system scientists the concern is out-of-scope. [130]

Mainstream AI is most interested in how a program behaves. [131] According to Russell and Norvig, "as long as the program works, they do not care if you call it genuine or a simulation." [130] If the program can behave as if it has a mind, then there is no requirement to understand if it in fact has mind - indeed, there would be no other way to tell. For AI research, Searle's "weak AI hypothesis" is equivalent to the declaration "synthetic basic intelligence is possible". Thus, according to Russell and Norvig, "most AI scientists take the weak AI hypothesis for given, and do not care about the strong AI hypothesis." [130] Thus, for scholastic AI research study, "Strong AI" and "AGI" are two different things.


Consciousness


Consciousness can have various significances, and some elements play substantial functions in science fiction and the principles of artificial intelligence:


Sentience (or "incredible consciousness"): The ability to "feel" understandings or emotions subjectively, instead of the ability to factor about understandings. Some philosophers, such as David Chalmers, use the term "awareness" to refer solely to sensational awareness, which is approximately comparable to sentience. [132] Determining why and how subjective experience arises is called the tough problem of consciousness. [133] Thomas Nagel described in 1974 that it "feels like" something to be mindful. If we are not conscious, then it doesn't seem like anything. Nagel utilizes the example of a bat: we can smartly ask "what does it feel like to be a bat?" However, we are unlikely to ask "what does it seem like to be a toaster?" Nagel concludes that a bat appears to be conscious (i.e., has consciousness) however a toaster does not. [134] In 2022, a Google engineer declared that the business's AI chatbot, LaMDA, had accomplished life, though this claim was widely challenged by other experts. [135]

Self-awareness: To have mindful awareness of oneself as a different individual, specifically to be consciously familiar with one's own ideas. This is opposed to just being the "topic of one's thought"-an os or debugger has the ability to be "knowledgeable about itself" (that is, to represent itself in the exact same way it represents everything else)-but this is not what people normally suggest when they utilize the term "self-awareness". [g]

These qualities have an ethical measurement. AI life would offer rise to concerns of welfare and legal security, similarly to animals. [136] Other elements of awareness associated to cognitive capabilities are also relevant to the principle of AI rights. [137] Figuring out how to integrate advanced AI with existing legal and social frameworks is an emergent issue. [138]

Benefits


AGI might have a variety of applications. If oriented towards such objectives, AGI could assist mitigate different issues on the planet such as hunger, poverty and illness. [139]

AGI could improve performance and efficiency in most tasks. For instance, in public health, AGI could speed up medical research study, significantly versus cancer. [140] It might look after the elderly, [141] and democratize access to quick, high-quality medical diagnostics. It could use fun, cheap and individualized education. [141] The need to work to subsist could become outdated if the wealth produced is effectively redistributed. [141] [142] This likewise raises the concern of the place of people in a drastically automated society.


AGI might also help to make reasonable choices, and to anticipate and avoid catastrophes. It could likewise assist to gain the advantages of possibly disastrous technologies such as nanotechnology or environment engineering, while preventing the associated risks. [143] If an AGI's main goal is to prevent existential disasters such as human extinction (which might be tough if the Vulnerable World Hypothesis ends up being real), [144] it might take procedures to dramatically minimize the dangers [143] while decreasing the effect of these procedures on our quality of life.


Risks


Existential threats


AGI might represent multiple types of existential risk, which are threats that threaten "the premature termination of Earth-originating smart life or the irreversible and extreme damage of its capacity for preferable future advancement". [145] The threat of human termination from AGI has been the topic of numerous disputes, but there is likewise the possibility that the advancement of AGI would result in a permanently flawed future. Notably, it might be utilized to spread and maintain the set of worths of whoever develops it. If humanity still has ethical blind areas similar to slavery in the past, AGI might irreversibly entrench it, preventing moral development. [146] Furthermore, AGI could help with mass monitoring and brainwashing, which might be used to develop a steady repressive around the world totalitarian routine. [147] [148] There is likewise a risk for the makers themselves. If makers that are sentient or otherwise worthy of moral consideration are mass created in the future, engaging in a civilizational course that indefinitely neglects their welfare and interests might be an existential disaster. [149] [150] Considering how much AGI could improve humanity's future and aid reduce other existential threats, Toby Ord calls these existential risks "an argument for continuing with due care", not for "abandoning AI". [147]

Risk of loss of control and human termination


The thesis that AI postures an existential risk for people, which this danger requires more attention, is controversial but has been backed in 2023 by many public figures, AI scientists and CEOs of AI business such as Elon Musk, Bill Gates, Geoffrey Hinton, Yoshua Bengio, Demis Hassabis and Sam Altman. [151] [152]

In 2014, Stephen Hawking slammed widespread indifference:


So, dealing with possible futures of enormous advantages and risks, the professionals are definitely doing everything possible to ensure the very best result, right? Wrong. If a superior alien civilisation sent us a message saying, 'We'll get here in a couple of years,' would we simply reply, 'OK, call us when you get here-we'll leave the lights on?' Probably not-but this is more or less what is occurring with AI. [153]

The potential fate of humankind has sometimes been compared to the fate of gorillas threatened by human activities. The comparison mentions that higher intelligence allowed humankind to dominate gorillas, which are now susceptible in manner ins which they could not have actually anticipated. As an outcome, the gorilla has actually ended up being a threatened types, not out of malice, but just as a security damage from human activities. [154]

The skeptic Yann LeCun thinks about that AGIs will have no desire to dominate mankind which we must take care not to anthropomorphize them and interpret their intents as we would for people. He said that people won't be "wise adequate to design super-intelligent devices, yet extremely foolish to the point of offering it moronic goals without any safeguards". [155] On the other side, the principle of crucial merging suggests that nearly whatever their objectives, intelligent agents will have factors to attempt to endure and acquire more power as intermediary steps to attaining these objectives. Which this does not require having feelings. [156]

Many scholars who are concerned about existential risk supporter for more research into solving the "control problem" to answer the concern: what kinds of safeguards, algorithms, or architectures can programmers implement to increase the probability that their recursively-improving AI would continue to act in a friendly, rather than devastating, way after it reaches superintelligence? [157] [158] Solving the control problem is complicated by the AI arms race (which could result in a race to the bottom of safety preventative measures in order to release items before rivals), [159] and using AI in weapon systems. [160]

The thesis that AI can pose existential danger also has detractors. Skeptics normally say that AGI is unlikely in the short-term, or that concerns about AGI sidetrack from other issues connected to existing AI. [161] Former Google fraud czar Shuman Ghosemajumder considers that for many people outside of the innovation market, existing chatbots and LLMs are currently viewed as though they were AGI, leading to further misconception and fear. [162]

Skeptics sometimes charge that the thesis is crypto-religious, with an illogical belief in the possibility of superintelligence replacing an illogical belief in an omnipotent God. [163] Some scientists think that the interaction projects on AI existential risk by particular AI groups (such as OpenAI, Anthropic, DeepMind, and Conjecture) may be an at effort at regulative capture and to pump up interest in their products. [164] [165]

In 2023, the CEOs of Google DeepMind, OpenAI and Anthropic, together with other market leaders and scientists, issued a joint declaration asserting that "Mitigating the danger of extinction from AI must be a global top priority along with other societal-scale risks such as pandemics and nuclear war." [152]

Mass joblessness


Researchers from OpenAI estimated that "80% of the U.S. labor force might have at least 10% of their work jobs affected by the introduction of LLMs, while around 19% of workers might see a minimum of 50% of their jobs affected". [166] [167] They consider workplace workers to be the most exposed, for example mathematicians, accountants or web designers. [167] AGI might have a better autonomy, ability to make choices, to interface with other computer tools, but likewise to manage robotized bodies.


According to Stephen Hawking, the result of automation on the lifestyle will depend on how the wealth will be rearranged: [142]

Everyone can take pleasure in a life of glamorous leisure if the machine-produced wealth is shared, or many people can wind up badly poor if the machine-owners effectively lobby against wealth redistribution. Up until now, the trend seems to be towards the second option, with technology driving ever-increasing inequality


Elon Musk thinks about that the automation of society will need governments to embrace a universal standard income. [168]

See also


Artificial brain - Software and hardware with cognitive capabilities similar to those of the animal or human brain
AI impact
AI security - Research location on making AI safe and advantageous
AI positioning - AI conformance to the designated objective
A.I. Rising - 2018 film directed by Lazar Bodroža
Expert system
Automated artificial intelligence - Process of automating the application of artificial intelligence
BRAIN Initiative - Collaborative public-private research study initiative announced by the Obama administration
China Brain Project
Future of Humanity Institute - Defunct Oxford interdisciplinary research study centre
General video game playing - Ability of synthetic intelligence to play various video games
Generative expert system - AI system capable of generating content in action to prompts
Human Brain Project - Scientific research study project
Intelligence amplification - Use of information innovation to augment human intelligence (IA).
Machine principles - Moral behaviours of manufactured machines.
Moravec's paradox.
Multi-task learning - Solving multiple maker finding out jobs at the very same time.
Neural scaling law - Statistical law in artificial intelligence.
Outline of expert system - Overview of and topical guide to artificial intelligence.
Transhumanism - Philosophical motion.
Synthetic intelligence - Alternate term for or type of expert system.
Transfer knowing - Artificial intelligence strategy.
Loebner Prize - Annual AI competitors.
Hardware for expert system - Hardware specifically developed and enhanced for artificial intelligence.
Weak expert system - Form of artificial intelligence.


Notes


^ a b See below for the origin of the term "strong AI", and see the scholastic meaning of "strong AI" and weak AI in the short article Chinese space.
^ AI founder John McCarthy writes: "we can not yet characterize in general what sort of computational procedures we want to call smart. " [26] (For a conversation of some meanings of intelligence used by synthetic intelligence scientists, see philosophy of artificial intelligence.).
^ The Lighthill report specifically slammed AI's "grand objectives" and led the dismantling of AI research study in England. [55] In the U.S., DARPA became identified to fund only "mission-oriented direct research, instead of basic undirected research". [56] [57] ^ As AI founder John McCarthy composes "it would be an excellent relief to the rest of the employees in AI if the innovators of new general formalisms would express their hopes in a more guarded form than has actually often been the case." [61] ^ In "Mind Children" [122] 1015 cps is utilized. More just recently, in 1997, [123] Moravec argued for 108 MIPS which would approximately correspond to 1014 cps. Moravec talks in terms of MIPS, not "cps", which is a non-standard term Kurzweil presented.
^ As specified in a standard AI textbook: "The assertion that devices might perhaps act intelligently (or, perhaps much better, act as if they were intelligent) is called the 'weak AI' hypothesis by philosophers, and the assertion that machines that do so are really believing (rather than imitating thinking) is called the 'strong AI' hypothesis." [121] ^ Alan Turing made this point in 1950. [36] References


^ Krishna, Sri (9 February 2023). "What is artificial narrow intelligence (ANI)?". VentureBeat. Retrieved 1 March 2024. ANI is created to carry out a single job.
^ "OpenAI Charter". OpenAI. Retrieved 6 April 2023. Our objective is to guarantee that artificial basic intelligence benefits all of humanity.
^ Heath, Alex (18 January 2024). "Mark Zuckerberg's brand-new goal is developing synthetic general intelligence". The Verge. Retrieved 13 June 2024. Our vision is to build AI that is better than human-level at all of the human senses.
^ Baum, Seth D. (2020 ). A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy (PDF) (Report). Global Catastrophic Risk Institute. Retrieved 28 November 2024. 72 AGI R&D projects were identified as being active in 2020.
^ a b c "AI timelines: What do professionals in expert system anticipate for the future?". Our World in Data. Retrieved 6 April 2023.
^ Metz, Cade (15 May 2023). "Some Researchers Say A.I. Is Already Here, Stirring Debate in Tech Circles". The New York City Times. Retrieved 18 May 2023.
^ "AI pioneer Geoffrey Hinton quits Google and cautions of threat ahead". The New York Times. 1 May 2023. Retrieved 2 May 2023. It is hard to see how you can avoid the bad actors from using it for bad things.
^ Bubeck, Sébastien; Chandrasekaran, Varun; Eldan, Ronen; Gehrke, Johannes; Horvitz, Eric (2023 ). "Sparks of Artificial General Intelligence: Early explores GPT-4". arXiv preprint. arXiv:2303.12712. GPT-4 shows triggers of AGI.
^ Butler, Octavia E. (1993 ). Parable of the Sower. Grand Central Publishing. ISBN 978-0-4466-7550-5. All that you touch you change. All that you alter modifications you.
^ Vinge, Vernor (1992 ). A Fire Upon the Deep. Tor Books. ISBN 978-0-8125-1528-2. The Singularity is coming.
^ Morozov, Evgeny (30 June 2023). "The True Threat of Artificial Intelligence". The New York Times. The genuine risk is not AI itself but the method we release it.
^ "Impressed by synthetic intelligence? Experts state AGI is following, and it has 'existential' threats". ABC News. 23 March 2023. Retrieved 6 April 2023. AGI could position existential risks to humankind.
^ Bostrom, Nick (2014 ). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. ISBN 978-0-1996-7811-2. The first superintelligence will be the last creation that mankind requires to make.
^ Roose, Kevin (30 May 2023). "A.I. Poses 'Risk of Extinction,' Industry Leaders Warn". The New York City Times. Mitigating the danger of termination from AI should be an international top priority.
^ "Statement on AI Risk". Center for AI Safety. Retrieved 1 March 2024. AI experts caution of danger of extinction from AI.
^ Mitchell, Melanie (30 May 2023). "Are AI's Doomsday Scenarios Worth Taking Seriously?". The New York Times. We are far from developing makers that can outthink us in general methods.
^ LeCun, Yann (June 2023). "AGI does not provide an existential risk". Medium. There is no reason to fear AI as an existential danger.
^ Kurzweil 2005, p. 260.
^ a b Kurzweil, Ray (5 August 2005), "Long Live AI", Forbes, archived from the initial on 14 August 2005: Kurzweil describes strong AI as "maker intelligence with the complete variety of human intelligence.".
^ "The Age of Expert System: George John at TEDxLondonBusinessSchool 2013". Archived from the original on 26 February 2014. Retrieved 22 February 2014.
^ Newell & Simon 1976, This is the term they use for "human-level" intelligence in the physical sign system hypothesis.
^ "The Open University on Strong and Weak AI". Archived from the initial on 25 September 2009. Retrieved 8 October 2007.
^ "What is synthetic superintelligence (ASI)?|Definition from TechTarget". Enterprise AI. Retrieved 8 October 2023.
^ "Expert system is transforming our world - it is on everyone to make certain that it goes well". Our World in Data. Retrieved 8 October 2023.
^ Dickson, Ben (16 November 2023). "Here is how far we are to accomplishing AGI, according to DeepMind". VentureBeat.
^ McCarthy, John (2007a). "Basic Questions". Stanford University. Archived from the initial on 26 October 2007. Retrieved 6 December 2007.
^ This list of intelligent characteristics is based upon the subjects covered by major AI books, including: Russell & Norvig 2003, Luger & Stubblefield 2004, Poole, Mackworth & Goebel 1998 and Nilsson 1998.
^ Johnson 1987.
^ de Charms, R. (1968 ). Personal causation. New York: Academic Press.
^ a b Pfeifer, R. and Bongard J. C., How the body shapes the way we think: a brand-new view of intelligence (The MIT Press, 2007). ISBN 0-2621-6239-3.
^ White, R. W. (1959 ). "Motivation reevaluated: The concept of skills". Psychological Review. 66 (5 ): 297-333. doi:10.1037/ h0040934. PMID 13844397. S2CID 37385966.
^ White, R. W. (1959 ). "Motivation reevaluated: The principle of skills". Psychological Review. 66 (5 ): 297-333. doi:10.1037/ h0040934. PMID 13844397. S2CID 37385966.
^ Muehlhauser, Luke (11 August 2013). "What is AGI?". Machine Intelligence Research Institute. Archived from the initial on 25 April 2014. Retrieved 1 May 2014.
^ "What is Artificial General Intelligence (AGI)?|4 Tests For Ensuring Artificial General Intelligence". Talky Blog. 13 July 2019. Archived from the initial on 17 July 2019. Retrieved 17 July 2019.
^ Kirk-Giannini, Cameron Domenico; Goldstein, Simon (16 October 2023). "AI is closer than ever to passing the Turing test for 'intelligence'. What happens when it does?". The Conversation. Retrieved 22 September 2024.
^ a b Turing 1950.
^ Turing, Alan (1952 ). B. Jack Copeland (ed.). Can Automatic Calculating Machines Be Said To Think?. Oxford: Oxford University Press. pp. 487-506. ISBN 978-0-1982-5079-1.
^ "Eugene Goostman is a genuine kid - the Turing Test states so". The Guardian. 9 June 2014. ISSN 0261-3077. Retrieved 3 March 2024.
^ "Scientists contest whether computer system 'Eugene Goostman' passed Turing test". BBC News. 9 June 2014. Retrieved 3 March 2024.
^ Jones, Cameron R.; Bergen, Benjamin K. (9 May 2024). "People can not distinguish GPT-4 from a human in a Turing test". arXiv:2405.08007 [cs.HC]
^ Varanasi, Lakshmi (21 March 2023). "AI designs like ChatGPT and GPT-4 are acing everything from the bar exam to AP Biology. Here's a list of difficult examinations both AI variations have passed". Business Insider. Retrieved 30 May 2023.
^ Naysmith, Caleb (7 February 2023). "6 Jobs Artificial Intelligence Is Already Replacing and How Investors Can Profit From It". Retrieved 30 May 2023.
^ Turk, Victoria (28 January 2015). "The Plan to Replace the Turing Test with a 'Turing Olympics'". Vice. Retrieved 3 March 2024.
^ Gopani, Avi (25 May 2022). "Turing Test is unreliable. The Winograd Schema is obsolete. Coffee is the response". Analytics India Magazine. Retrieved 3 March 2024.
^ Bhaimiya, Sawdah (20 June 2023). "DeepMind's co-founder suggested testing an AI chatbot's ability to turn $100,000 into $1 million to measure human-like intelligence". Business Insider. Retrieved 3 March 2024.
^ Suleyman, Mustafa (14 July 2023). "Mustafa Suleyman: My brand-new Turing test would see if AI can make $1 million". MIT Technology Review. Retrieved 3 March 2024.
^ Shapiro, Stuart C. (1992 ). "Artificial Intelligence" (PDF). In Stuart C. Shapiro (ed.). Encyclopedia of Artificial Intelligence (Second ed.). New York: John Wiley. pp. 54-57. Archived (PDF) from the initial on 1 February 2016. (Section 4 is on "AI-Complete Tasks".).
^ Yampolskiy, Roman V. (2012 ). Xin-She Yang (ed.). "Turing Test as a Defining Feature of AI-Completeness" (PDF). Expert System, Evolutionary Computation and Metaheuristics (AIECM): 3-17. Archived (PDF) from the original on 22 May 2013.
^ "AI Index: State of AI in 13 Charts". Stanford University Human-Centered Artificial Intelligence. 15 April 2024. Retrieved 27 May 2024.
^ Crevier 1993, pp. 48-50.
^ Kaplan, Andreas (2022 ). "Artificial Intelligence, Business and Civilization - Our Fate Made in Machines". Archived from the initial on 6 May 2022. Retrieved 12 March 2022.
^ Simon 1965, p. 96 estimated in Crevier 1993, p. 109.
^ "Scientist on the Set: An Interview with Marvin Minsky". Archived from the original on 16 July 2012. Retrieved 5 April 2008.
^ Marvin Minsky to Darrach (1970 ), quoted in Crevier (1993, p. 109).
^ Lighthill 1973; Howe 1994.
^ a b NRC 1999, "Shift to Applied Research Increases Investment".
^ Crevier 1993, pp. 115-117; Russell & Norvig 2003, pp. 21-22.
^ Crevier 1993, p. 211, Russell & Norvig 2003, p. 24 and see also Feigenbaum & McCorduck 1983.
^ Crevier 1993, pp. 161-162, 197-203, 240; Russell & Norvig 2003, p. 25.
^ Crevier 1993, pp. 209-212.
^ McCarthy, John (2000 ). "Respond to Lighthill". Stanford University. Archived from the original on 30 September 2008. Retrieved 29 September 2007.
^ Markoff, John (14 October 2005). "Behind Artificial Intelligence, a Squadron of Bright Real People". The New York City Times. Archived from the initial on 2 February 2023. Retrieved 18 February 2017. At its low point, some computer scientists and software engineers avoided the term expert system for worry of being deemed wild-eyed dreamers.
^ Russell & Norvig 2003, pp. 25-26
^ "Trends in the Emerging Tech Hype Cycle". Gartner Reports. Archived from the initial on 22 May 2019. Retrieved 7 May 2019.
^ a b Moravec 1988, p. 20
^ Harnad, S. (1990 ). "The Symbol Grounding Problem". Physica D. 42 (1-3): 335-346. arXiv: cs/9906002. Bibcode:1990 PhyD ... 42..335 H. doi:10.1016/ 0167-2789( 90 )90087-6. S2CID 3204300.
^ Gubrud 1997
^ Hutter, Marcus (2005 ). Universal Expert System: Sequential Decisions Based on Algorithmic Probability. Texts in Theoretical Computer Science an EATCS Series. Springer. doi:10.1007/ b138233. ISBN 978-3-5402-6877-2. S2CID 33352850. Archived from the original on 19 July 2022. Retrieved 19 July 2022.
^ Legg, Shane (2008 ). Machine Super Intelligence (PDF) (Thesis). University of Lugano. Archived (PDF) from the initial on 15 June 2022. Retrieved 19 July 2022.
^ Goertzel, Ben (2014 ). Artificial General Intelligence. Lecture Notes in Computer Technology. Vol. 8598. Journal of Artificial General Intelligence. doi:10.1007/ 978-3-319-09274-4. ISBN 978-3-3190-9273-7. S2CID 8387410.
^ "Who created the term "AGI"?". goertzel.org. Archived from the initial on 28 December 2018. Retrieved 28 December 2018., by means of Life 3.0: 'The term "AGI" was popularized by ... Shane Legg, Mark Gubrud and Ben Goertzel'
^ Wang & Goertzel 2007
^ "First International Summer School in Artificial General Intelligence, Main summertime school: June 22 - July 3, 2009, OpenCog Lab: July 6-9, 2009". Archived from the original on 28 September 2020. Retrieved 11 May 2020.
^ "Избираеми дисциплини 2009/2010 - пролетен триместър" [Elective courses 2009/2010 - spring trimester] Факултет по математика и информатика [Faculty of Mathematics and Informatics] (in Bulgarian). Archived from the initial on 26 July 2020. Retrieved 11 May 2020.
^ "Избираеми дисциплини 2010/2011 - зимен триместър" [Elective courses 2010/2011 - winter trimester] Факултет по математика и информатика [Faculty of Mathematics and Informatics] (in Bulgarian). Archived from the initial on 26 July 2020. Retrieved 11 May 2020.
^ Shevlin, Henry; Vold, Karina; Crosby, Matthew; Halina, Marta (4 October 2019). "The limits of machine intelligence: Despite progress in machine intelligence, synthetic basic intelligence is still a major challenge". EMBO Reports. 20 (10 ): e49177. doi:10.15252/ embr.201949177. ISSN 1469-221X. PMC 6776890. PMID 31531926.
^ Bubeck, Sébastien; Chandrasekaran, Varun; Eldan, Ronen; Gehrke, Johannes; Horvitz, Eric; Kamar, Ece; Lee, Peter; Lee, Yin Tat; Li, Yuanzhi; Lundberg, Scott; Nori, Harsha; Palangi, Hamid; Ribeiro, Marco Tulio; Zhang, Yi (27 March 2023). "Sparks of Artificial General Intelligence: Early explores GPT-4". arXiv:2303.12712 [cs.CL]
^ "Microsoft Researchers Claim GPT-4 Is Showing "Sparks" of AGI". Futurism. 23 March 2023. Retrieved 13 December 2023.
^ Allen, Paul; Greaves, Mark (12 October 2011). "The Singularity Isn't Near". MIT Technology Review. Retrieved 17 September 2014.
^ Winfield, Alan. "Expert system will not develop into a Frankenstein's beast". The Guardian. Archived from the initial on 17 September 2014. Retrieved 17 September 2014.
^ Deane, George (2022 ). "Machines That Feel and Think: The Role of Affective Feelings and Mental Action in (Artificial) General Intelligence". Artificial Life. 28 (3 ): 289-309. doi:10.1162/ artl_a_00368. ISSN 1064-5462. PMID 35881678. S2CID 251069071.
^ a b c Clocksin 2003.
^ Fjelland, Ragnar (17 June 2020). "Why general synthetic intelligence will not be understood". Humanities and Social Sciences Communications. 7 (1 ): 1-9. doi:10.1057/ s41599-020-0494-4. hdl:11250/ 2726984. ISSN 2662-9992. S2CID 219710554.
^ McCarthy 2007b.
^ Khatchadourian, Raffi (23 November 2015). "The Doomsday Invention: Will synthetic intelligence bring us utopia or damage?". The New Yorker. Archived from the original on 28 January 2016. Retrieved 7 February 2016.
^ Müller, V. C., & Bostrom, N. (2016 ). Future development in artificial intelligence: A survey of skilled viewpoint. In Fundamental issues of expert system (pp. 555-572). Springer, Cham.
^ Armstrong, Stuart, and Kaj Sotala. 2012. "How We're Predicting AI-or Failing To." In Beyond AI: Artificial Dreams, edited by Jan Romportl, Pavel Ircing, Eva Žáčková, Michal Polák and Radek Schuster, 52-75. Plzeň: University of West Bohemia
^ "Microsoft Now Claims GPT-4 Shows 'Sparks' of General Intelligence". 24 March 2023.
^ Shimek, Cary (6 July 2023). "AI Outperforms Humans in Creativity Test". Neuroscience News. Retrieved 20 October 2023.
^ Guzik, Erik E.; Byrge, Christian; Gilde, Christian (1 December 2023). "The creativity of devices: AI takes the Torrance Test". Journal of Creativity. 33 (3 ): 100065. doi:10.1016/ j.yjoc.2023.100065. ISSN 2713-3745. S2CID 261087185.
^ Arcas, Blaise Agüera y (10 October 2023). "Artificial General Intelligence Is Already Here". Noema.
^ Zia, Tehseen (8 January 2024). "Unveiling of Large Multimodal Models: Shaping the Landscape of Language Models in 2024". Unite.ai. Retrieved 26 May 2024.
^ "Introducing OpenAI o1-preview". OpenAI. 12 September 2024.
^ Knight, Will. "OpenAI Announces a New AI Model, Code-Named Strawberry, That Solves Difficult Problems Step by Step". Wired. ISSN 1059-1028. Retrieved 17 September 2024.
^ "OpenAI Employee Claims AGI Has Been Achieved". Orbital Today. 13 December 2024. Retrieved 27 December 2024.
^ "AI Index: State of AI in 13 Charts". hai.stanford.edu. 15 April 2024. Retrieved 7 June 2024.
^ "Next-Gen AI: OpenAI and Meta's Leap Towards Reasoning Machines". Unite.ai. 19 April 2024. Retrieved 7 June 2024.
^ James, Alex P. (2022 ). "The Why, What, and How of Artificial General Intelligence Chip Development". IEEE Transactions on Cognitive and Developmental Systems. 14 (2 ): 333-347. arXiv:2012.06338. doi:10.1109/ TCDS.2021.3069871. ISSN 2379-8920. S2CID 228376556. Archived from the original on 28 Au

commentaires