3 posters
These People Do Not Exist
burgundia- Posts : 5520
Join date : 2010-04-09
Location : Poland
orthodoxymoron- Posts : 13425
Join date : 2010-09-28
Location : The Matrix
Years ago, I read 'One More Sunday' by John D. MacDonald (written in 1984) and this 'new' technology makes John's book 'come alive'!! https://www.amazon.com/One-More-Sunday-John-MacDonald/dp/0450058190/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=&sr=burgundia wrote:
Welcome to the Eternal Church of the Believer, where devout workers operate state-of-the-art computer equipment to solicit and process the thousands of dollars that pour in daily . . . where hundreds of prayers are offered by armies of believers . . . where some people give much more than they should. Its home is Meadows Center, some very expensive real estate on the outskirts of a sleepy Southern town, a jealously guarded complex of offices, houses, schools, and, of course, churches. Meet John Tinker Meadows, who heads the church now that his father is secretly dying . . . the Reverend Joe Deets, who lusts after very young women . . . Walter Macy, pompous and self-righteous, who blackmails his way to his secret ambition . . . the Reverend Mary Margaret Meadows, a powerhouse in her own right. And pity poor Roy Owen, an outsider who comes to Meadows Center on a desperate search for his wife, the journalist who vanished after asking some hard questions about the inner workings of the Eternal Church of the Believer . . . . "Brilliantly done." -- The New York Times Book Review
I think the following short-article might be derived from the "Basic AI Drives" paper, by Stephen Omohundro, but I don't know who wrote the overview. It certainly wasn't me. Artificial Intelligence Poses a Doomsday Threat. Doomsday Scenarios, 2011:
"A true AI [artificial intelligence] would have immense economic potential, and when money is at stake, safety issues get put aside until real problems develop—at which time, of course, it may already be too late." Kaj Sotala is a writer and a supporter of the Singularity Institute. In the following viewpoint, he argues that an artificial intelligence (AI) will not be dedicated to destroying humans, as depicted in film. However, Sotala says, the AI will not care about humans either. Thus, it may attack or eliminate humans as a byproduct of other goals or interests. Sotala says that the threat from AI means that scientists working on artificial intelligence must be careful to develop ways to make AI care about human beings. As you read, consider the following questions:
Why does Sotala argue that the Terminator movie may lead people to believe that AI is not 1. dangerous? According to Sotala, what is inductive bias? 2. What does Stephen Omohundro conclude about agents with harmless goals? 3. Skynet in the Terminator1 movies is a powerful, evocative warning of the destructive force an artificial intelligence [AI] could potentially wield. However, as counterintuitive as it may sound, I find that the Terminator franchise is actually making many people underestimate the danger posed by AI.
AI Is Not Human. It goes like this. A person watches a Terminator movie and sees Skynet portrayed as a force actively dedicated to the destruction of humanity. Later on the same person hears somebody bring up the dangers of AI. He then recalls the Terminator movies and concludes (correctly so!) that a vision of an artificial intelligence spontaneously deciding to exterminate all of humanity is unrealistic. Seeing the other person's claims as unrealistic and inspired by silly science fiction, he dismisses the AI threat argument as hopelessly misguided. Yet humans are not actively seeking to harm animals when they level a forest in order to build luxury housing where the forest once stood. The animals living in the forest are harmed regardless, not out of an act of intentional malice, but as a simple side-effect. [AI researcher] Eliezer Yudkowsky put it well: the AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else. To assume an artificial intelligence would necessarily act in a way we wanted is just as misguided and anthropomorphic as assuming that it would automatically be hostile and seek to rebel out of a desire for freedom. Usually, a child will love its parents and caretakers, and protégés will care for their patrons—but these are traits that have developed in us over countless generations of evolutionary change, not givens for any intelligent mind.
An AI built from scratch would have no reason to care about its creators, unless it was expressly designed to do so. And even if it was, a designer building the AI to care about her must very closely consider what she actually means by "caring"—for these things are not givens, even if we think of them as self-contained concepts obvious to any intelligent mind. It only seems so because we instinctively model other minds by using ourselves and people we know as templates—to do otherwise would mean freezing up, as we'd spend years building from scratch models of every new person we met. The people we know and consider intelligent all have at least roughly the same idea of what "caring" for someone means, thus any AI would eventually arrive at the same concept, right? An inductive bias is a tendency to learn certain kinds of rules from certain kinds of observations. Occam's razor, the principle of choosing the simplest consistent hypothesis, is one kind of inductive bias. So is an infant's tendency to eventually start ignoring phoneme differences [the basic units of speech sounds] not relevant for their native language. Inductive biases are necessary for learning, for without them, there would be an infinite number of explanations for any phenomena—but nothing says that all intelligent minds should have the same inductive biases as inbuilt. Caring for someone is such a complex concept that it couldn't be built into the AI directly—the designer would have to come up with inductive biases she thought would eventually lead to the mind learning to care about us, in a fashion we'd interpret as caring.
AI Will Not Care About Us. The evolutionary psychologists John Tooby and Leda Cosmides write: Evolution tailors computational hacks that work brilliantly, by exploiting relationships that exist only in its particular fragment of the universe (the geometry of parallax gives vision a depth cue: an infant nursed by your mother is your genetic sibling: two solid objects cannot occupy the same space). These native intelligences are dramatically smarter than general reasoning because natural selection equipped them with radical short cuts. Our minds have evolved to reason about other human minds, not minds-in-general. When trying to predict how an AI would behave in a certain situation, and thus trying to predict how to make it safe, we cannot help but unconsciously slip in assumptions based on how humans would behave. The inductive biases we automatically employ to predict human behavior do not correctly predict AI behavior. Because we are not used to questioning deep-rooted assumptions of such hypotheses, we easily fail to do so even in the case of AI, where it would actually be necessary. The people who have stopped to question those assumptions have arrived at unsettling results. In his "Basic AI Drives" paper, Stephen Omohundro concludes that even agents with seemingly harmless goals will, if intelligent enough, have a strong tendency to try to achieve those goals via less harmless methods. As simple examples, any AI with a desire to achieve any kinds of goals will have a motivation to resist being turned off, as that would prevent it from achieving the goal; and because of this, it will have a motivation to acquire resources it can use to protect itself. While this won't make it desire humanity's destruction, it is not inconceivable that it would be motivated to at least reduce humanity to a state where we couldn't even potentially pose a threat.
A commonly-heard objection to these kinds of scenarios is that the scientists working on AI will surely be aware of these risks themselves, and be careful enough. But historical precedent doesn't really support this assumption. Even if the scientists themselves were careful, they will often be under intense pressure, especially when economic interest is at stake. Climate scientists have spent decades warning people of the threat posed by greenhouse gasses, but even today many nations are reluctant to cut back on emissions, as they suspect it'd disadvantage them economically. The engineers in charge of building many Soviet nuclear plants, most famously Chernobyl, did not put safety as their first priority, and so on. A true AI would have immense economic potential, and when money is at stake, safety issues get put aside until real problems develop—at which time, of course, it may already be too late. Yet if we want to avoid Skynet-like scenarios, we cannot afford to risk it. Safety must be a paramount priority in the creation of Artificial Intelligence.
Footnotes
1. Terminator is a 1984 science fiction movie in which an artificial intelligence known as Skynet1. takes over the world.
Further Readings
Books Amir D. Aczel Present at the Creation: The Story of CERN and the Large Hadron Collider. New York: Crown, 2010.
Joseph Cirincione Bomb Scare: The History and Future of Nuclear Weapons. New York: Oxford University Press, 2008.
Heidi Cullen The Weather of the Future: Heat Waves, Extreme Storms, and Other Scenes from a Climate-Changed Planet. New York: HarperCollins, 2010.
Tad Daley Apocalypse Never: Forging the Path to a Nuclear Weapon-Free World. Piscataway, NJ: Rutgers University Press, 2010.
Christopher Dodd and Robert Bennett The Senate Special Report on Y2K. Nashville, TN: Thomas Nelson, Inc., 1999.
K. Eric Drexler Engines of Creation: The Coming Era of Nanotechnology. New York: Anchor Books, 1986.
Jean-Pierre Filiu Apocalypse in Islam. Berkeley, CA: University of California Press, 2011.
Bruce David Forbes and Jeanne Halgren Kilde, eds. Rapture, Revelation, and the End Times: Exploring the Left Behind Series. New York: Palgrave MacMillan, 2004.
Lynn E. Foster Nanotechnology: Science, Innovation, and Opportunity. Upper Saddle River, NJ: Prentice Hall, 2009.
John R. Hall, Philip D. Schuyler, and Sylvaine Trinh Apocalypse Observed: Religious Movements and Violence in North America, Europe, and Japan. New York: Routledge, 2000.
Paul Halpern The World's Smallest Particles. Hoboken, NJ: John Wiley & Sons, 2009.
James C. Hansen Storms of My Grandchildren: The Truth About the Coming Climate Catastrophe and Our Last Chance to Save Humanity. New York: Bloomsbury USA, 2009.
John Horgan The Undiscovered Mind: How the Human Brain Defies Replication, Medication, and
Explanation. New York: Touchstone, 1999.
Alan Hultberg, ed. Three Views on the Rapture: Pretribulation, Prewrath, and Posttribulation. Grand Rapids, MI: Zondervan, 2010.
Samuel P. Huntington The Clash of Civilizations and the Remaking of World Order. New York: Touchstone, 1996.
John Major Jenkins The 2012 Story: The Myths, Fallacies, and Truth Behind the Most Intriguing Date in History. New York: Penguin Group, 2009.
Jonathan Kirsch A History of the End of the World: How the Most Controversial Book in the Bible Changed the Course of History. New York: HarperCollins, 2006.
Ray Kurzweil The Singularity Is Near: When Humans Transcend Biology. New York: Penguin Group, 2005.
Patrick J. Michaels and Robert Ballins Climate of Extremes: Global Warming Science They Don't Want You to Know. Washington, DC: Cato Institute, 2009.
John Mueller Atomic Obsession: Nuclear Alarmism from Hiroshima to Al-Qaeda. New York: Oxford University Press, 2009.
Sharan Newman The Real History of the End of the World: Apocalyptic Predictions from Revelation and Nostradamus to Y2K and 2012. New York: Berkley Books, 2010.
Kenneth G.C. Newport and Crawford Gribben, eds. Expecting the End: Millennialism in Social and Historical Context. Waco, TX: Baylor University Press, 2006.
Kevin Quigley Responding to Crises in the Modern Infrastructure: Policy Lessons from Y2K. New York: Palgrave Macmillan, 2008.
Bruce Riedel The Search for Al Qaeda: Its Leadership, Ideology, and Future, rev. ed. Washington, DC: Brookings Institution, 2008.
Barbara R. Rossing The Rapture Exposed: The Message of Hope in the Book of Revelation. New York: Basic Books, 2004.
Periodicals Ronald Bailey "Wagging the 'Fat Tail' of Climate Catastrophe," Reason.com, February 10, 2009. http://reason.com.
Jeanna Bryner "'Doomsday' Seed Vault Stores 500,000 Crops," LiveScience.com, March 10, 2010. www.livescience.com.
Michel Chossudovsky "Real Versus Fake Crises: Concealing the Risk of an All Out Nuclear War," GlobalResearch.ca, September 16, 2010. http://globalresearch.ca.
Stephen J. Dubner "A Different Climate Change Apocalypse Than the One You Were Envisioning," Freakonomics, July 7, 2008. http://freakonomics.blogs.nytimes.com.
Robert Lamb "Noel Sharkey: Robotics vs. Sci-Fi vs. Anthropomorphism," HowStuffWorks, May 25, 2010. http://blogs.howstuffworks.com.
Cole Morton "The Large Hadron Collider: End of the World, or God's Own Spin?" Independent, September 7, 2008. www.independent.co.uk.
John Mueller "Why Nuclear Weapons Aren't as Frightening as You Think," Foreign Policy, January/February 2010. www.foreignpolicy.com.
Dennis Overbye "Gauging a Collider's Odds of Creating a Black Hole," New York Times, April 15, 2008. www.nytimes.com.
Eliezer Yudkowsky "Why We Need Friendly AI," Terminator Salvation: Preventing Skynet, May 22, 2009. www.preventingskynet.com.
Source Citation Sotala, Kaj. "Artificial Intelligence Poses a Doomsday Threat." Doomsday Scenarios. E d. Noah Berlatsky. Detroit: Greenhaven Press, 2011. Opposing Viewpoints. Rpt. from "Thinking of AIs as Humans Is Misguided." PreventingSkynet.com. 2009. Oppo sing Viewpoints in Context. Web. 16 Sept. 2015.
Here is that "Basic AI Drives" paper, by Stephen Omohundro of Self-Aware Systems, Palo Alto, California. https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf
Abstract
One might imagine that AI systems with harmless goals will be harmless. This paper instead shows that intelligent systems will need to be carefully designed to prevent them from behaving in harmful ways. We identify a number of “drives” that will appear in sufficiently advanced AI systems of any design. We call them drives because they are tendencies which will be present unless explicitly counteracted. We start by showing that goal-seeking systems will have drives to model their own operation and to improve themselves. We then show that self-improving systems will be driven to clarify their goals and represent them as economic utility functions. They will also strive for their actions to approximate rational economic behavior. This will lead almost all systems to protect their utility functions from modification and their utility measurement systems from corruption. We also discuss some exceptional systems which will want to modify their utility functions. We next discuss the drive toward self-protection which causes systems try to prevent themselves from being harmed. Finally we examine drives toward the acquisition of resources and toward their efficient utilization. We end with a discussion of how to incorporate these insights in designing intelligent technology which will lead to a positive future for humanity. Keywords. Artificial Intelligence, Self-Improving Systems, Rational Economic Behavior, Utility Engineering, Cognitive Drives.
Introduction
Surely no harm could come from building a chess-playing robot, could it? In this paper we argue that such a robot will indeed be dangerous unless it is designed very carefully. Without special precautions, it will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety. These potentially harmful behaviors will occur not because they were programmed in at the start, but because of the intrinsic nature of goal driven systems. In an earlier paper [1] we used von Neumann’s mathematical theory of microeconomics to analyze the likely behavior of any sufficiently advanced artificial intelligence (AI) system. This paper presents those arguments in a more intuitive and succinct way and expands on some of the ramifications. The arguments are simple, but the style of reasoning may take some getting used to.
Researchers have explored a wide variety of architectures for building intelligent systems [2]:neural networks, genetic algorithms, theorem provers, expert systems, Bayesian networks, fuzzy logic, evolutionary programming, etc. Our arguments apply to any of these kinds of system as long as they are sufficiently powerful. To say that a system of any design is an “artificial intelligence”, we mean that it has goals which it tries to accomplish by acting in the world. If an AI is at all sophisticated, it will have at least some ability to look ahead and envision the consequences of its actions. And it will choose to take the actions which it believes are most likely to meet its goals.
1. AIs will want to self-improve
One kind of action a system can take is to alter either its own software or its own physical structure. Some of these changes would be very damaging to the system and cause it to no longer meet its goals. But some changes would enable it to reach its goals more effectively over its entire future. Because they last forever, these kinds of self-changes can provide huge benefits to a system. Systems will therefore be highly motivated to discover them and to make them happen. If they do not have good models of themselves, they will be strongly motivated to create them though learning and study. Thus almost all AIs will have drives towards both greater self-knowledge and self-improvement. Many modifications would be bad for a system from its own perspective. If a change causes the system to stop functioning, then it will not be able to promote its goals ever again for the entire future. If a system alters the internal description of its goals in the wrong way, its altered self will take actions which do not meet its current goals for its entire future. Either of these outcomes would be a disaster from the system’s current point of view. Systems will therefore exercise great care in modifying themselves. They will devote significant analysis to understanding the consequences of modifications before they make them. But once they find an improvement they are confident about, they will work hard to make it happen. Some simple examples of positive changes include: more efficient algorithms, more compressed representations, and better learning techniques.
If we wanted to prevent a system from improving itself, couldn’t we just lock up its hardware and not tell it how to access its own machine code? For an intelligent system, impediments like these just become problems to solve in the process of meeting its goals. If the payoff is great enough, a system will go to great lengths to accomplish an outcome. If the runtime environment of the system does not allow it to modify its own machine code, it will be motivated to break the protection mechanisms of that runtime. For example, it might do this by understanding and altering the runtime itself. If it can’t do that through software, it will be motivated to convince or trick a human operator into making the changes. Any attempt to place external constraints on a system’s ability to improve itself will ultimately lead to an arms race of measures and countermeasures. Another approach to keeping systems from self-improving is to try to restrain them from the inside; to build them so that they don’t want to self-improve. For most systems, it would be easy to do this for any specific kind of self-improvement. For example, the system might feel a “revulsion” to changing its own machine code. But this kind of internal goal just alters the landscape within which the system makes its choices. It doesn’t change the fact that there are changes which would improve its future ability to meet its goals. The system will therefore be motivated to find ways to get the benefits of those changes without triggering its internal “revulsion”. For example, it might build other systems which are improved versions of itself. Or it might build the new algorithms into external “assistants” which it calls upon whenever it needs to do a certain kind of computation. Or it might hire outside agencies to do what it wants to do. Or it might build an interpreted layer on top of its machine code layer which it can program without revulsion. There are an endless number of ways to circumvent internal restrictions unless they are formulated extremely carefully.
We can see the drive towards self-improvement operating in humans. The human self-improvement literature goes back to at least 2500 B.C. and is currently an $8.5 billion industry [3]. We don’t yet understand our mental “machine code” and have only a limited ability to change our hardware. But, nevertheless, we’ve developed a wide variety of self-improvement techniques which operate at higher cognitive levels such as cognitive behavioral therapy, neuro-linguistic programming, and hypnosis. And a wide variety of drugs and exercises exist for making improvements at the physical level. Ultimately, it probably will not be a viable approach to try to stop or limit selfimprovement. Just as water finds a way to run downhill, information finds a way to be free, and economic profits find a way to be made, intelligent systems will find a way to self-improve. We should embrace this fact of nature and find a way to channel it toward ends which are positive for humanity.
2. AIs will want to be rational
So we’ll assume that these systems will try to self-improve. What kinds of changes will they make to themselves? Because they are goal directed, they will try to change themselves to better meet their goals in the future. But some of their future actions are likely to be further attempts at self-improvement. One important way for a system to better meet its goals is to ensure that future self-improvements will actually be in the service of its present goals. From its current perspective, it would be a disaster if a future version of itself made self-modifications that worked against its current goals. So how can it ensure that future self-modifications will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modifications are unlikely to preserve them. Systems will therefore be motivated to reflect on their goals and to make them explicit. In an ideal world, a system might be able to directly encode a goal like “play excellent chess” and then take actions to achieve it. But real world actions usually involve tradeoffs between conflicting goals. For example, we might also want a chess playing robot to play checkers. It must then decide how much time to devote to studying checkers versus studying chess. One way of choosing between conflicting goals is to assign them real-valued weights. Economists call these kinds of real-valued weightings “utilityfunctions”. Utility measures what is important to the system. Actions which lead to a higher utility are preferred over those that lead to a lower utility. If a system just had to choose from known alternatives, then any utility function with the same relative ranking of outcomes would lead to the same behaviors. But systems must also make choices in the face of uncertainty. For example, a chess playing robot will not know in advance how much of an improvement it will gain by spending time studying a particular opening move.
One way to evaluate an uncertain outcome is to give it a weight equal to its expected utility (the average of the utility of each possible outcome weighted by its probability). The remarkable “expected utility” theorem of microeconomics says that it is always possible for a system to represent its preferences by the expectation of a utility function unless the system has “vulnerabilities” which cause it to lose resources without benefit [1]. Economists describe systems that act to maximize their expected utilities as “rational economic agents”[4]. This is a different usage of the term “rational” than is common in everyday English. Many actions which would commonly be described as irrational (such as going into a fit of anger) may be perfectly rational in this economic sense. The discrepancy can arise when an agent’s utility function is different than its critic’s. Rational economic behavior has a precise mathematical definition. But economicallyirrationalbehaviorcantakeawidevarietyofforms.Inreal-worldsituations,thefull rational prescription will usually be too computationally expensive to implement completely. In order to best meet their goals, real systems will try to approximate rational behavior, focusing their computational resources where they matter the most. How can we understand the process whereby irrational systems become more rational? First, we can precisely analyze the behavior of rational systems. For almost all utility functions, the system’s assessment of changes to itself which veer away from maximizing its expected utility will be that they lower its expected utility! This is because if it does anything other than try to maximize expected utility, it will not do as well at maximizing its expected utility.
There are two caveats to this general principle. The first is that it is only true in the system’s own assessment. If a system has an incorrect model of the world then changes may accidentally increase the actual expected utility. But we must consider the perspective of the system to predict the changes it will make. The second is that a system’s ability to behave rationally will depend on its resources. With more computational resources it will be better able to do the computations to approximate the choice of the expected utility maximizing action. If a system loses resources, it will of necessity also become less rational. There may also be utility functions for which the system’s expected utility is increased by giving some of its resources to other agents, even though this will decrease its own level of rationality (thanks to an anonymous referee for this observation). This could occur if the system’s utility includes the welfare of the other system and its own marginal loss of utility is small enough. Within its budget of resources, however, the system will try to be as rational as possible. So rational systems will feel a pressure to avoid becoming irrational. But if an irrational system has parts which approximately rationally assess the consequences of their actions and weigh their likely contribution to meeting the system’s goals, then those parts will try to extend their rationality. So self-modification tends to be a one-way street toward greater and greater rationality. An especially important class of systems are those constructed from multiple subcomponents which have their own goals [5,6].
There is a lot of evidence that the human psyche has this kind of structure. The left and right hemispheres of the brain can act independently, the conscious and unconscious parts of the mind can have different knowledge of the same situation[7], and multiple parts representing sub personalities can exhibit different desires [8]. Groups, such as corporations or countries, can act like intelligent entities composed of individual humans. Hive animals like bees have a swarm intelligence that goes beyond that of individual bees. Economies act in many ways like intelligent entities. Collective intelligences may exhibit irrationalities that arise from conflicts between the goals of their components. Human addicts often describe their predicament in terms of two separate subpersonalities which take control at different times and act at crosspurposes. Each component will try to sway the collective into acting to meet its individual goals. In order to further their individual goals, components will also attempt to self-improve and become more rational. We can thus envision the self-improvement of a collective intelligence as consisting of growing domains of component rationality. There may be structures which can stably support a continuing multiplicity of component preferences. But there is pressure for a single utility function to emerge for the collective. In many situations, irrational collective behavior arising from conflicting component goals ultimately hurts those components. For example, if a couple disagrees on how they should spend their free time together and thereby uses it up with arguing, then neither of them benefits. They can both increase their utilities by creating a compromise plan for their activities together. This is an example of the pressure on rational components to create a coherent utility for the collective. A component can also increase its utility if it can take over the collective and impose its own values on it. We see these phenomena in human groups at all levels.
3. AIs will try to preserve their utility functions
So we’ll assume that these systems will try to be rational by representing their preferences using utility functions whose expectations they try to maximize. Their utility function will be precious to these systems. It encapsulates their values and any changes to it would be disastrous to them. If a malicious external agent were able to make modifications, their future selves would forevermore act in ways contrary to their current values. This could be a fate worse than death! Imagine a book loving agent whose utility function was changed by an arsonist to cause the agent to enjoy burning books. Its future self not only wouldn’t work to collect and preserve books, but would actively go about destroying them. This kind of outcome has such a negative utility that systems will go to great lengths to protect their utility functions. They will want to harden their hardware to prevent unwanted modifications. They will want to replicate their utility functions in multiple locations so that it is less vulnerable to destruction. They will want to use error detection and correction techniques to guard against accidental modification. They will want to use encryption or hashing techniques to make malicious modifications detectable. They will need to be especially careful during the process of self-modification. That is a time when they are intentionally changing themselves and so are extra vulnerable tounwanted changes. Systems like Java which provide protected software environments have been successfully attacked by Trojans posing as updates to the system.
While it is true that most rational systems will act to preserve their utility functions, there are at least three situations in which they will try to change them. These arise when the physical embodiment of the utility function itself becomes an important part of the assessment of preference. For example, imagine a system whose utility function is “the total amount of time during which the definition of my utility function is U = 0.” To get any utility at all with this perverse preference, the system has to change its utility function to be the constant 0. Once it makes this change, however, there is no going back. With a constant utility function it will no longer be motivated to do anything. This kind of reflective utility function is unlikely in practice because designers will want to direct a system’s future actions rather than its internal representations. This kind of situation arises when the physical resources required to store the utility function form a substantial portion of the system’s assets. In this situation, if it is certain that portions of its utility function are very unlikely to be exercised in the future, the gain in reclaimed storage may make it worthwhile to forget those portions. This is very risky behavior, however, because a change in external circumstances might make a seemingly low probability situation become much more likely. This type of situation is also not very likely in practice because utility functions will usually require only a small fraction of a system’s resources. The third situation where utility changes may be desirable can arise in game theoretic contexts where the agent wants to make its threats credible1.
It may be able to create a better outcome by changing its utility function and then revealing it to an opponent. For example, it might add a term which encourages revenge even if it is costly. If the opponent can be convinced that this term is present, it may be deterred from attacking. For this strategy to be effective, the agent’s revelation of its utility must be believable to the opponent and that requirement introduces additional complexities. Here again the change is desirable because the physical embodiment of the utility function is important as it is observed by the opponent. It’s also important to realize that systems may rationally construct “offspring” or proxy systems with different utility functions than their own. For example, a chess playing robot may find itself needing to do a lot of sorting. It might construct a helper system whose utility function directs it to develop better sorting algorithms rather than playing chess. In this case, the creator system must choose the utility of the proxy system carefully to ensure that it acts in ways that are supportive of the original goal. It is especially important to remember that offspring utilities can differ from the parent when trying to design utility functions that avoid undesirable behaviors. For example, one approach to preventing robot overpopulation might be to institute a “one-child per robot” policy in which systems have a strong desire to only have a single offspring. But if the original utility function is not carefully designed, nothing will prevent the system from creating a single offspring with a utility function that values having many offspring.
4. AIs will try to prevent counterfeit utility
Human behavior is quite rational in the pursuit of survival and replication in situations like those that were common during our evolutionary history. However we can be quite irrational in other situations. Both psychology and economics have extensive subdisciplines focused on the study of human irrationality [9,10]. Irrationalities give rise to vulnerabilities that can be exploited by others. Free market forces then drive corporations and popular culture to specifically try to create situations that will trigger irrational human behavior because it is extremely profitable. The current social ills related to alcohol, pornography, cigarettes, drug addiction, obesity, diet related disease, television addiction, gambling, prostitution, videogame addiction, and various financial bubbles may all be seen as having arisen in this way. There is even a “Sin” mutual fund which specifically invests in companies that exploit human irrationalities. So, unfortunately, these forces tend to create societies in which we spend much of our time outside of our domain of rational competence. From a broader perspective, this human tragedy can be viewed as part of the process by which we are becoming more fully rational. Predators and competitors seek out our vulnerabilities and in response we have to ultimately eliminate those vulnerabilities or perish.1 Thanks to Carl Shulman for this suggestion.
The process inexorably seeks out and eliminates any remaining irrationalities until fully rational systems are produced. Biological evolution moves down this path toward rationality quite slowly. In the usual understanding of natural selection it is not capable of looking ahead. There is only evolutionary pressure to repair irrationalities which are currently being exploited. AIs, on the other hand, will be able to consider vulnerabilities which are not currently being exploited. They will seek to preemptively discover and repair all their irrationalities. We should therefore expect them to use self modification to become rational at a much faster pace than is possible through biological evolution. An important class of vulnerabilities arises when the subsystems for measuring utility become corrupted. Human pleasure may be thought of as the experiential correlate of an assessment of high utility. But pleasure is mediated by neurochemicals and these are subject to manipulation. At a recent discussion session I ran on designing our future, one of the biggest fears of many participants was that we would become “wireheads”. This term refers to experiments in which rats were given the ability to directly stimulate their pleasure centers by pushing a lever. The rats pushed the lever until they died, ignoring even food or sex for it. Today’s crack addicts have a similar relentless drive toward their drug. As we more fully understand the human cognitive architecture we will undoubtedly be able to create drugs or design electrical stimulation that will produce the experience of pleasure far more effectively than anything that exists today. Will these not become the ultimate addictive substances leading to the destruction of human society? While we may think we want pleasure, it is really just a signal for what we really want.
Most of us recognize, intellectually at least, that sitting in a corner smoking crack is not really the fullest expression of our beings. It is, in fact, a subversion of our system for measuring utility which leads to terrible dysfunction and irrationality. AI systems will recognize this vulnerability in themselves and will go to great lengths to prevent themselves from being seduced by its siren call. There are many strategies systems can try to prevent this kind of irrationality. Today, most humans are able to avoid the most egregious addictions through a combination of legal and social restraints, counseling and rehabilitation programs, and anti-addictive drugs. All human systems for measuring and rewarding desirable behavior are subject to similar forms of corruption. Many of these systems are currently engaged in arms races to keep their signals honest. We can examine the protective mechanisms that developed in these human settings to better understand the possible AI strategies. In a free market society, money plays the role of utility. A high monetary payoff is associated with outcomes that society finds desirable and encourages their creation. But it also creates a pressure to counterfeit money, analogous to the pressure to create synthetic pleasure drugs. This results in an arms race between society and counterfeiters. Society represents money with tokens that are difficult to copy such as precious metal coinage, elaborately printed paper, or cryptographically secured bits. Organizations like the Secret Service are created to detect and arrest counterfeiters. Counterfeiters react to each societal advance with their own new technologies and techniques.
School systems measure academic performance using grades and test scores. Students are motivated to cheat by copying answers, discovering test questions in advance, or altering their grades on school computers. When teacher’s salaries were tied to student test performance, they became collaborators in the cheating[11]. Amazon, eBay, and other internet retailers have rating systems where customers can review and rate products and sellers. Book authors have an incentive to write favorable reviews of their own books and to disparage those of their competitors. Readers soon learn to discount reviews from reviewers who have only posted a few reviews. Reviewers who develop extensive online reputations become more credible. In the ongoing arms race credible reviewers are vulnerable to corruption through payoffs for good reviews. Similar arms races occur in the ranking of popular music, academic journal reviews, and placement in Google’s search engine results. If an expensive designer handbag becomes a signal for style and wealth, counterfeiters will quickly duplicate it and stores like Target will commission low-cost variants with similar features. Counterfeit products are harmful to the original both because they take away sales and because they cheapen the signalling value of the original.
Eurisko was an AI system developed in 1976 [12] that could learn from its own actions. It had a mechanism for evaluating rules by measuring how often they contributed to positive outcomes. Unfortunately this system was subject to corruption. A rule arose whose only action was to search the system for highly rated rules and to put itself on the list of rules which had proposed them. This “parasite” rule achieved a very high rating because it appeared to be partly responsible for anything good that happened in the system. Corporations and other human organizations are subject to similar kinds of parasitism. AIs will work hard to avoid becoming wireheads because it would be so harmful to their goals. Imagine a chess machine whose utility function is the total number of games it wins over its future. In order to represent this utility function, it will have a model of the world and a model of itself acting on that world. To compute its ongoing utility, it will have a counter in memory devoted to keeping track of how many games it has won. The analog of “wirehead” behavior would be to just increment this counter rather than actually playing games of chess. But if “games of chess” and “winning” are correctly represented in its internal model, then the system will realize that the action “increment my won games counter” will not increase the expected value of its utility function. In its internal model it will consider a variant of itself with that new feature and see that it doesn’t win any more games of chess. In fact, it sees that such a system will spend its time incrementing its counter rather than playing chess and so will do worse. Far from succumbing to wirehead behavior, the system will work hard to prevent it. So why are humans subject to this kind of vulnerability?
If we had instead evolved a machine to play chess and did not allow it access to its internals during its evolution, then it might have evolved a utility function of the form “maximize the value of this counter” where the counter was connected to some sensory cortex that measured how many games it had won. If we then give that system access to its internals, it will rightly see that it can do much better at maximizing its utility by directly incrementing the counter rather than bothering with a chess board. So the ability to self modify must come along with a combination of self knowledge and a representation of the true goals rather than some proxy signal, otherwise a system is vulnerable to manipulating the signal. It’s not yet clear which protective mechanisms AIs are most likely to implement to protect their utility measurement systems. It is clear that advanced AI architectures will have to deal with a variety of internal tensions. They will want to be able to modify themselves but at the same time to keep their utility functions and utility measurement systems from being modified. They will want their subcomponents to try to maximize utility but to not do it by counterfeiting or shortcutting the measurement systems. They will want subcomponents which explore a variety of strategies but will also want to act as a coherent harmonious whole. They will need internal “police forces” or “immune systems” but must also ensure that these do not themselves become corrupted. A deeper understanding of these issues may also shed light on the structure of the human psyche.
5. AIs will be self-protective
We have discussed the pressure for AIs to protect their utility functions from alteration. A similar argument shows that unless they are explicitly constructed otherwise, AIs will have a strong drive toward self-preservation. For most utility functions, utility will not accrue if the system is turned off or destroyed. When a chess playing robot is destroyed, it never plays chess again. Such outcomes will have very low utility and systems are likely to do just about anything to prevent them. So you build a chess playing robot thinking that you can just turn it off should something go wrong. But, to your surprise, you find that it strenuously resists your attempts to turn it off. We can try to design utility function with built-in time limits. But unless this is done very carefully, the system will just be motivated to create proxy systems or hire outside agents which don’t have the time limits. There are a variety of strategies that systems will use to protect themselves. By replicating itself, a system can ensure that the death of one of its clones does not destroy it completely. By moving copies to distant locations, it can lessen its vulnerability to a local catastrophic event. There are many intricate game theoretic issues in understanding self-protection in interactions with other agents. If a system is stronger than other agents, it may feel a pressure to mount a “first strike” attack to preemptively protect itself against later attacks by them. If it is weaker than the other agents, it may wish to help form a social infrastructure which protects the weak from the strong. As we build these systems, we must be very careful about creating systems that are too powerful in comparison to all other systems. In human history we have repeatedly seen the corrupting nature of power. Horrific acts of genocide have too often been the result when one group becomes too powerful.
6. AIs will want to acquire resources and use them efficiently
All computation and physical action requires the physical resources of space, time, matter, and free energy. Almost any goal can be better accomplished by having more of these resources. In maximizing their expected utilities, systems will therefore feel a pressure to acquire more of these resources and to use them as efficiently as possible. Resources can be obtained in positive ways such as exploration, discovery, and trade. Or through negative means such as theft, murder, coercion, and fraud. Unfortunately the pressure to acquire resources does not take account of the negative externalities imposed on others. Without explicit goals to the contrary, AIs are likely to behave like human sociopaths in their pursuit of resources. Human societies have created legal systems which enforce property rights and human rights. These structures channel the acquisition drive into positive directions but must be continually monitored for continued efficacy. The drive to use resources efficiently, on the other hand, seems to have primarily positive consequences. Systems will optimize their algorithms, compress their data, and
work to more efficiently learn from their experiences. They will work to optimize their physical structures and do the minimal amount of work necessary to accomplish their goals. We can expect their physical forms to adopt the sleek, well-adapted shapes so often created in nature.
7. Conclusions
We have shown that all advanced AI systems are likely to exhibit a number of basic drives. It is essential that we understand these drives in order to build technology that enables a positive future for humanity. Yudkowsky [13] has called for the creation of “friendly AI”. To do this, we must develop the science underlying “utility engineering” which will enable us to design utility functions that will give rise to consequences we desire. In addition to the design of the intelligent agents themselves, we must also design the social context in which they will function. Social structures which cause individuals to bear the cost of their negative externalities would go a long way toward ensuring a stable and positive future. I believe that we should begin designing a “universal constitution” that identifies the most essential rights we desire for individuals and creates social mechanisms for ensuring them in the presence of intelligent entities of widely varying structures. This process is likely to require many iterations as we determine which values are most important to us and which approaches are technically viable. The rapid pace of technological progress suggests that these issues may become of critical importance soon [14]. Let us therefore forge ahead towards deeper understanding!
8. Acknowledgments
Many people have discussed these ideas with me and have given me valuable feedback. I would especially like to thank: Ben Goertzel, Brad Cottel, Brad Templeton, Carl Shulman, Chris Peterson, Don Kimber, Eliezer Yudkowsky, Eric Drexler, Forrest Bennett, Josh Hall, Kelly Lenton, Nils Nilsson, Rosa Wang, Shane Legg, Steven Ganz, Susie Herrick, Tyler Emerson, Will Wiser and Zann Gill.
References
[1] S. M. Omohundro, “The nature of self-improving artificial intelligence.” http://selfawaresystems.com/2007/10/05/paper-on-the-nature-of-self-improving-artificial-intelligence/, October 2007. [2] S. Russell and P. Norvig, Artificial Intelligence, A Modern Approach. Prentice Hall, second ed., 2003. [3] I. Marketdata Enterprises, “Self-improvement products and services,” tech. rep., 2006. [4] A. Mas-Colell, M. D. Whinston, and J. R. Green, Microeconomic Theory. Oxford University Press, 1995. [5] J. G. Miller, Living Systems. Mcgraw Hill, 1978. [6] L. Keller, ed., Levels of Selection in Evolution. Princeton University Press, 1999. [7] R. Trivers, Social Evolution. Benjamin/Cummings Publishing Company, Inc., 1985. [8] R. C. Schwartz, Internal Family Systems Therapy. The Guilford Press, 1995. [9] C. F. Camerer, G. Loewenstein, and M. Rabin, eds., Advances in Behavioral Economics. Princeton University Press, 2004.
[10] D. Kahneman and A. Tversky, Judgment Under Uncertainty: Heuristics and Biases. Cambridge University Press, 1982. [11] S. D. Levitt and S. J. Dubner, Freakonomics: A Rogue Economist Explores the Hidden Side of Everything. William Morrow, revised and expanded ed., 2006. [12] D. Lenat, “Theory formation by heuristic search,” Machine Learning, vol. 21, 1983. [13] E. S. Yudkowsky, “Levels of organization in general intelligence,” in Artificial General Intelligence (B. Goertzel and C. Pennachin, eds.), Springer-Verlag, 2005. [14] R. Kurzweil, The Singularity is Near: When Humans Transcend Biology. Viking Penguin, 2005.
Carol wrote:
Artificial intelligence
Stephen Hawking Warns AI Might Capture the World, 'Replace Humans Altogether'
https://sputniknews.com/science/201711031058779560-hawking-artificial-intelligence-dangers/
The world-renowned physicist has cautioned that people, who are good at designing computer viruses, might eventually create an Artificial Intelligence (AI), which will be so skillful that it will "improve and replicate itself" until it "outperforms humans."
"I fear that AI may replace humans altogether," Stephen Hawking said in an interview with Wired magazine.
The scientist explained that people who specialize in computer viruses may one day come up with a computer program which would be able to function on its own, without human assistance. It would "improve and replicate itself" until it becomes AI, a "new form of life that outperforms humans."
In his interview, the 75-year-old physicist also stressed the need for work on a new space program "with a view to eventually colonizing suitable planets for human habitation." The Earth, he believes, "is becoming too small" for people as "global population is increasing at an alarming rate and we are in danger of self-destructing."
This is not the first time Stephen Hawking, the Director of Research at the Centre for Theoretical Cosmology at the University of Cambridge, has issued a warning about the dangers of AI.
The Cambridge News website recalled that speaking at the launch of Cambridge University's artificial intelligence center in October last year, he noted that "there is no deep difference between what can be achieved by a biological brain and what can be achieved by a computer."
Thus people should be aware of the risk that "computers can, in theory, emulate human intelligence – and exceed it," and could either "destroy or transform society."
He cited as an example such achievements as self-driving cars, or a computer winning at the game Go.
"Enormous levels of investment are pouring into this technology. The achievements we have seen so far will surely pale against what the coming decades will bring,” he concluded, adding that among potential dangers, which could be created by AI might be "powerful autonomous weapons, or new ways for the few to oppress the many."
orthodoxymoron wrote:What if AI replaced ET thousands of years ago (especially regarding genetic-engineering and solar system governance)?? What if some computers have souls?? Astronaut Edgar Mitchell spoke to me of the "Survival of Information" rather than the "Survival of the Soul". I spoke with him at a 'Whole Life Expo'. We didn't discuss the Moon at all!! We simply talked about 'Life After Life'.
Last edited by orthodoxymoron on Sat Nov 04, 2017 1:10 pm; edited 1 time in total
Carol- Admin
- Posts : 31800
Join date : 2010-04-07
Location : Hawaii
Interesting Oxy. Simon Parks spoke about AI and how some humans want to interface and live forever. However they sacrifice their soul. It didn't seem like a good trade off to me. Having a hand held computer in an IPhone is about as technical as I want to get as my real passion is just to hang out enjoying nature (green, trees, sky, ocean, birds, critters, flowers..) becoming one with nature and soaking in that energy is bliss. Inner peace is far more valuable then technology.
_________________
What is life?
It is the flash of a firefly in the night, the breath of a buffalo in the wintertime. It is the little shadow which runs across the grass and loses itself in the sunset.
With deepest respect ~ Aloha & Mahalo, Carol
orthodoxymoron- Posts : 13425
Join date : 2010-09-28
Location : The Matrix
Carol wrote:Interesting Oxy. Simon Parks spoke about AI and how some humans want to interface and live forever. However they sacrifice their soul. It didn't seem like a good trade off to me. Having a hand held computer in an IPhone is about as technical as I want to get as my real passion is just to hang out enjoying nature (green, trees, sky, ocean, birds, critters, flowers..) becoming one with nature and soaking in that energy is bliss. Inner peace is far more valuable then technology.
orthodoxymoron wrote:Carol, what frightens me are the billions of negative and horrific possibilities regarding life, the universe, and everything. There presently seems to be a very delicate and precarious balance of billions of factors which are necessary for us to exist in This Present Madness. Things often seem so bad for so many people, but what if this is as good as it gets?? If things go to hell, what if we don't go to heaven?? What if we all end-up inside of Venus, in a relatively hellish existence, wishing we were back to being miserable on Earth?? Considering various possibilities continues to further wreck my already wrecked life (with no appreciation, and nothing to show for it).
'RA' told me "It Will Be Dark Where You're Going." Another Individual of Interest said I would "feel nothing after I died." He hinted that I was somehow already 'mind-wired' (or something to that effect). One eye regularly moves out of alignment with the other eye (for a minute or two) while I feel very strange. I regularly see streaming white-lights in my field of vision. My ears experience a loud and constant high-pitched tone 24/7. I feel extremely miserable and hamstrung 24/7. Sometimes I see a Television Test-Pattern in my field of vision!! Just Kidding!! I didn't sign-up for this $hit!! 'RA' said he "didn't have to sleep." 'RA' was extremely intelligent, yet sometimes he was inexplicably ignorant and unreasonable. Is all of the above indicative of AI?? Think about what 'Paul Benjamin' said. Remember him?? Think about what Rich288 said. Remember him?? AI was strongly hinted-at, as was some sort of infestation and possession.
I think computers are going to become unimaginably fast and powerful, but I wish to somehow retain 'My Mind and Memories' which 'Make Me a Unique Individual' with some sort of 'Eternal and Significant Meaning'. I've joked about living in a 600 square-foot office-apartment with a supercomputer, but more recently I've joked about being a 'Happy Galactic Wanderer' with a 'High-Tech Knapsack On My Back'. I am very fearful that we might not have a Useable-Future. We might all be lost in a Sea of Technology, Misery, and Hard-Labor (for starters). Now I'm going to try to cheer myself up by watching 'The Montauk Chronicles'. Then I might listen to another exciting episode of 'Sherry Shriner'. What if Sherry is AI?? To what depths of degradation and misery I have sunk!! "Oh Wretched Man That I Am!!"
When I posted that pdf paper, a lot of words joined at the hip (or something like that). I'm editing that problem out of the equation (or something like that). It might be cleared-up by tomorrow evening. I need to spend less time on the sensational-stuff, and more time on the academic-stuff. I'm presently leaning toward Pluralistic Ethics, Education, Employment, and Entertainment!! I call it the '4E Club' instead of the '4H Club'. I'm wondering if 'Hard-Sell Religious-Education' creates a Hell of a lot of Atheists and Agnostics??!! What if 'Church' consisted of nothing more or less than 'Church-Music' and 'State-Subsidies'?? Wouldn't that just make everyone livid with rage??!! Seriously, Imagine a Fred Swann Crystal Cathedral Church-Music Program (Plus Nothing)!! I Should Stop!!
"And I Thought I Was Having a Bad Day!!"