No Comments on THE DANGERS OF SUPER AI PART 2

Part 1 of this series can be found here.


In order to truly understand the potential dangers that super intelligent AI presents us in the future we must figure out what forms it could take. There are three main possibilities that come to mind when considering this.

The first one resembles a psychic or an oracle, it would take questions as input and output answers to those questions. The second one resembles a task completing system that would take commands as input and it would execute the command as an output. The third one resembles a piece of software that acts like a tool.

This third possibility has been put forward by a number of experts on the premise that it would be a safer alternative to the first two. However, when you dive a bit deeper you’ll see that it’s not so simple. Let’s break down each of these forms and consider the dangers that will coincide with them.


A program that possesses intelligence that astronomically towers over any human would be able to answer any question given to it with greater accuracy than we have ever seen before. There would be many advantages to having access to a program such as this. There would also be many potential risks. In the world of AI you simply can’t have it both ways.

This program would probably have to answer questions in a yes/no format to keep things simple with perhaps some way to express a level of confidence or a statistical chance that it’s answer is right or wrong according to its own estimates. It could be created to answer questions in a specific narrow domain. For instance, one could create a mathematics oracle that could answer any math problem that would take humans centuries to solve in only a few minutes. In some form AI’s similar to this already exist. A calculator could be viewed as an extremely weak version of the program we are imagining.

Once could also create a question answering AI that had the capacity to answer questions across an infinite number of domains. Modern day search engines can be pointed to as an example of a very weak version of this. This AI would possess super intelligence and would make whoever has access to its answers extremely powerful.


This AI would receive a high level command and carry it out before pausing to receive the next one. It would resemble a hyper efficient mechanical soldier that exists on a server and performs whatever you need it to with unbelievable accuracy. This idea has been described as one of the most practical and sensible forms of super AI.

The first problem that comes to mind is that unlike the question answering AI our ability to control a super AI that executes tasks in this manner is extremely limited because such a program would need to be able to manipulate the world around it in order to carry out our commands as opposed to an oracle that would be kept in a box where it could do no harm.

The other problem is interpretation. It’s easy to assume that a command executing AI that possessed super intelligence would know exactly what we wanted it to do when it receives a task and would therefore execute them according to our liking. But there is no guarantee whatsoever that this program would actually do what we wanted it to despite knowing our exact intentions. The only real way to solve this issue would be to gain complete control over its motivations and its values.


This design, in contrast to the others, would probably be easier to control since it would not have a will of its own. It would act similarly to most pieces of software we see today but would be much more flexible and capable. On the surface this seems like the best option. The drawback is this Ai design would not have sufficient capabilities to solve the many existential problems that plague humanity. It has been theorized before that we could eliminate the need for super intelligent AI by expanding upon the range of tasks that current AI programs can perform. However, most experts agree that this line of thinking is inherently flawed.

The range of tasks that a program with general intelligence could perform is vastly wider, making it infeasible to create special-purpose software without general intelligence to handle all of that. Even if it could be done it would take an extremely long time, so long that by the time it was completed the methods for performing these tasks efficiently would change. This is especially true when you consider that writing software requires a programmer to understand the task to be performed in enough detail to do it on their own. A program that can teach itself things eliminates this need.

The bottom line here is the risks of developing an AI with general intelligence do not measure up to the various existential risks to humanity from not developing this technology but that is beyond the scope of this post and will be covered in more detail later on.


Now that we have a general idea of what futuristic AI might look like we need to consider ways in which we could control something that possesses far greater intelligence than humans. This will require lots of preparation on humanity’s part and a willingness to understand the importance of not developing certain technologies until we have a firm grip on the control problem.

Given the three ideas mentioned in the beginning of this post the question answering AI kept in a tight box is probably what we should aim for until we can figure out a way to control the command executing AI. We will have to be very careful when assigning a question answering AI a specific goal during the programming process.

Consider the possibility that we create a program like this with the goal of maximizing the accuracy of its answers. This would seemingly motivate the AI to teach itself everything it would need to know in order to make this happen right? Wrong, a super intelligent AI of this type given a goal like that would simply manipulate us into giving it easier questions. In order to solve that problem you could just scrap the AI after it answered one question and use the code to run a new one for each new question but that would render the AI’s ability to continually learn over time useless.


Alternatively it could be given a goal of maximizing the impact of its answers on humanity as time goes on but this would only be possible if we had a very tight grip on the motivations and values of such a program. The values would have to be perfectly aligned with our own or this won’t work. As you can see there are no easy answers to even the most basic questions. The image below highlights what we might be dealing with.

Image Source: Work of Iyad Rahwan and used through Creative Commons Attribution-ShareALike 4.0 International License.

Hijacking the program’s motivation system would be a possible means to control it because it stands to reason that if the program experiences no motivation to ever leave the it’s box then it won’t have a reason to try and manipulate us into letting it out somehow. This appears to be our best course of action to start out with. If we could control the motivations in this manner we would be able to use the intelligent question answering AI to help us solve the control problem associated with the command executing program which would be a necessary first step.


In this post we described more in depth the possible forms super AI might end up taking and some potential dangers associated with each of them as well as some general precautions we must take after they are created. The next post in the series will explore the various methods of “boxing” an AI and ways we could control the motivations and values of an AI to make sure they align with our own which will ensure that our future remains bright.

Leave a Reply

Your email address will not be published. Required fields are marked *