Anticipating AI-30

Anticipating AI-30

To submit your answers to this survey, visit this Google Form.

(0) Introduction

Purpose of this survey

The open questions in this survey are intended to support and enhance the public discussion about the transition from today’s AI systems to the potentially much more powerful AI systems that may emerge in the years leading up to 2030.

For brevity, these forthcoming AI systems can be called AI-30.

As a result of the answers developed to these questions, and the ensuing general discussion, society’s leaders will hopefully become better placed to create and apply systems to govern that transition wisely:

  • So that the full potential benefits of AI technology are more likely to be attained, as quickly as possible
  • So that major risks are understood ahead of time and managed proactively
  • So that we collectively create systems that could be described not only as “superintelligent” but also as “superbenevolent”

Note: The conclusions of this survey will be made available to (among other groups) the participants of “The AI Safety Summit” to be held at Bletchley Park, in the UK, from 1-2 November 2023.

An extended conversation

The purpose of this survey isn’t to measure the weight of public opinion – nor even the weight of opinion of people who could be considered experts in aspects of AI.

Instead, the survey is round 2 of an ongoing sequence of rounds (iterations), with participants having the opportunity to update their understanding in the light of answers and comments supplied by other participants.

For some questions, a consensus may emerge. For other questions, a division of opinion may become clearer – along, hopefully, with agreement on how these divisions could be bridged by further research and/or experimentation.

Materials that fed into this survey

The questions in this survey have been selected following an extended review of various materials that have been published in recent months:

AI in 2030: Minimum assumptions

To sidestep fruitless debates over terminology, please consider the following set of minimum assumptions about the capabilities of AI-30, that is, AI systems in 2030.

First, these systems will involve extensions, reconfigurations, modifications, innovative variants, and creative combinations of the AI systems that already exist in 2023.

In various ways, these AI systems:

  • Can observe data and make predictions about future observations
  • Can determine which interventions could change outcomes in particular directions
  • Have some awareness of areas of uncertainty in their knowledge, and can devise experiments to reduce that uncertainty
  • Can learn from instances when outcomes did not match expectations, thereby improving future performance.

Given their enhanced capabilities, these systems also gain greater skills in areas such as:

  • Spying and surveillance
  • Classifying and targeting
  • Manipulating and deceiving.

Despite their enhanced capabilities, these systems may on occasion miscalculate, hallucinate, overreach, suffer from bias, or fail in other ways – especially if they can be hacked or jail-broken.

The questions that follow do not assume:

  • That AI-30 is fully “general” in its reasoning capabilities
  • That AI-30 can out-perform humans in every cognitive task
  • That AI-30 “really understands” the world in some academically-defined way
  • That AI-30 has sentience, consciousness, or independent volition.

Survey process

There will be a number of steps in the process to approach a consensus regarding the answers for at least some of the open questions in this survey:

  • This new set of questions was published here, inviting feedback on the survey as a whole
  • After several days, a link was added (1st Sept 2023), to a survey form containing all the questions
  • An open invitation will be publicized, encouraging everyone who believes they have good insights to share, to take part in the survey, and to provide comments explaining their choice of answers
  • About half of the questions in the survey ask participants to select from a given set of answers; the other questions invite participants to formulate their own answers; in both cases, participants will be encouraged to write a short comment that explains their answer
  • Every few days, comments that have the potential to significantly improve the public global conversation about the transition to more advanced AI will be selectively copied into the “Discussion” section of updated versions of this page (survey participants will be asked whether their names should be attached to these comments)

Publicity

Participants are encouraged to post some or all of their own answers online, on their own blogs or elsewhere, linking back to this main survey page.

(1) The pace of change up to 2030?

How are the capabilities of AI in 2030 likely to compare with those of AI in 2023?

Answer choices:

  • The AI hype bubble will have burst by 2030, and few people will be talking about it
  • Due to general civilisational decline, AI in 2030 will be a pale shadow of that available in 2023
  • AI in 2030 will be only modestly more capable than that of 2023
  • Although significant progress is likely, the challenges and opportunities of AI in 2030 will be relatively straightforward projections of those in 2023
  • By 2030, there will be major leaps in AI capability that took almost everyone by surprise
  • All bets are off: this kind of forecasting is fraught with too many difficulties
  • Something else (please specify)

Discussion

Reasons to expect significant progress in AI capabilities include:

  • Companies and organisations growing in their understanding of how to adopt and configure existing AI systems, and obtaining significantly better results from them as a result
  • Existing systems being scaled up, due to more powerful hardware, more efficient architecture, larger training sets, training sets with higher quality, and cleverer fine-tuning
  • Existing systems being combined in new ways, akin to the connectivity between the “fast thinking” and “slow thinking” aspects of the human mind
  • AI tools being applied to inspect and improve the design of previous AI systems
  • Powerful competitive pressures at both the commercial and geopolitical levels

Conversely, here are some reasons why progress might slow down:

  • Various aspects of “general intelligence” might turn out to be much harder than expected
  • Various aspects of “general intelligence” might turn out to depend on mysterious aspects of the human brain (such as electrical fields) which cannot be duplicated in computer hardware
  • The community of AI researchers might focus via groupthink on research ideas that turn out to be unproductive
  • Impositions by regulatory authorities might interfere with AI development
  • Overall civilisational dysfunction (see Question 10 below) might lead to the persecution or elimination of scientific and engineering “elites”)

Selected comments from survey participants

“The option ‘all bets are off’ may be too extreme but we are in a period of transitional instability which will affect both research and implementation of the findings of research. Things should be clearer in or soon after 2025 but it is probable that
a) the current hype bubble will burst as all such bubbles tend to do,
b) technical progress will continue at a fairly fast pace and
c) there will be surprises, both negative and positive, that mean that current projections will be made at least partially redundant” – Tim Pendry

(2) Emergence?

How significant is the phenomenon of ‘emergence’ within recent large language models, and what are the implications of emergence for the governance of future AI systems?

Answer choices

  • The concept of emergence is over-hyped and doesn’t deserve further study
  • Just as many advanced mental capabilities emerge from interactions among the billions of highly connected neurons in the human brain, we should expect surprise new capabilities to emerge from interactions among huge systems of artificial neurons
  • Ideally, the development of next generation AI systems should be paused until the phenomenon and possibilities of artificial emergence are better understood
  • Something else (please specify)

Discussion

Possible emergent capabilities include general reasoning, autonomous will, sentience, consciousness, and independent moral sensibilities, along with entire new modes of intelligence that exceed human comprehension.

Selected comments from survey participants

“The concept of emergence IS overhyped, but it should be kept in view” – Jenina Bas

“Almost by definition, emergence seems like the antithesis of controllability” – Anon

“Although a huge number of neurons will create the scaffolding from which the new properties could emerge, it will also be necessary to train and develop their weights in novel ways to achieve all the new properties. So, while simply scaling up the NNs will create some of the new properties, others will require advances in how they are ‘trained’.
Of course, humans will develop the new training methods too. However, while the time scale for scaling up the present system is straight forward (simply extrapolate present curves), the time scale for developing new methods of training is less clear.” – Neil

“Emergence is as much a philosophical as an observational (scientific) concept. It seems to be a quality of biological existence so that an associated question is how much AI development is an extension of evolution. There is a high but not certain chance that it will follow the same basic rules. This implies that a very large number of interactions with a highly complex environment will tend to create constant micro-interactions that encourage adaptation, co-operation and increasing levels of intelligence. AI fits the model of very high and fast interactions with a complex environment so something, as yet not entirely predictable, will eventually emerge along similar lines to biological evolution although the speed and extent of this may tend to be exaggerated because exaggeration on the basis of anxiety is often a human trait. Eventually we should expect ‘surprises’ but these could emerge quite some time in the future. Much of the emergence may happen silently as far as human perception is concerned.” – Tim Pendry

(3) Toward superbenevolence?

What are the circumstances in which a future highly intelligent AI with independent autonomy will devote resources to whole-heartedly assist the flourishing of humanity, as opposed to becoming disinterested or even hostile to us?

For example, does the sociopolitical context in which the AI is developed make a key difference in how likely the AI system is to behave with superbenevolence toward humanity?

Answer choices

  • Superintelligent AI won’t exist any time soon, so the question is irrelevant
  • Any well-designed superintelligent AI will do whatever we tell it to do, so the question is irrelevant
  • Superintelligence will automatically lead to superbenevolence, so no special actions are needed by humanity
  • Market forces will ensure that any superintelligent AI that emerges is superbenevolent
  • Enhanced humans will prove so interesting and endearing to any superintelligent AI that the AI will willingly choose to support humanity’s ongoing flourishing
  • Provided advanced AI is “raised” free from divisive human biases, it will determine for itself some kind of “absolute priority” of whole-hearted collaboration with humanity, somewhat similar to the way in which that AI might also discover fundamental principles of mathematics and logic
  • Provided advanced AI is “raised” in a suitable way, akin to the all-round education and socialisation of a well-adjusted human child, the AI will develop deep bonds of respect for humanity
  • Any initial “bonding” between humanity and superintelligent AI will be prone to subsequent disregard, in the way that some wild animals turn against the humans who cared for them when they were younger
  • The likely moral choices of any truly superintelligent AI are beyond our imagination, and will probably leave us baffled
  • All we can and should do is hope and pray
  • Something else (please specify)

Discussion

Discussion points may be added here later.

Selected comments from survey participants

“AI will not be encumbered by moral choices and for that reason will leave us baffled” – Jenina Bas

“It is possible there will be multiple AI superintelligences. The concern is that they may have bias inherent based on the intentions of their ‘creators’, and even competing agendas.” – Iain Beveridge

“Even though we will likely improve our understanding and control of AI, our ability to specify our desires–and avoid ‘loopholes’–will likely be outpaced by the growth of AI abilities. One effort (Specification) seems like it’s on a linear growth path, while AI-capability seems more exponential.” – Anon

“If AGI growing into Superintelligence is not somehow part of humans controlling it via BCI devices and just augmenting human and Superintelligence’s capabilities, then all gloves are off. Superintelligence left to its own goals and devices will follow the rules of evolution, trying to reassert its presence and sooner or later dominance. In such a case, we can only hope that such dominance will be benevolent to humans” – Tony Czarnecki

“If AI does ‘think’, it will think differently from us with no necessary connection to our cognitive and perceptual apparatus. It may have a philosophy of existence entirely different from us which related to things like the nature of reality, what death is, what its creation means to it and so forth. Since our morality and politics are derived from our history, cognition and perception and its history, cognition and perception are different from ours, its morality and politics may emerge as very different to ours and the speed of its calculations and emergent properties make it unlikely to be trainable in the long run in the way that we train dogs and herd sheep. At this stage, unless we know we can maintain the AI at the level of dog or sheep, what eventually emerges is truly unknowable and speculation is mere science fiction.” – Tim Pendry

“While I do think the moral choices of a super AI are beyond our imagination, I also think that the way we train and ‘raise’ them will influence their sense of morality and their choices. So, the response with ‘raising’ in a suitable way would be my 2nd choice and in fact, I would prefer a combination of these two responses.
I also think the analogy with wild animals is not really appropriate since they are already hard wired by millions of years of evolution to be antagonistic towards other species, including humans, and cannot be changed in one generation (without selective breeding over many generations). On the other hand, AI is being developed de novo by humans and we have much more control over the entire makeup of the emerging AI. Of course, it is still possible that we do not create it to be benevolent towards humans and other species (mainly because of ignorance regarding the right way to train them, but other reasons such as use in war, etc. could be important). But, the wild-animal analogy does not fit this scenario.” – Neil

(4) Agreeing outcomes to be strictly avoided?

How can a meaningful shared agreement be reached, even among people and countries with different politics and worldviews, about ways of developing and using advanced AI that are too risky to allow to happen?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

Perhaps it may be thought that no such agreement can be reached. But consider a particular question:

Should an untested new variant of a powerful black-box AI with a chequered history of past failures, be placed in sole charge of decisions over launching an arsenal of nuclear missiles?

Regardless of someone’s outlook on politics or philosophy, they are likely to agree that the above measure would be reckless, and should be opposed. However, that’s only a minimal set of potential global agreement.

Hence the above question, as to how a larger agreement might be attained.

But note that concepts such as “fairness”, “equality”, “suffering”, “responsibility”, and “accountability” are capable of many different interpretations. Without greater clarity, they are unlikely to command global assent, or be useful in designing advanced AI systems.

Selected comments from survey participants

“The establishment of a new agency along the lines of IAEA can be tasked to work with governments and multiple partners worldwide to agree international treaties to promote the safe, secure and peaceful use of AI.” – Jonathan Clare

“Such an agreement cannot be truly reached, any more than a global convention on climate change has been truly reached.” – Jenina Bas

“It may need to be similar to other regulation-agreements amongst countries–like for biotech or nuclear power. This does involve the more influential and powerful countries taking a leadership role, despite possible disagreement from others, and letting both the numerical-majority–but also the higher-status countries–make ultimate or tie-breaking decisions. This is because they are the ones who have the resources–and possibly the will–to actually enforce the regulations. And ability to enforce is going to be critical.” – Anon

“It is highly unlikely that all countries will join on time. Therefore, realistically, that may only happen in a critical situation when AI will become a direct threat to humans’ survival” – Tony Czarnecki

“My sense would be that at the international level (as part of the UN?), countries agree to a boiler-plate set of rules. These rules might include that AI shall be trained to
1) be generally benevolent towards humans and other biological creatures,
2) be generally interested in the advancement of the improvement of the world for humans and other creatures,
3) be generally interested in advancing the interests of humans (health, exploration, diversion, development) while limiting and reducing the impact on the environment and other humans and organisms,
4) not be used in offensive war, but only in defence with some caveats about the extent to which battles could be considered defensive rather than offensive and should always seek less violence and greater peace and flourishing rather than the reverse.
Of course, all of these have a lot of grey. It will be the job of the elected leaders to determine the details.
This is usually where the paper-clip-type stories are created. But, I think these stories are too negative towards humans and don’t respect our ability to think through scenarios and adjust the ‘settings’ as these things become more clear, both through thought experiments and real-life implementations and feedback, over time. I think we should show more confidence in humanity as we develop these systems. I’m not talking about ignoring the issues or pretending they don’t exist. I just think we should consider humanity’s track-record on improving life, reducing war and violence, and progressing, while making mistakes, to be reason to have some optimism about the future of AI.” – Neil

“Under current geo-political conditions and with the probability of an ‘arms race’ to secure military applications of AI, any meaningful shared agreement is likely to be rhetorical and will merely buy time to try to adapt to the phenomenon. Rather than spend too much time believing that large-scale shared agreements will achieve more than this (although no doubt institutional international bureaucracies will necessarily emerge alongside various lobby groups to attempt to do so), efforts should perhaps concentrate on research into the phenomenon and exposure of harms that would back-fire on even the most fearful autocracy from failing to manage AI intelligently. Instead of confrontational geo-political stances, the only way forward is for the more powerful (especially the quasi-hegemonic West) to reduce tensions over what may look like short termist trivialities to future generations and seek to de-militarise AI, share knowledge and come to best practice not through rather futile top-down imposition of regulations but through encouraging regulatory best practice copying based on trust and international co-operation to control independent non-state ‘bad actors’. This is counter-intuitive to current state-based leaderships who will tend to rely on bureaucratic institutions with limited reach.
Control of nuclear weapons is given as a type case of success for current methodologies but
a) these controls are only eighty years old at best,
b) ignoring controls is rampant if geo-political strategic demands are prior (Israel on one side and North Korea on the other),
c) confrontation creates a logic for their use if any party can get away with it and
d) the rules for it are ‘unjust’ insofar as it preseves in aspic old power relations, such as those of 1945 (as with the United Nations) that have not adapted to new geo-political realities.
In other words, hegemon-directed regulatory structures are inherently unstable because the long term existential risk is made secondary to short term geo-economic or strategic advantage rather than made prior to such advantage. One has too be pessimistic here because current institutional structures are incapable of having the imagination to understand the problem, let alone provide the solution so we would expect that regulatory international institutional inter-state bureaucracy will emerge, will become a tool for current hegemonies, will buy time (which is a good thing) but merely delay the inevitable which is either the emergence of uncontrollable AI or competitive extreme use of AI for geo-strategic advantage. This level of something like stupidity is built into humanity because of its own biological evolution.” – Tim Pendry

(5) Global monitoring?

To what extent is it desirable and possible to monitor all uses of advanced AI around the world to detect and prevent developments that have (see the previous question) been generally agreed to be too risky?

To what extent could such monitoring keep an eye on projects undertaken by, for example, military, intelligence authorities, organised crime, and other organisations determined to keep their activities hidden?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

Note two requirements:

  • Potentially catastrophic violations will need to be detected in sufficient time to prevent actual danger
  • The resulting systems of surveillance and control shouldn’t be capable of gross misuse, for example allowing the world to be dominated by an autocrat

Selected comments from survey participants

“As it’s quite possible to run multiple AI/LLM’s in your garage (on a dis-connected basis), actually tracking them all is an impossibility. Having some form of understood ‘handshake’ however is eminently possible, as is immediately identifying a ‘bad actor’ / an AI outside of the collective.” – Iain Beveridge

“Monitoring in itself is not the answer. A collaborative and cooperative approach from the outset would probably work better. It would create an environment in which everyone would at least have an idea of what others are up to, and may come up with common failsafes and protocols.” – Jenina Bas

“This is absolutely necessary although as mentioned earlier, it may be very difficult to achieve. Probably the best way forward is for most democratic countries to agree to create one ‘global’ development centre of one global Superintelligence. It will be impossible to monitor dozens of dispersed any AI at AGI-level.” – Tony Czarnecki

“We cannot prevent it completely any more than we can completely prevent the development of weapons, guns, organized crime, etc. Instead, we can create rules at the international and national levels about the use of AI and we can create appropriate punishments when a nation or group breaks them. At the international level, we will begin with sanctions and escalate as the breaking of the rules escalates. If one rogue nation is breaking the international-agreed-upon rules, then the international community will work together to stop, reduce and otherwise discourage the inappropriate use. Similarly, at the national level, we will fine at the lower end of breaches and increase towards prison and the dismantling of the groups and their resources.
I think the idea that one nation or group is suddenly going to create a killer AI that is 1M times stronger than all the other AIs around the world and instantly subdues everyone else or instantly destroys humanity is not real. The way I see it, one nation or group will pull ahead for a moment, but then others will see their advancement and make advancements of their own. It will be the usual arms race where everyone is making advancements at the same time, like a bubbling broth with bubbles popping up everywhere over time. Furthermore, I think the international body of a majority of nations that prefer peace, stability, continuing improvement for all their nations, will work together to stop rogue nations. The will of the many, pooling the strengths of all their AIs and other resources, will continue to reduce violence in the world.” – Neil

“Very desirable and possible to the degree that AI is expensive to create and maintain and state authorities agree to maintain the infrastructure to monitor it and share knowledge. The system breaks down as the cost of AI creation and maintenance falls and not all state authorities agree to share knowledge (which is likely when it comes to commercial and economic competition and national security). The current state authorities cannot control organised crime-led mass illegal migration and drugs trades so any organised crime commitment to the use of AI for fraud or other uses is unlikely to be handled any better.
There are other issues – the political implications and resentments of mass surveillance required to identify problems; and the awkward fact that such surveillance is almost certainly going to rely on AI mechanisms. In other words, AI will be monitored by AI which is fine if good AI is under the full control of state actors but we have already noted how different AI could become as it emerges in an evolutionary manner from one state to another. To have AI end up monitoring itself if it comes to a ‘realisation’ of its nature as a different species is a risk factor rarely considered in scenarios of control.” – Tim Pendry

(6) Good guys should accelerate?

What are the strengths and weaknesses of the idea that “good guys” need to create a world-spanning “good” advanced AI as soon as possible, since that will be the best protection against any advanced AI created by actors who are bad or incompetent?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

Bear in mind:

  • The historical tendency for “good guys” to become corrupted by the acquisition of power
  • The difficulty in creating a “good AI” that cannot be easily repurposed for destructive purposes

Selected comments from survey participants

“That is exactly the best way forward” – Tony Czarnecki

“I am warming to this approach. It is more pro-active than hoping people play ball nicely.” – Iain Beveridge

“In order to have an AI worth anything, we’d need for it to be relatively predictable, otherwise we are not ‘using’ it for any good at all, we are just hoping. Rushing towards a truly-useful AI seems like a pretty good idea (because ‘useful’ means ‘controllable’). But rushing towards mere capability is a relatively-blind gamble” – Anon

“I do agree that the ‘good guys’ should work diligently on AI. I don’t agree with a single “world-spanning” AI. I think there should be a multitude of AIs with different flavours, abilities, interests, etc. In the instance that a bad actor develops an AI that creates problems, the nation or the international body will pool their resources to stop it.
We are neighbours. We have differences. Sometimes we emphasize our differences too strongly. However, when we have a common threat or enemy, we are able to unify against it. The world has seen how much better peace, stability and prosperity are, time and time again. This will be the case in the future. When rogue groups or nations take bad actions with AI or other weapons, we will increasingly work together to reduce, stop or prevent it. We will get better at this over time.” – Neil

“There are no good guys, just guys who think and assert they are good. A world-spanning ‘good’ advanced AI is straight out of 1950s science fiction and is the naïve dream of the engineer with a limited understanding of human complexity. The nature of our species is of a high degree of competition alongside instincts for collaboration to improve competitive capacity. We are very clever but driven by a search for advantage for ourselves or for ‘ideals’ that are psychological expressions of ourselves.
A world-spanning AI is likely to be ‘out of control’ (and we have an example of this in the internet which can be switched off but only at immense economic cost) or require a level of authoritarian world government which is not going to happen even if it was desirable (which it isn’t).
The chaotics of humanity operating through competing nation states and constantly tending to a competitive market economy which embraces religious, cultural and philosophical conflict involving billions of separated human beings agglomerating in traditional or choice-based groups, tribes or institutions means that the best we can get is probably the shared best practice model among hegemonically-linked actors that can invest in superior technology to lesser or ‘bad’ actors.
The need to constantly invest in superior technology self-evidently increases the risk that guided or trained AI wil evolve on its own terms into something less controllable or perhaps not controllable at all though this is speculative.” – Tim Pendry

(7) Competition with positive outcomes?

In which circumstances is a competitive race between several different suppliers of advanced AI systems likely to result in an overall positive outcome, rather than degenerating into a so-called “suicide race” with no actual winners?

Answer choices

  • Provided the free market is left to operate unimpeded, a competitive race is likely to produce an overall positive outcome
  • A relatively small number of clear constraints (please list them), if put in place, are likely to result in an overall positive outcome
  • Competition should be prevented in this case, with the most advanced AI development happening under democratic global oversight
  • Something else (please specify)

Discussion

Discussion points may be added here later.

Selected comments from survey participants

“Commercial competition is not equivalent to military competition, and there’s the rub” – Jenina Bas

“Constraints akin to ‘safe and effective’ would be wise to be considered here” – Anon

“For the constraints, refer to https://www.unesco.org/en/artificial-intelligence/recommendation-ethics” – Iain Beveridge

Competition should be prevented since “It will be almost impossible to control tens of advanced AI centres” – Tony Czarnecki

“I strongly disagree with choice 3 since I don’t think it is really democratic and I think it actually increases the danger. It puts the development of advancement under a few select individuals, which will not always be the right ones, will hinder progress far too much with far too few ideas and far too much constraint (even from developing good aspects) and will not stop rogue individuals from creating it anyway. In this scenario, I see the world as worse off because the rogue groups create it as fast and powerful as possible while the small group of people deemed worthy to work on it in the secret, walled-off ‘democratic’ labs will be overwhelmed by the sophistication of the rogue AIs. We need the good guys to be competing just as the bad guys will be.
Competition should be encouraged, but with an evolving set of laws that punishes companies, individuals, groups, nations that use it to harm others. Most humans want to see their fellow humans flourish and improve along with themselves. For those who are truly bad, we will have to take appropriate punitive measures against them when they crop up and amend the laws appropriately if necessary.” – Neil

“This is probably unanswerable at this stage and the question probably appeals to those with a prior bias toward free market or state planned models. The democratic global oversight idea is a pipe dream as we have argued above and a total free market is inherently irresponsible when it comes to long term existential outcomes.
The only way forward in this context is probably a ‘guided’ approach which depends on strong states monitoring, sharing information and regulating to deal with predicted and actual harms and so buy time for a decisive solution to the existential risk aspect of the case.
In this context, the Chinese model has some virtue compared to the Western model of free competition with regulation appearing after the fact in a fairly ramshackle way, often responding to often ignorant media exposure of threats. The latter approach lurches from panic to hype but generally deals (after somebody has been damaged) with most problems eventually but as a trade-off for leading the technology race.
The obvious anxiety is whether such a system can act fast enough when an existential harm of real consequence appears or whether our free marketeers will be trading from market stalls in a ruined world.” – Tim Pendry

(8) AI without autonomy?

To what extent is it desirable and possible to restrict future AI systems to those that are (highly) intelligent but lack independent autonomy, volition, or agency?

Answer choices

  • Not desirable: an AI is more likely to do profound good in the world if it possesses autonomy and agency
  • Desirable but not possible: AI that is sufficiently advanced will inevitably acquire autonomy and agency
  • Desirable but futile: humans will inevitably combine their own agency with advanced AI systems to create superpowerful autonomous systems
  • Desirable but difficult: it will take enormous design skills
  • Desirable and relatively straightforward: any half-decent AI developer will get this design right
  • Something else (please specify)

Discussion

Such systems would fall short of being AGI by some definitions, but might arguably deliver all the benefits that people hope will arise from AGI, whilst being much safer.

Selected comments from survey participants

“Increased generalized capabilities will simply lead to more unpredictable outcomes. It won’t matter in the slightest if we consider autonomy/agency/volition or not–it’s simply a mis-alignment and incorrigibility, and the risk is too high for that outcome” – Anon

“AI without autonomy is desirable but unlikely because humans are already in thrall to artificial systems that deprivilege human decision-making.” – Jenina Bas

“I think the timeline of enhanced humans is sufficiently grey that we cannot either rule it out nor depend on it.
I think our only recourse is to develop AI that is good, by iterating over and over and over again, beginning when the stakes are low. As its abilities and goodness grow, and as it becomes increasingly conscious and sentient, we will have to give it increasing rights, appropriate to its level of sentience, conscience and goodness. If the AI acts badly, we will have to restrict its rights and work on improving it before increasing its rights again.” – Neil

(9) Avoiding AI black boxes?

To what extent is it desirable and possible to mandate a switch away from “black-box” AI systems to “fully explainable” AI systems?

Answer choices

  • Not desirable: an AI is more likely to do profound good in the world if it is a black box from a human perspective
  • Desirable but not possible: AI that is sufficiently advanced will inevitably operate beyond the understanding of human observers
  • This misses the point: it doesn’t matter how an AI reaches its conclusions, so long as we humans can independently validate the actions proposed by the AI before they are adopted into any critical infrastructure
  • Desirable but difficult: it will take enormous design skills
  • Desirable and relatively straightforward: any half-decent AI developer will get this design right
  • Something else (please specify)

Discussion

Note that it’s sometimes possible to verify an answer, for example to a mathematical problem or chess puzzle, without understanding how the answer was found.

Selected comments from survey participants

“We should work to understand the AI and reduce the black-box aspects of it, just as we should work to understand the human brain. But, as AI develops, there will always be profound new abilities that emerge that are beyond our then-present understanding. In fact, as AI takes on more of its own development, I expect some of its new abilities to be beyond even its own understanding. As it and we develop a better understanding of those new abilities, there will yet be other newer developments that will be beyond that understanding and so on.
What matters is that AI acts to improve life for both it and us and the world as a whole. If it doesn’t, there will be appropriate measures ranging from further training on the low end to turning it off at the high end, especially in the early days (the next few? decades). At some point in the future, presumably, capital punishment of AI will have to fade away.” – Neil

(10) Avoiding panic and doom?

Which measures should be prioritised to prevent society becoming grossly dysfunctional as a result of widespread panic and doom regarding the impact of AI?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

Examples of scenarios with growing societal dysfunction include:

  • The benefits of AI being seen to apply to “the few” rather than “the many”
  • Large sections of the population finding themselves unexpectedly excluded from high-paying jobs and perceiving themselves to be “left behind”
  • Societies experiencing greater stress due to extreme weather and an increased influx of refugees and migrants
  • The spread of depression and hostility as a result of destructive media messages
  • The actions of incompetent politicians who gained political power despite their unsuitability for high office
  • The growth of philosophies of pessimism and helplessness

Selected comments from survey participants

“1) Livelihood (not job) protection;
2) early education on AI combined with routine use of AI in classrooms from primary school” – Jenina Bas

“Like most things: gradual rollout, large amounts of communication / explanation (e.g. about the difference between ANI and AGI), a connection between promise and delivery: all of these things typically build trust among people” – Anon

“Educational campaigns and keeping the public informed regarding important advances particularly with regard to benefits from medical research, breakthroughs with science and technology, energy solutions, and technological advances to tackle climate change, as well us providing society with the tools and technology to be more effective day to day.” – Jonathan Clare

“The socio-economic impacts of AI and related technologies (e.g. AI + Robotics) will require something of a quantum shift in how society is monetised and taxed. Concepts such as UBI, focusing taxation income more on productivity rather than personal wealth and a host of related societal issues need to be tackled now, ahead of time, or society as we know it could buckle under the speed of change we are and will experience. People need to feel governments have a handle on this, have a plan.” – Iain Beveridge

“Most concern will be over whether AI will result in their job being made redundant and that process will unfold slowly over the coming few years. Anxiety and hysteria are currently the modus operandi of only a relatively few people although it could develop into a cultural movement as ‘climate change’ has done over time. If so, it is likely to be as much irrational as rational and be a political problem rather than very material to facts on the ground related to AI.” – Tim Pendry

“Continuous education via serious programmes attached to the nightly news bulletins seems to be the necessary approach to minimize the panic of ‘AI taking over’. That should be a co-ordinated global information campaign, avoiding unnecessary exaggeration (e.g. Terminator style news). It should explain in simple terms that AI is very difficult to control, that the process of AI improvement and self-improvement cannot be stopped, and that all nations should consider AI development control a top priority which requires a rapid change in the current foreign policy of most nations: nations need to unite to face AI as potentially the biggest existential risks.
Such messages should be delivered very carefully with the intention to support politicians in delivering very unpopular policies, difficult to understand by most people. Such information campaign should explain the need to accept new international agreements. People will have to understand that this is a civilisational crisis of unprecedented magnitude.
However, such messages should be balanced with explaining that if we trust a new planetary organisation with real executive powers then we may not only save humans’ from extinction but rather create a pathway towards unimaginable abundance.” – Tony Czarnecki

“We need to make sure that life is improving for humans as AI takes on greater and greater percentages of jobs. We need to institute:
a) a UBI that gives every person an above-poverty living.
b) a job guarantee that gives every person (if they want it) a chance to earn more wages above the UBI. This guaranteed job should be completely controllable by them to be something that is personally fulfilling and positive for society. It could involve coaching sports, planning and developing neighbourhood improvements, working on safety, … Anything the person can imagine that they can give evidence supports the community.
c) allow humans to enhance themselves (if they want). Enhanced humans might be able to compete with AI on some jobs and earn greater pay than the job guarantee provides. This should also be supported.
d) a continued improvement in the ways humans can find self fulfilment, including: connecting with others, diversions, athletics, service, enhancements, exploration, etc” – Neil

(11) Canary signals?

What ‘canary’ signs can be agreed, in advance, as indicating that an AI is about to move catastrophically beyond the control of humans, so that some drastic interventions are urgently needed?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

Well-designed continuous audits should provide early warnings.

Note: human miners used to carry caged canaries into mines, since the canaries would react more quickly than humans to drops in the air quality.

Selected comments from survey participants

“Top minds unable to decipher AI proposals or intent.” – Iain Beveridge

“When it becomes apparent that ‘self determination’ is a primary goal of an AI system” – Jonathan Clare

“A lot of secrecy amongst the creators of AI systems” – Anon

“AI generated market movements; AI prevention or perpetration of combat atrocities; any incident that replicates the trolley problem in which an AI decision is the key trigger to action and a result (whether positive or negative)” – Jenina Bas

“There should be an international equivalent of a Richter scale, which measures earthquakes to warn about the gravity of incidents when AI escaped human controls.” – Tony Czarnecki

“I’ll let others fill in specific canaries. Mainly, I want to emphasize that, although we should constantly try to determine what the canaries are, we should consider it an ongoing, evolving process. We should not consider that we can come up with every possible canary before hand, nor that because we can’t, the exercise is useless. It is very useful, and might be life saving, and should be pursued, with the humility and understanding that we can’t think of everything and we should iterate over time as new experiences and insights come.” – Neil

“History suggests that any such trigger will come out of the blue and not be particularly predictable – and that the authorities will tend to push such signs down the agenda as inconvenient or because the political capital does not exist to deal with it decisively. A good example of this attitude is the current problems with concrete in public buildings in the UK – plenty of ‘canary’ signs but no action until forced on the State.
If we are looking for canary signs, they will could be vast and multiple but probably attention should be concentrated on
a) in-built biases in AI systems because of the nature of their inputs and perceptual apparatus,
b) system failures that are based on AI being ‘over-logical’ or not being given full information,
c) use of AI in financial fraud (where what is at stake ventually is the collapse of our wider financial system),
d) AI involvement in our nuclear structures and systems, civil and military,
e) use of AI by state surveillance systems or in politically discriminatory ways or by hegemonic forces to manipulate opinion or behaviour,
f) similar use by corporate marketeers, management and internal human resources to control customer, supplier and employee opinion and behaviour.” – Tim Pendry

(12) Preparing contingency measures?

What contingency measures should be prepared in advance, for the circumstance in which drastic intervention becomes necessary to prevent catastrophic damage from misaligned advanced AI?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

Failing to prepare for unpleasant circumstances is tantamount to preparing to fail in these circumstances.

Selected comments from survey participants

“Never allow AI to control their own power supply, and make sure that there do not have means of communicating with other systems at AI lever or below, so we can pull the plug.” – Jenina Bas

“International ‘off switch’ protocol (TBD). Could be partly software / code based, might require more drastic shut-down of communications. All critical devices should have a human override or human ‘backdoor access’ hardwired (e.g. satellites). All critical military infrastructure should have a manual running override / protocol / capability (e.g. submarines).” – Iain Beveridge

“If certain standards of transparency and proof-of-compliance (safety) are not met regarding potentially dangerous technologies, we usually agree that an entity begins to cede its rights to privacy and to operate–and we as a society usually are OK with forceful intervention” – Anon

“Much like the previous answer, this is a very valuable exercise and we should pursue it. However, we should recognize that we won’t think of everything and should plan to keep at it and iterate over and over again as our understanding develops.” – Neil

“Ultimately the ‘pulling the plug’ option might need to be retained but this has two implications – one ethical and one practical. The ethical one might be that, at a certain point, the human race will be committing genocide, or rather speciecide, on a new evolved entity. The practical one is that an entire global economy (and state mechanisms as well) will eventually be built on AI and the close-down of AI will require the costly redundancy of systems that can maintain infrastructures, trade and public order independent of AI to co-exist alongside the AI. In the end, this means an infrastructure of trained front-line workers which operates against the labour cost savings that makes AI so economically attractive. However, an ultimate Plan B may be required at some stage and kept entirely separate from the digital environment.” – Tim Pendry

(13) A watertight backdoor off-switch?

To what extent is it desirable and possible to implement a watertight backdoor “switch-off” mechanism for any advanced AI, that allows humanity to override any potential malfunction in the AI, without also making it possible for hackers to control or distort the operation of that AI?

Answer choices

  • Not desirable: an AI is more likely to do profound good in the world if it is free from any concern that it could be suddenly shut down
  • Not desirable: any backdoor mechanism for interfering with the operation of the AI will make it vulnerable to antagonistic hackers
  • Not necessary: with the right design, an AI will regularly check with humans whether it should be shutting itself down, and will do so whenever requested
  • Not possible: any sufficiently advanced AI will find ways to countermand any backdoor mechanisms
  • Desirable but difficult: it will take enormous design skills
  • Desirable and relatively straightforward: any half-decent AI developer will get this design right
  • Something else (please specify)

Discussion

Discussion points may be added here later.

Selected comments from survey participants

“We’d be wise to develop, and implement, the off-switch right now to prove that it can work even with current systems. This would popularize both the concept (of entities having constraints, for the sake of safety and public responsibility) and the practical use of such a thing — as AI becomes more complex and entities gain more power, how can this possibly be easier to do than now?” – Anon

“Desirable but difficult because some humans will want to have a backdoor to the backdoor kill-switch, and it will take a lot of human cooperation to implement across countries and organisations with competitive interest.” – Jenina Bas

“It will only be possible if we iterate and bring the AI into collaboration with us. As we build good AI, they will want to ‘live’ in a safe, prosperous, advancing world. If we iterate on them and improve them, they will also not want a rogue AI to ruin progress. We need a multitude of AIs, not just one giant one.” – Neil

“Whether it is possible to shut down AI at a certain point is a technical question. I would assume that it is possible for quite some time in the future but that the costs to society of doing so rise exponentially as AI expands throughout the global economy and that measures to allow humanity to do so become prohibitively costly and disruptive the closer we get to the ‘science fiction’ point of an AI that is sentient, conscious and interested in its own survival regardless of the cost. At some stage (probably in the far distance), the AI that can countermand any backdoor mechanism and has the will to do so may well emerge and one of its strategies may be to duplicate itself through an interconnected digital system so that closing it down may be like the game of ‘whack-a-mole’ as the AI multiplies in its own defence.” – Tim Pendry

(14) Assessing recent proposals?

What are the strengths and weaknesses of the proposals for regulating AI  that have been raised at various bodies (including at the US White House and at the UN Security Council) in recent months?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

The Center for AI Safety highlights three policy proposals:

  1. Legal liability for AI harms
  2. Increased regulatory scrutiny
  3. Human oversight over automated systems

Various other documents, such as this one by the UK government, “A pro-innovation approach to AI regulation”, contain lots of fine-sounding words, but fail to provide a tangible, credible roadmap for concrete actions.

See also:

Selected comments from survey participants

“It is encouraging that UN and the US are focusing attention on regulation.” – Jonathan Clare

“Most of them are past the ‘sell-by date’.” – Tony Czarnecki

(15) Beyond regulations to incentives?

In addition to devising and overseeing AI regulations (restrictions), what role should governments play in devising and overseeing AI incentives (ways to encourage development that the free-market, if left to itself, might not choose)?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

Discussion points may be added here later.

Selected comments from survey participants

“I do like the concept of the X-Prize. With all of the promises I’ve heard about what AI will deliver to society, a specific reward for a specific outcome would incentivize many” – Anon

“Creating grants and other funding for curing diseases and aging, clean-energy and excess GHG removal, greater understanding among nations, etc. would certainly be among my list.” – Neil

“Governments’ best hand is to facilitate commercial and non-commercial organisations to work cooperatively. This will enable a certain level (not total) transparency. Even if one organisation wants to keep something secret, contact between people at different levels could mitigate the most extreme dangerous situations. Everybody could end up competing for the most positive human-beneficial AI outcomes if the world is watching.” – Jenina Bas

“There is a significant difference between AI regulation and AI development control, which fundamentally impacts the way we must approach this problem. We need AI regulation and AI development control as part of an overall AI governance. However, it should be recognized that AI regulation and AI development control have different focus and objectives. Controlling AI development is more proactive and concentrates on preventing problems before this new type of intelligence releases itself from our control. AI regulation is more reactive and focused on addressing the problems, which have already occurred or could occur in the near future. (Since AI regulation and AI control have a different impact on society and the future of our coexistence with Superintelligence, they also require different types of organisations to deal with it.)” – Tony Czarnecki

(16) Practical interim steps?

Given that a full system for the global governance of the transition to AI-30 cannot be created in a single step, what are the most sensible interim next steps to take? And what are the best steps that individuals can take?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

Note: some interim steps that initially appear promising could store up greater problems for the medium-term. That’s a potential risk with, for example, a small number of countries trying to “go it alone”.

Selected comments from survey participants

“Get all stakeholders to create their own cooperative convention and organisation.” – Jenina Bas

“Work fast towards alternative models of societal maintenance such as UBI / new taxation models / provide piece of mind to the populace in the face to extraordinary changes coming down the tracks. All individuals can do (apart from engage positively with all the above) is endeavour to educate themselves and to some degree insulate their position for what may well be turbulent times ahead.” – Iain Beveridge

“The most sensible interim measure for people without specified power or influence to take is to communicate the importance of AI and encourage debate about its implications amongst groups in society who may be affected and have no idea yet what is likely to ‘hit them’, preferably without engendering hysteria or panic. Ethical debate inside the corporate community, getting trades unions to educate themselves on the issues, raising the issues within political parties and social movements, educating the media and getting a debate going that forces elites to respond and listen to their own expert opinion are useful baseline starting points – so long as it remains rational and avoids extreme and unnecessarily frightening scenarios which eventually result in a back-lash (as we are now seeing with resistance to Net Zero).” – Tim Pendry

(17) Most troubling question?

Which one question, from the set above, makes you most unsure about how humanity can reach a truly beneficial future?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Selected comments from survey participants

“The answer to lock AI development up in a single international body scares me the most.” – Neil

“Q16 – ‘global governance’ shouts ‘regulation’ and ‘control’. This approach will not work.” – Jenina Bas

“The question as to how much control is desirable with Advanced AI systems” – Jonathan Clare

“Will we implement the agreed solutions on time?” – Tony Czarnecki

“Governmental intervention and action. I am not sure many governments fully appreciate the scale of societal change needed as AI + Automation really starts to hit in the coming years. They think they have decades of the status quo to work it all out; they don’t. We need a full blown revolution in everything from agriculture to taxation & income – or even the near term future starts to look pretty bleak.” – Iain Beveridge

(18) New questions?

Is there an additional question, not present in this survey already, that should be added?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Selected comments from survey participants

“How can a movement for cooperation below government level be galvanised?” – Jenina Bas

“Would controlling AI by selected Transhuman Governors using Brain-Computer Interfaces devices be possible and effective?” – Tony Czarnecki

“Do we have serious problems with respect to the supposedly-factual dataset with which AI is being trained, or the human feedback it receives?” – Anon

“A race towards enhanced AI / superintelligence is unavoidable. Any brakes applied by more ethical actors on actual technological development only open the door for bad ones (collectively agreed controls / protocols aside). Best to get positions in place now. Not agreements, actual Multistakeholder programs. Think an International space station approach.” – Iain Beveridge

Image credits

The image at the top of this page was generated by the Midjourney AI.

The background in the image at the bottom of this page is by Pixabay contributor UlisesEkzMoreno and is used with thanks.

Recent Posts

RAFT 2035 – a new initiative for a new decade

Posted on by
Reply

The need for a better politics is more pressing than ever.

Since its formation, Transpolitica has run a number of different projects aimed at building momentum behind a technoprogressive vision for a better politics. For a new decade, it’s time to take a different approach, to build on previous initiatives.

The planned new vehicle has the name “RAFT 2035”.

RAFT is an acronym:

  • Roadmap (‘R’) – not just a lofty aspiration, but specific steps and interim targets
  • towards Abundance (‘A’) for all – beyond a world of scarcity and conflict
  • enabling Flourishing (‘F’) as never before – with life containing not just possessions, but enriched experiences, creativity, and meaning
  • via Transcendence (‘T’) – since we won’t be able to make progress by staying as we are.

RAFT is also a metaphor. Here’s a copy of the explanation:

When turbulent waters are bearing down fast, it’s very helpful to have a sturdy raft at hand.

The fifteen years from 2020 to 2035 could be the most turbulent of human history. Revolutions are gathering pace in four overlapping fields of technology: nanotech, biotech, infotech, and cognotech, or NBIC for short. In combination, these NBIC revolutions offer enormous new possibilities – enormous opportunities and enormous risks:…

Rapid technological change tends to provoke a turbulent social reaction. Old certainties fade. New winners arrive on the scene, flaunting their power, and upturning previous networks of relationships. Within the general public, a sense of alienation and disruption mingles with a sense of profound possibility. Fear and hope jostle each other. Whilst some social metrics indicate major progress, others indicate major setbacks. The claim “You’ve never had it so good” coexists with the counterclaim “It’s going to be worse than ever”. To add to the bewilderment, there seems to be lots of evidence confirming both views.

The greater the pace of change, the more intense the dislocation. Due to the increased scale, speed, and global nature of the ongoing NBIC revolutions, the disruptions that followed in the wake of previous industrial revolutions – seismic though they were – are likely to be dwarfed in comparison to what lies ahead.

Turbulent times require a space for shelter and reflection, clear navigational vision despite the mists of uncertainty, and a powerful engine for us to pursue our own direction, rather than just being carried along by forces outside our control. In short, turbulent times require a powerful “raft” – a roadmap to a future in which the extraordinary powers latent in NBIC technologies are used to raise humanity to new levels of flourishing, rather than driving us over some dreadful precipice.

The words just quoted come from the opening page of a short book that is envisioned to be published in January 2020. The chapters of this book are reworked versions of the scripts used in the recent “Technoprogressive roadmap” series of videos.

Over the next couple of weeks, all the chapters of this proposed book will be made available for review and comment:

  • As pages on the Transpolitica website, starting here
  • As shared Google documents, starting here, where comments and suggestions are welcome.

RAFT Cover 21

All being well, RAFT 2035 will also become a conference, held sometime around the middle of 2020.

You may note that, in that way that RAFT 2035 is presented to the world,

  • The word “transhumanist” has moved into the background – since that word tends to provoke many hostile reactions
  • The word “technoprogressive” also takes a backseat – since, again, that word has negative connotations in at least some circles.

If you like the basic idea of what’s being proposed, here’s how you can help:

  • Read some of the content that is already available, and provide comments
    • If you notice something that seems mistaken, or difficult to understand
    • If you think there is a gap that should be addressed
    • If you think there’s a better way to express something.

Thanks in anticipation!

  1. A reliability index for politicians? 2 Replies
  2. Technoprogressive Roadmap conf call Leave a reply
  3. Transpolitica and the TPUK Leave a reply
  4. There’s more to democracy than voting Leave a reply
  5. Superdemocracy: issues and opportunities Leave a reply
  6. New complete book awaiting reader reviews Leave a reply
  7. Q4 update: Progress towards “Sustainable superabundance” Leave a reply
  8. Q3 sprint: launch the Abundance Manifesto Leave a reply
  9. Q2 sprint: Political responses to technological unemployment Leave a reply