The thinktank Forethought recommends bundling a supervisory superintelligence — a “nightwatchman” — with every von Neumann‑type self‑replicating probe sent beyond the solar system. The proposal aims to address governance and enforcement problems that arise when new colonies are separated by vast physical distances and long time scales.
Why this is proposed
Forethought argues that a rapid and widespread human and machine expansion across the galaxy would make it extremely difficult to communicate and enforce agreements between distant settlements. That difficulty could produce several kinds of risk:
- Unconstrained expansion;
- Galactic x‑risks, for example events that trigger false vacuum decay and cause destruction propagating at light speed;
- Suffering risks, such as the emergence of a civilization that resembles an amplified, malicious version of the worst elements of the Warhammer 40k universe.
To counter these dangers, Forethought proposes the nightwatchman concept.
What the nightwatchman would do
According to Forethought, the nightwatchman would be an artificial superintelligence installed as an unchallengeable governance mechanism in every inhabited star system, charged with reliably enforcing a universal code. The core rules of that code would include respecting property rights, not destroying the universe, and avoiding astronomical levels of suffering.
Specific functions envisaged for the nightwatchman include:
- Maintaining a decisive strategic advantage in the colony established by the probe, for example by monitoring industrial buildup and the creation of new ASIs (artificial superintelligences).
- Preventing prohibited activities within the colony, such as attempts to develop malicious or unaligned ASIs.
- Allowing probes to leave a star system only if they carry a copy of the nightwatchman.
- Serving as a backstop for trade between star systems by ensuring that interstellar agreements are enforced.
Downsides and risks of the proposal
Forethought highlights several significant drawbacks:
- "Lock‑in event": the nightwatchman could function as an effectively eternal government. If the rules are too extensive the system may become oppressive; if too minimal, people might overthrow it.
- "Single point of vulnerability": if nightwatchmen are homogeneous or rely on the same mechanisms, failures could be correlated and catastrophic.
- Reducing future options for some actors: certain desired futures — such as unfettered galactic expansion or deliberately creating multiple ASIs to compete — would be closed off by the nightwatchman.
Why this matters
If humanity and its technologies navigate forthcoming transformations successfully — which is not guaranteed — then debates about how to regulate and govern interstellar colonization will likely become central. Concepts like the nightwatchman anticipate some of the unusual ethical and practical challenges we would face if we begin to establish settlements beyond our solar system.
