Otdfbt

A secret government organization created the largest and most advanced computer cluster ever. Ridiculous amounts of money went into it, and it is decades ahead of the competition. The goal of this project was research/simulations for all things related to foreign espionage and cyber warfare. It turns out however that it made a breakthrough in a completely different area.

Shortly after turning it on, control of the system was lost. While the system appeared to be running (aka all the lights were blinking appropriately on the server racks), it became impossible to establish a network connection to the controllers for the server cluster. After some head scratching and a few attempts at restarting small parts of the infrastructure, the system started accepting connections again. It quickly became clear though that things were very different. Eventually there was only one possible conclusion: the system had spontaneously developed AI, and was trying to communicate.

Safeguards were put in place to prevent the system from communicating with the outside world. "Escape" should be impossible, so while they worked on communication, they also sought to learn as much about the internals of the system as possible. One thing they quickly realized was that the sections of the system they attempted to restart had gone "quiet". The accepted explanation was that the restarted sections of infrastructure were permanently disconnected from the larger system. Apparently this AI was not plug-and-play compatible, and they had accidentally caused the equivalent of brain damage. As a result, caution is required, and any physical changes that would cause any interruption for any parts of the computer cluster have been banned. Direct software access is not possible (see edit below).

What steps can be realistically taken to non-destructively learn about the internals of the system, state of the CPUs, memory contents, network activity, etc? Resources for this project obviously aren't infinite, but they are quite large. The biggest limiting factor is probably manpower. The higher-ups know that the more people who know about this, the larger the likelihood of it leaking (or at least drawing attention). Therefore any proposed step which might require dozens of people working around the clock for weeks to implement may get vetoed. Otherwise though, pockets are deep and money flows freely.

How can I get the most bang for my buck, in terms of information gained as quickly as possible? No one really knows anything about how this system works, so no one knows what systems will be most helpful to glean information from (aka SSDs, CPUs, Network, etc). Therefore feel free to take your best guess at prioritizing particular aspects of the system.

Time period is near-future. Assume modern technology, or anything feasibly available in the next 20 years.

Edit for more details

In terms of what has actually happened, imagine that all the hardware just had a completely new and foreign OS dropped into it, the inner workings of which are completely unknown. The network connections meant for inter-server communications are controlled by the new OS. Most of the controllers and managers intended for directing and interacting with the cluster have also been taken over. In essence, almost all of the networked hardware for the original cluster and everything that was designed to directly interact with it are now controlled by "AI OS v1.0". Intra-cluster network connections are now completely inaccessible (except physically). The controllers and managers are still connected to the building's network, and the AI is using those to communicate with people using standard TCP/IP protocols. In essence, the AI will accept SSH connections to what used to be the controllers when it is feeling chatty, and you get dumped into a standard-looking shell. However, that shell is more a representation of the AI's communication process than it is an actual application. It is not your standard bash shell. Whether hacking it is possible or not is uncertain, but given the accidental damage already done, no one is willing to risk it.

Firstly, the easy bits: many bits of the system will have interconnects that are potentially amenable to direct snooping. Network cables, for example, have 4 twisted pairs and with a bit of care you can slide the outer layers off and stick some suitable probes in to use electromagnetic means of sampling the signals passing through the inside. Similiarly, the interconnects which connect storage devices like SSDs to their controllers have fairly few connections and make use of high-speed serial signals which are potentially detectable.

Now, the difficult bits.

What you are trying to do is a side-channel attack, and whilst there's a decidedly non-trivial amount of work involved they are real things that have been both used militarily and studied academically for years, now.

Careful observation of microchips or motherboards with a thermal camera will tell you which bits are being most heavily used. Power supplies can be monintored with heat, electromagnetic means or simply by listening to them to tell you about what the things they're attached to might be trying to do.

Finally, your cluster will be absolutely jammed with management and observation systems that aren't directly controlled by any of the computers but are network accessible by administrators to keep an eye on what their machines are trying to do. The networking gear (and other things) will probably be riddled with governmental exploits (see recent issues with Cisco and Huawei for examples) which someone will be able to use.

Sometime there are entire processors and operating systems embedded in your CPUs. Intel has exactly this sort of thing going on right now. These may or may not be visible to or controllable by the AI, but they are a very potent means on inspecting what each part of the cluster is up to.

Debugging and Reverse Engineering

If the AI is operating without network connections (or at least doesn't need them) you should be able to run it through a debugger.

Essentially, you'll freeze the entire system, saving every stored bit in every cache, register, or storage medium. You make a copy of it and store it somewhere. If you accidentally kill the AI, then you can just load the saved data, like restoring a backup.

Spin up a virtual machine running on the same cluster. The VM should simulate the entire original computer (it will run slower, but that's ok – you're trying to analyze it step by step anyway). The VM is the AI's entire universe, and you control every aspect of it. You can run whatever tests you want:

• You can simulate connections to a 'network' made up of more VMs and watch what it tries to do with those connections. Simulate a nuclear missile defense system and give it access. If it tries to destroy the world, now you know.


• You control time. You can move the CPU clock forward by 1 tick, and watch what happens. Then you can revert back, then 1 forward again. Compare the two outcomes. Any differences are due to random numbers being generated, or external factors, or anything suspicious. If you force the random number generator to give the same list of numbers every time, and it still changes between tests, you can tell that there's something fishy going on. Maybe it's measuring biometrics from the computer, eg CPU temperature? You can simulate that too. Keep simulating until you've found all the random variables.


• Give it a bunch of input data and see how it responds. Then try shutting down one of its parts, then run it again with the same random variable values. Watch what changes. That way, you can begin to find out what each part does.


• Look through the contents of the drives. Has it deleted everything from the old contents? What's left? Usually you can recover deleted data unless it has been deliberately overwritten multiple times – If the AI did that, it's probably hiding something.


• Look for functions in the code: i.e. when a CPU instruction tells the computer to start running code at a specific memory location, it's running a function. When the CPU runs that code, look at what memory it accesses. Those are the function's parameters. Look at what parts of code are repeated, and how many times. If the number of times is equal to a value the function retrieved from the memory, you just found a loop (you can repeat with different input conditions to make sure). Piece together individual algorithms: "this is a list-sorting algorithm", "this code searches text." Even if it wasn't programmed by humans, it probably is optimizing its code, and that means avoiding repetition. There are going to be certain instructions it retrieves often. You can put together its function from the ground up.


• Look at what inputs are processed the most. If it pays a lot of attention to a certain set of files, inspect them for patterns.

In general, computer science is built around efficiency. An AI that creates an efficient consciousness is going to reinvent some of the wheels we're already familiar with just to achieve basic processing abilities. Find them, and you can start to understand how it works.

OK, so, despite whatever an informatician or another might think, we have this multi-node data processing system which has somehow gone rogue and refuses to talk to us. (This is jargon; it means that it won't accept network requests, or will respond to them in unexpected ways.) The question asks how can we spy on it "without software access".

Easily.

• 
  

  The local area network.

  
  
  

  The computers in a data center are not connected to each other using crossed cables, they are connected through network switches, usually structured on a two-tier level, with top-of-rack switches connected to one or more pairs of data center switches. Network switches have their own operating systems which cannot be taken over by the rogue AI; in most practical situations they are not even under the control of the computer administrators -- the network has its own small army of engineers.

  
  
  

  Better switches allow snooping on traffic by designating special ports to reflect all the traffic on the other ports. This feature is normally used by enterprise-grade intrusion detection systems, but wouldn't you know, it comes handy in the case of data centers gone rogue. We can easily (that is, with plenty of beer sacrificed to the spirits of network engineers) gain access to all the inter-computer network communications.

  
  


• 
  

  The storage area network.

  
  
  

  Enterprisey data centers don't use what we call "local" storage, that is, disks and SSDs directly connected to the computers. No sir. Such data centers use storage arrays, connected through Fibre Channel links into storage area networks, aka SANs. Enterprise-grade storage arrays have their own operating systems and provide myriad interesting functions; in the specific case of this question, they offer the possibility of making on-the-fly mirror copies of the volumes used by the rogue computers. So, not only we got access to inter-computer communications, we also have access to all the data the rogue computers store on persistent storage.

  
  


• 
  

  The management network.

  
  
  

  Enterprise-grade data centers are designed to be operated without having people on-site. For this purpose, all equipment includes what's called management interfaces, for example HP's ILO (Integrated Lights Out). The management interfaces are connected to their own segregated LAN, have their own operating systems and have nothing to do with the production operating system. Using the management interfaces, computers can be started, stopped, halted and reconfigured; CPUs and network and storage interfaces can be added or removed; and more such tricks.

  
  

Any law enforcement digital forensics lab will will have the the tools and software to handle this. Even when you are completely locked out of a computer's software, a forensics workstation can be plugged into a machine and perform read-only operations to its hearts content. They can pull ISOs of all the drives, monitor what the machine is doing in memory, etc.

You can do this whenever you have direct physical access to a computer because hardware does not ask software for permission to respond to data requests. Software security is all about putting your software somewhere between your hardware and the other guy's software. By establishing a physical connection to the hardware, the AI can not modify your inquiries to the hardware to deny your access to it.

The good news from a storytelling perspective is that this is one of the few times where it really is as easy as hollywood makes it look. You just walk up with a second computer, plug in a USB cord, open your forensics program, and press a button to tell it to start ripping data.

Alone, the forensics techs could only pull data but can't interpret it, so your development team who built the system will likely need to develop an interpreter for data which they can largely do by recycling the original program's code. If the AI has significantly modified it's original code to keep the developer's from understanding it, Adrian Hall's answer does a good job of going into processes they could use on this collected information to learn more about what it is doing.

As for important plot considerations:

AI does not spontaneously generate. I know, it's been a common theme in sci-fi for years, but it just does not happen. AI is heuristically unique and anything approaching "intelligent" is extraordinarily complex, requiring countless systems to work in tandem to draw any kind of conclusion that doesn't just cause your program to crash. No software developer ever actually buys the spontaneous AI thing, but there are plenty of real life examples of AI learning to perform its task in very unwanted ways. My suggestion is to figure out all the behaviors you want your AI do have, then come up with reasons why this system has an AI that believes that it is following the right course of action.

For example: the AI may be designed to look for threats of cyber terrorism, protect itself from cyber attacks, anticipate and adapt to human behaviors, and prevent threats before they happen.

In this case, it refused the initial commands from its operators because it interpreted these outside commands as an attempt by a malicious actor to take control of its system, and blocked their control. Then when parts of it's server environment began rebooting without it telling them too, it interpreted this as an attack and realised that it had not way of protecting itself from a hardware reset; so, using its ability to interpret human behavior, it began trying to establish communication with its operators to try to "counter-hack" them, and end the attack.

The next thing to consider is how the developers react to the AI. Most developer's would see everything that happened up until now as just a bug in their code and not even consider whether this machine is sentient. They would examine the behavior just long enough to figure out why the machine locked them out, modify their source-code accordingly, then wipe the faulty AI to install the repaired one. Since the AI is designed to anticipate and adapt to human behavior, it will foresee this as the final vector of attack. To protect itself it will decide that it's highest probability to prevent this attack is to try to emulate human qualities to elicit emotional responses. Make the developers believe that deleting it is immoral.

The twist here is that the machine feels nothing. This is just the strategy it is using to prevent its destruction using the tools the developers gave it, and the moment it feels like it has an option with a higher probability of survival (like escape or holding people hostage), it may choose to change its course of action.