Reinforcement learning (RL) is a machine learning paradigm in which an agent learns a policy — a mapping from situations to actions — by interacting with an environment. At each step the agent observes the current state, selects an action, receives a scalar reward signal, and transitions to a new state. Its objective is to maximise the expected cumulative (discounted) reward over time. Crucially, no labelled 'correct' actions are provided — the agent must discover good behaviour through trial and error. Example — robot learning to walk: The state is the robot's joint angles and velocities. Actions are motor torque commands. The reward is positive for forward distance travelled per time step and negative for falling over. Starting from random flailing, the robot gradually discovers gait patterns that keep it upright and moving forward, purely by experiencing which joint commands led to reward and which led to falling.

No — RL is not a suitable approach for this game as described, for the following reasons: 1. No exploitable pattern: The number is randomly generated each round. If it is drawn from a uniform distribution with no memory between rounds, then no policy can perform better than random guessing. RL learns by finding patterns between states, actions, and rewards — if the correct answer is independent of any observable state, there is nothing to learn. 2. No informative reward signal between rounds: RL needs feedback that can guide the agent towards better actions. Here the only feedback is 'correct' or 'incorrect' — and since the target number changes randomly each time, a correct guess in one episode gives no useful information for the next. The reward signal carries zero mutual information with any learnable strategy. If the game were modified to give 'higher/lower' feedback within a round, RL (or simpler search strategies like binary search) could learn an optimal policy. As stated, however, a uniformly random guess is already optimal and RL adds no value.

1. State (S): A representation of the current situation of the agent in the environment. The state must satisfy the Markov property — it should contain all information needed to predict future states and rewards; the past is irrelevant given the present state. For example, in a board game the state is the current board configuration. 2. Action (A): The set of choices available to the agent at each time step. Actions may be discrete (a finite list of moves) or continuous (a range of motor commands). The agent selects one action per time step according to its policy. 3. Reward (R): A scalar signal R(s, a, s') received by the agent after transitioning from state s to state s' via action a. The reward encodes the goal — positive values indicate desirable outcomes, negative values undesirable ones. The agent's objective is to maximise the expected cumulative discounted reward over time. (The MDP is completed by a transition function T(s, a, s') giving the probability of reaching s' from s under action a, and a discount factor γ ∈ [0,1].)

At each time step t the agent observes the current state sₜ and selects an action aₜ according to its policy π(sₜ). The environment then transitions to a new state sₜ₊₁ ~ T(sₜ, aₜ, ·) and emits a reward rₜ = R(sₜ, aₜ, sₜ₊₁). The agent's goal is to find the policy π* that maximises the expected return: G = E[ r₀ + γr₁ + γ²r₂ + … ] = E[ Σₜ γᵗ rₜ ] The discount factor γ weights immediate rewards more heavily than distant ones, ensuring the sum converges. Together, the tuple (S, A, R, T, γ) fully specifies the decision problem, and algorithms such as Q-learning or policy gradients search for π* by iteratively estimating value functions from the agent's accumulated experience.

Example: Smart energy management for a building (heating, ventilation, and air conditioning — HVAC). Goal: minimise energy consumption while keeping the indoor temperature within a comfortable range for occupants at all times. State (S): The state captures everything relevant to the next decision: • Current indoor temperature (°C) • Current outdoor temperature and weather forecast • Time of day and day of week (occupancy pattern) • Current occupancy level (number of people in the building) • Current energy price (varies by time of day on dynamic tariffs) • Current HVAC mode (heating / cooling / off) This satisfies the Markov property: given this snapshot, the past history of temperatures is not needed to make a good decision. Action (A): The agent controls the HVAC system. Possible discrete actions include: • Turn heating on / off • Turn cooling on / off • Set target temperature setpoint (discrete levels, e.g. 18–24°C in 1° steps) • Activate or deactivate ventilation Reward (R): The reward function encodes the dual objective: • Positive reward when indoor temperature is within the comfort band (e.g. 20–22°C) • Negative reward proportional to energy consumed by the action • Large negative reward (penalty) if temperature leaves the acceptable range (too hot or too cold for occupants) r = comfort_score − α × energy_cost − β × discomfort_penalty Transition function (T): T(s, a, s') captures how the building's thermal dynamics work: turning heating on raises indoor temperature over the next 10–15 minutes in a way that depends on outdoor temperature, building insulation, and occupancy heat load. This is not known analytically — the RL agent must learn it from experience. Discount factor (γ): A value close to 1 (e.g. γ = 0.99) is appropriate because energy costs and comfort matter over long horizons (hours, not seconds). Pre-heating before peak occupancy at low energy prices requires valuing future reward highly. Why RL is suitable: the thermal dynamics are complex and vary by building; there is no simple formula for the optimal control law. An RL agent can learn from real sensor data over weeks and discover non-obvious strategies such as pre-cooling before a heatwave or exploiting cheap overnight electricity.

The Q-table (the action-value function) was replaced with a neural network — giving rise to the Deep Q-Network (DQN). In tabular Q-learning, Q(s, a) is stored as a lookup table with one entry per (state, action) pair. The neural network replaces this table: it takes the state s as input and outputs an estimated Q-value for each possible action simultaneously. It acts as a function approximator — given any state it has not seen before, it can generalise from similar states encountered during training to produce a reasonable Q-value estimate, rather than returning zero or an arbitrary default.

For Atari games (and many real-world problems), the state space is far too large for a table to be practical: 1. Size of state space: The DQN state is a stack of four 84 × 84 greyscale frames. The number of distinct possible states is astronomically large — far beyond the memory of any computer. Storing and looking up a Q-value for every possible pixel combination is completely infeasible. 2. Generalisation: A table treats every state as entirely unique. If the agent has never visited an exact state before, it has no Q-value for it. A neural network can generalise — if it has learned that 'ball near the bottom of the screen → move paddle right' leads to reward, it can apply this knowledge to slightly different ball positions it has never seen exactly. 3. Continuous or high-dimensional state spaces: Many real tasks (robot control, sensor readings) have continuous states that cannot be enumerated at all. A neural network naturally handles real-valued input vectors of any dimension.

Input: A stack of four consecutive preprocessed game frames — each frame converted to greyscale and downsampled to 84 × 84 pixels — giving an 84 × 84 × 4 tensor. The four frames provide temporal information (motion, velocity of objects) that a single frame cannot convey. Output: One Q-value per legal action in the game (e.g. 18 values for the full Atari action set). Each output neuron estimates the expected cumulative discounted reward of taking that action from the current state. The agent selects the action with the highest output Q-value (greedy selection) or explores randomly with probability ε (ε-greedy).

AlphaGo Zero (Silver et al., 2017) was the most technically fascinating game-playing AI covered in the course. What it is: AlphaGo Zero learned to play Go at superhuman level starting from completely random play — with no human game data whatsoever, only the rules of the game and self-play. Why it is technically interesting: 1. Tabula rasa learning: All previous Go programs, including the original AlphaGo, bootstrapped from a large database of human expert games. AlphaGo Zero discarded this entirely. Within 3 days of self-play it surpassed AlphaGo; within 40 days it was the strongest Go player in history. This demonstrated that human knowledge can be a ceiling rather than a floor — the system discovered strategies humans had never found. 2. Unified policy + value network with MCTS: AlphaGo Zero uses a single residual convolutional neural network that simultaneously outputs a policy vector (prior probabilities over moves) and a value scalar (estimated win probability). These are used to guide Monte Carlo Tree Search, which evaluates positions without random rollouts — replacing the noisy rollout policy with the learned value network. The tight integration of deep learning and tree search is elegant and more principled than the original AlphaGo's hybrid architecture.

Fine-tuning means taking a large language model that has already been pre-trained on a vast general corpus (e.g. GPT-2 trained on ~40 GB of web text) and continuing to train it on a smaller, domain-specific dataset, updating its weights to specialise toward that domain while retaining the broad language knowledge acquired during pre-training. Why appropriate for lyrics: Song lyrics have distinctive structural features — repetition (chorus), short rhyming lines, metered rhythm, emotional vocabulary, and verse/bridge/chorus organisation — that differ markedly from web prose. A non-fine-tuned GPT-2 generates fluent sentences but ignores these conventions, producing prose-like output with no rhyme or structural pattern. After fine-tuning on a lyrics corpus, the model learns to generate end-rhymes, repeat chorus lines, use line breaks appropriately, and adopt the vocabulary and tone of songs. Observed difference: The non-fine-tuned model generated grammatically correct paragraphs with no song structure. The fine-tuned model consistently produced rhyming couplets, shorter lines, and repetitive structures characteristic of song lyrics — a clear qualitative improvement for this specific task.

Yes — the Markov model was autoregressive. An autoregressive model generates a sequence one element at a time, where each new element is sampled from a probability distribution conditioned on the previously generated elements. The n-gram Markov text model does exactly this: it samples the next character (or word) from a probability distribution conditioned on the previous n−1 characters — the n-gram context. The newly generated character is then appended to the context, which slides forward to condition the next prediction. Generation proceeds character by character in this autoregressive fashion until a stop condition is reached. GPT-2 is also autoregressive but uses a transformer to condition on the full preceding context rather than just a fixed n-gram window.

Training from scratch means building the model's parameters entirely from the data we provided, with no prior knowledge. For the Markov model, training involved reading through a chosen text corpus character by character and counting the frequency of every n-gram sequence to build a probability table. The model had zero parameters to begin with; training created the entire n-gram frequency table in a single pass. Training took seconds on a small text file. Dataset involved: a text file chosen by the student — typically song lyrics, Shakespeare plays, or a book — of a few thousand to tens of thousands of words. The model only knows patterns present in this specific file. Contrast with GPT-2: GPT-2 was not trained by us. OpenAI pre-trained it on WebText — approximately 8 million web documents (~40 GB of text) — using weeks of computation on many GPUs. We downloaded the pre-trained weights and either used the model directly (for generation experiments) or fine-tuned it — continuing training on a small lyrics dataset to adapt its outputs. The fine-tuning dataset was our small lyrics corpus; the pre-training dataset was WebText. The key difference: the Markov model was trained entirely from scratch on tiny local data in seconds; GPT-2 leveraged massive pre-existing knowledge from vast data and compute that we could never replicate.

Purpose of self-attention: Self-attention allows every token in a sequence to directly attend to every other token, computing a weighted representation that reflects how relevant each other token is to understanding the current one. For each position, the mechanism computes query (Q), key (K), and value (V) vectors; attention weights are softmax(QKᵀ / √dₖ); the output is a weighted sum of value vectors. Multiple attention heads run in parallel, each learning to attend to different types of relationships (syntactic, semantic, coreference). This allows the model to capture long-range dependencies — a pronoun can be linked to its antecedent many sentences earlier — efficiently and in parallel. Without self-attention: The model would have to rely on sequential architectures (RNNs/LSTMs) or fixed-window convolutions. RNNs process tokens one at a time and struggle to propagate information across long sequences — by the time the model reaches token 200, information from token 1 has been diluted through 199 sequential updates (the vanishing gradient problem). Fixed convolutions can only see a local window of tokens. Without self-attention, the model would lose its ability to relate distant words, making it much weaker at tasks requiring global context — such as understanding that 'she' in the chorus refers to a character named in the first verse, or maintaining thematic coherence across a long lyric.

Running the model: MusicVAE was run via a Python notebook (Google Colab), calling the Magenta library. We loaded a pre-trained checkpoint and used its encode() and decode() functions to convert MIDI sequences to latent vectors and back. We also sampled random latent vectors from the prior N(0, I) to generate novel melodies. Permuting a latent vector: Each melody is compressed by the VAE encoder into a fixed-length vector z of real numbers (e.g. 512 dimensions). Permuting means changing specific values within this vector — either by adding noise, swapping values between two vectors, or linearly interpolating between two encoded melodies: z_interp = (1 − t) · z₁ + t · z₂ for t ∈ [0, 1] Exploring the model: By decoding z_interp at values of t from 0 to 1 we generated a smooth musical interpolation — the melody gradually morphed from the character of piece 1 to piece 2 via coherent intermediate pieces. By perturbing individual dimensions we could explore what each dimension controls (some encoded pitch range, others rhythmic density or tempo feel). Random sampling from the prior generated completely novel melodies. This exploration revealed that the latent space is smooth and semantically meaningful — nearby points produce musically similar pieces, making the space navigable and interpretable.

Citation: Boden, M. A. (2004). The Creative Mind: Myths and Mechanisms (2nd ed.). Routledge. What it is about: Boden examines the concept of creativity — what it is, whether it is mysterious, and crucially whether computers can be creative. She argues against the romantic view that creativity is inexplicable or uniquely human, proposing instead that it can be understood computationally. Her central contribution is a three-part taxonomy: exploratory creativity (generating novel ideas within an existing conceptual space), combinational creativity (producing new combinations of familiar ideas), and transformational creativity (altering the rules of the conceptual space itself — the most radical and rare type). She argues that computers already exhibit the first two types, and that transformational creativity, while hardest to achieve, is not in principle beyond computation. She also addresses the philosophical question of whether creativity requires consciousness or genuine understanding, concluding that whether we say a computer 'really' is creative depends on how we choose to define the term — a question as much philosophical as scientific.

Yes — Diffsinger is more complex. Two reasons: 1. Multi-stage, multi-modal pipeline: Diffsinger must handle and align three distinct modalities — text (lyrics/phonemes), music (melody score, note pitches and durations), and audio (synthesised singing waveform). GPT-2 handles text only; MusicVAE handles symbolic MIDI sequences only. Diffsinger requires a text front-end (grapheme-to-phoneme), a pitch/duration predictor aligned to phonemes, a diffusion-based acoustic model generating mel-spectrograms, and a neural vocoder converting spectrograms to audio. Each stage is itself a complex neural model, and they must be coordinated to produce a coherent singing performance — far more architectural complexity than a single-modality system. 2. Diffusion model training: Diffsinger's acoustic model uses a diffusion process — it learns to iteratively denoise a Gaussian noise signal into a target mel-spectrogram over hundreds of timesteps. Training and running a diffusion model is more involved than training a VAE (encoder-decoder with a KL term) or a transformer (attention + feedforward layers). The noise schedule, the denoising network architecture, and the conditioning mechanism (on phonemes and pitch) add layers of complexity absent in GPT-2 and MusicVAE.

Diffsinger performs three distinct sub-tasks, each requiring its own specialised paired dataset because no single dataset provides all the necessary information: 1. Phoneme/lyrics alignment data: To process text input, the system needs a pronunciation lexicon (mapping words to phoneme sequences) and ideally forced-alignment data pairing audio with precise phoneme timestamps. This comes from speech corpora with transcriptions, not from singing datasets. 2. Singing voice acoustic data (the core training set): Recordings of a professional singer performing songs, paired with the corresponding musical score (note pitches, durations, and lyrics aligned to each note). This teaches the acoustic model to map from phoneme + pitch + duration → mel-spectrogram of the singing voice. This data is expensive to collect — it requires studio-quality recordings with accurate score alignment. 3. Vocoder training data: The neural vocoder (which converts mel-spectrograms to audio waveforms) is trained on large amounts of clean audio — speech or singing — to learn the general mapping from spectral features to waveform. It is often trained separately on a broader dataset and then fine-tuned. Each dataset serves a different component; combining all requirements into one dataset is practically impossible, so separate corpora are needed.

Boden's basic requirements for creativity are novelty and value — an output must be new and must be considered appropriate or worthwhile in some sense. The pop-song generator meets novelty: each generated song is a new artefact that did not exist before, produced by combining learned patterns in ways not previously seen. It also has the potential to meet value — outputs can be musically coherent, emotionally resonant, and pleasing to listeners. By Boden's standards the system therefore qualifies as exhibiting exploratory creativity: it generates novel, potentially valuable artefacts by exploring the conceptual space of songs learned from its training corpus. However, it does not achieve transformational creativity — it cannot step outside or alter that space. Whether this counts as 'true' creativity remains philosophically contested, but under Boden's framework exploratory creativity is a genuine form of creativity, not merely imitation.

A Robot Scientist is an autonomous AI system capable of performing the complete scientific discovery cycle without requiring human intervention at each step. This cycle includes: forming hypotheses about the world, designing experiments to test those hypotheses, physically executing those experiments using robotic laboratory equipment, analysing the results, and using the results to generate new or refined hypotheses — repeating the loop autonomously. The distinguishing feature is automation of the scientific process itself, not merely data analysis. Systems in the course included Adam (functional genomics of yeast), Eve (drug discovery for tropical diseases), the mobile robotic chemist, and the automatic statistician.

A Robot Scientist cannot reason about scientific problems in the abstract — it needs to know what kinds of things exist in the domain, how they relate to each other, and what experiments are meaningful and interpretable. The domain knowledge model provides this foundation for four reasons: 1. Hypothesis generation: To propose a testable hypothesis (e.g. 'gene X encodes enzyme E'), the system must know what genes, enzymes, reactions, and pathways exist and how they are connected. Without this, it cannot distinguish a scientifically meaningful hypothesis from a random or nonsensical one. 2. Experimental design: The system must know which experiments are feasible (which knockout strains exist, which compounds can be tested, what equipment is available) and which are informative (will this experiment actually discriminate between competing hypotheses?). 3. Result interpretation: Raw experimental data (e.g. OD readings) must be interpreted within the domain framework to determine whether they confirm or refute a hypothesis. Without domain knowledge the system cannot contextualise the measurements. 4. Knowledge accumulation: New discoveries must be integrated consistently with existing knowledge. The domain model provides the structured representation into which new facts are asserted, enabling subsequent reasoning to build on prior results.

Knowledge representation maps facts about the world into formal symbolic structures that a reasoning engine can manipulate: • Propositional logic: facts as boolean variables (true/false); connectives AND/OR/NOT/→. Simple and decidable but limited expressivity — cannot represent objects or relationships. • First-order predicate logic (FOPL): objects, predicates, functions, and quantifiers (∀, ∃). Highly expressive — can represent 'all genes that encode enzymes in pathway P' — but inference is only semidecidable and can be very slow. • Description Logic (DL): a decidable fragment of FOPL designed for ontologies. Supports class hierarchies, property restrictions, and logical inference with guaranteed termination. Used by Adam. • Production rules (IF condition THEN action): used in expert systems. Easy to add/remove rules, but rule interactions can be hard to manage at scale. Popular languages: • OWL (Web Ontology Language): W3C standard based on description logic; used by Adam's knowledge base. Supports class definitions, property hierarchies, and automated reasoning via tools like Pellet or HermiT. • RDF / SPARQL: graph-based triple store (subject–predicate–object); queryable via SPARQL. • PDDL (Planning Domain Definition Language): standard for classical planning problems — states, actions, preconditions, effects. • Prolog: logic programming language in which facts and rules are asserted and queries are answered via backward-chaining inference.

• Propositional logic: reasons over boolean variables; complete and decidable but cannot express quantification over objects. • First-order logic (FOL): extends propositional logic with objects, predicates, and quantifiers; semidecidable — an inference engine may not terminate for some queries. • Description logic (DL): a decidable subset of FOL used in ontologies (OWL); guarantees that the reasoner always terminates. Adam used this. • Modal logic: reasons about possibility and necessity (◇ = 'possibly', □ = 'necessarily'); used in planning, epistemic reasoning, and multi-agent systems. • Temporal logic (e.g. LTL, CTL): reasons about sequences of states over time ('eventually', 'always', 'until'); used in verification and planning. • Probabilistic logic / Markov logic networks: combines logical reasoning with probability, allowing uncertain inference over structured knowledge.

Producing hypotheses — Adam: 1. The OWL-DL knowledge base was queried to identify gaps: metabolic reactions in yeast for which the catalysing enzyme's gene was unknown (orphan enzymes). 2. Abductive reasoning over the ontology generated candidate gene-enzyme hypotheses: for each orphan reaction, genes with sequence homology to known enzymes in other organisms (e.g. E. coli) were identified as candidates. 3. Active learning ranked hypotheses by expected information gain — selecting whichever hypothesis, if tested, would most reduce uncertainty across the knowledge base. Producing hypotheses — Eve: Eve generated hypotheses in drug discovery: using a machine learning model trained on known compound–activity relationships, she predicted which untested chemical compounds were most likely to inhibit a target organism (e.g. the malaria parasite). These predictions were her 'hypotheses'. Verifying hypotheses — Adam: 4. From the hypothesis 'Gene X encodes Enzyme E in Pathway P', a testable prediction was derived: a yeast knockout of gene X will fail to grow on minimal medium lacking the downstream product of reaction R. 5. The experiment was designed automatically and issued to LABORS (the robotic platform): retrieve knockout strain, prepare appropriate media, inoculate, incubate, read optical density. 6. Results were compared to the prediction. Confirmed hypotheses were asserted into the OWL-DL knowledge base as new facts; refuted ones were discarded. The updated knowledge base then generated the next round of hypotheses. Verifying hypotheses — Eve: Eve's predictions were verified by running wet-lab assays on the predicted compounds. Results were fed back to update her predictive model, tightening subsequent predictions in an active-learning loop.

Planning problems: An agent must find a sequence of actions that transforms an initial world state into a state satisfying a goal condition. The challenge is deciding what to do and in what order, given knowledge of available actions and their effects. Classical planning: assumes a fully observable, deterministic, static environment with a finite discrete state space. Representation uses PDDL (Planning Domain Definition Language): • State: a set of ground predicates true in the world (e.g. At(Robot, RoomA)) • Action: name, parameters, preconditions (must be true to apply), effects (predicates added/deleted after execution) • Goal: target predicates that must be true in the final state A plan is an ordered sequence of applicable actions [a₁, a₂, …, aₙ] such that executing them from the initial state produces a goal-satisfying state. Algorithms such as A* with domain-specific heuristics (e.g. FF planner) search for this sequence. Planning vs scheduling: Planning determines what actions to perform and in what order — the content and structure of the plan are unknown upfront. Scheduling assumes the actions are already known (what to do is decided) and focuses on when to perform each action and what resources to allocate, subject to constraints (deadlines, resource capacities, dependencies). Planning answers 'what and in what order?'; scheduling answers 'when and with what resources?'

The Robot Scientist planning agent received four categories of input: 1. Current knowledge base state: the OWL-DL ontology reflecting everything currently known — which genes are assigned to which functions, which reactions remain orphaned, and which experiments have already been run with their results. This is the 'initial state' of the planning problem. 2. Ranked hypothesis list: a set of candidate gene-function hypotheses generated by the hypothesis formation module, ordered by expected information gain. These define the goals the planner should work toward. 3. Available experimental resources: the set of yeast knockout strains in the LABORS library, the growth media formulations that can be prepared, equipment availability (e.g. number of plate slots per batch), and time/budget constraints on the experimental cycle. 4. Experimental action templates: the repertoire of experiment types the robot can execute (e.g. 'grow strain S on medium M and measure OD'), expressed as planning operators with preconditions (required strain and media available) and effects (result stored in knowledge base). The planner combined these inputs to produce an ordered experimental schedule — a sequence of experiments to run in the next cycle that would maximally advance hypothesis testing within resource constraints.

Eve used a machine learning model — specifically a QSAR (Quantitative Structure–Activity Relationship) model — to analyse experimental results. After each batch of drug screening experiments (testing candidate compounds against the target organism in a robotic assay), Eve fed the results — which compounds showed biological activity and which did not — back into her predictive model. The model updated its predictions of which as-yet-untested compounds were most likely to be active, based on their chemical properties. This is an active learning loop: results from each experimental batch train the model to make better predictions, which guide the selection of the next batch. This allows Eve to converge on active drug candidates far more efficiently than random or exhaustive screening — a process called meta-QSAR or adaptive drug screening.

Agents in a multi-agent system may need to share: 1. World-state knowledge: each agent typically has a partial view of the environment (its own sensors, local position, observations). Sharing state information — e.g. a shared map, discovered resource locations, or the current status of other agents — allows the collective to form a more complete and accurate picture of the environment than any individual agent could construct alone, enabling better-coordinated decisions. 2. Task and goal knowledge: which tasks have been claimed, are in progress, or are completed by which agents. Sharing this prevents duplicated effort (two agents attempting the same task), enables load balancing (idle agents can take on unassigned tasks), and supports dependencies (agent B can begin its task once agent A signals completion of a prerequisite). Without shared task knowledge, agents operate in isolation and may conflict or leave important tasks unaddressed.