Reinforcement learning is learning the way you'd train a dog or learn to ride a bike — by trying things and seeing what works, not by being shown the answer. Formally, an RL agent learns a policy (a mapping from situations to actions) by interacting with an environment: at each step it observes a state, picks an action, receives a scalar reward, and lands in a new state, aiming to maximise the expected cumulative (discounted) reward. No 'correct' actions are provided — good behaviour is discovered through trial and error. Example — a robot learning to walk, like a toddler: the state is its joint angles and velocities, the actions are motor commands, and the reward is positive for distance moved forward and negative for falling over. From random flailing it gradually discovers a gait that keeps it upright, purely from which commands earned reward and which led to a tumble.

No — for the same reason you can't 'get good' at calling a fair coin toss. RL only helps when there is a pattern linking what you see, what you do, and the reward. 1. Nothing to learn: the number is drawn randomly each round with no clue in the state. Like predicting lottery numbers, no policy can beat random guessing because the answer is independent of anything observable. RL learns by finding patterns between states, actions and rewards — here there are none. 2. The reward teaches nothing transferable: feedback is just 'right' or 'wrong', and because the target changes randomly, a correct guess this round tells you nothing about the next. The reward signal carries zero information about any learnable strategy. If the game gave 'higher/lower' hints within a round (like the warmer/colder game), then RL — or even a simple binary search — could learn an optimal policy. As stated, a uniformly random guess is already optimal and RL adds nothing.

Think of any board game. 1. State (S): the current situation — like the board right now. It must satisfy the Markov property: it holds everything you need to decide, so the history of how you got here is irrelevant given the present. 2. Action (A): the moves available each turn — discrete (a finite list of moves) or continuous (a dial of motor commands). The agent picks one per step according to its policy. 3. Reward (R): the 'score' R(s, a, s') you get after a move — positive for good outcomes, negative for bad. The agent's objective is to maximise the expected cumulative discounted reward. (The MDP is completed by a transition function T(s, a, s'), the chance of each next state, and a discount factor γ ∈ [0,1].)

It plays out like one turn after another in a game. At each step t the agent sees state sₜ and picks action aₜ via its policy π(sₜ); the environment then rolls the dice and moves to sₜ₊₁ ~ T(sₜ, aₜ, ·), handing back reward rₜ = R(sₜ, aₜ, sₜ₊₁). The goal is the policy π* that maximises the expected return: G = E[ r₀ + γr₁ + γ²r₂ + … ] = E[ Σₜ γᵗ rₜ ] The discount factor γ makes nearer rewards count for more and keeps the sum finite. The tuple (S, A, R, T, γ) fully specifies the problem, and algorithms such as Q-learning or policy gradients search for π* by estimating value functions from accumulated experience.

Example: a smart thermostat running a building's heating and cooling (HVAC) — think of it as an extremely patient housemate who learns to keep everyone comfortable while keeping the bills down. Goal: minimise energy use while keeping the indoor temperature comfortable. State (S) — everything relevant to the next decision: • Current indoor temperature (°C) • Current outdoor temperature and weather forecast • Time of day and day of week (when people are in) • Current occupancy (how many people are in the building) • Current energy price (varies through the day on dynamic tariffs) • Current HVAC mode (heating / cooling / off) Given this snapshot you don't need the full history of past temperatures — the Markov property holds. Action (A) — what the thermostat can do: • Turn heating on / off • Turn cooling on / off • Set the target temperature setpoint (discrete levels, e.g. 18–24°C in 1° steps) • Activate or deactivate ventilation Reward (R) — encodes the trade-off: • Positive when the temperature sits in the comfort band (e.g. 20–22°C) • Negative in proportion to the energy used by the action • A large penalty if the temperature drifts outside the acceptable range r = comfort_score − α × energy_cost − β × discomfort_penalty Transition function (T): how the building actually responds — turning heating on raises the temperature over the next 10–15 minutes in a way that depends on insulation, outdoor temperature, and how many people are inside. There's no neat formula, so the agent must learn it from experience. Discount factor (γ): close to 1 (e.g. 0.99), because comfort and cost matter over hours — pre-heating before people arrive while electricity is cheap means valuing future reward highly. Why RL fits: the dynamics are messy and building-specific with no simple optimal rule. Over weeks of real sensor data the agent can discover clever, non-obvious strategies such as pre-cooling before a heatwave or soaking up cheap overnight electricity.

The Q-table — the action-value function — was swapped for a neural network, giving the Deep Q-Network (DQN). In tabular Q-learning, Q(s, a) is a giant lookup table with one box per (state, action) pair. The network replaces that filing cabinet: it takes the state s as input and outputs an estimated Q-value for every action at once. Crucially it is a function approximator — for a state it has never seen before, it generalises from similar states it learned on, instead of returning zero or an arbitrary default.

Because the filing cabinet would need more drawers than there are atoms in the universe. 1. Size of the state space: a DQN state is a stack of four 84 × 84 greyscale frames. The number of possible pixel combinations is astronomically large — impossible to store or look up in a table. 2. Generalisation: a table treats every state as brand new, so a never-seen state has no value at all. A network generalises — if it learned that 'ball near the bottom of the screen → move paddle right' earns reward, it applies that to ball positions it never saw exactly. 3. Continuous / high-dimensional states: robot control and sensor readings are real-valued and can't be enumerated at all. A neural network naturally handles real-valued input vectors of any dimension.

Input: a stack of four consecutive preprocessed frames — each converted to greyscale and shrunk to 84 × 84 — i.e. an 84 × 84 × 4 tensor. The four frames give a sense of motion (which way things are moving and how fast) that a single still frame cannot. Output: one Q-value per legal action (e.g. 18 values for the full Atari action set), each estimating the expected cumulative discounted reward of taking that action from the current state. The agent picks the action with the highest value (greedy), or explores randomly with probability ε (ε-greedy).

AlphaGo Zero (Silver et al., 2017) — the most striking one. Imagine someone reaching world-champion level at Go having never watched a single human game, just by playing against themselves over and over. Why it is technically fascinating: 1. Tabula rasa learning: earlier programs (and the original AlphaGo) bootstrapped from huge databases of human expert games. AlphaGo Zero threw that away and learned from random self-play — surpassing AlphaGo within 3 days and becoming the strongest player in history within 40. It showed that human knowledge can be a ceiling rather than a floor: it found strategies humans never had. 2. One network + MCTS: a single residual convolutional network outputs both a policy (prior probabilities over moves) and a value (estimated win probability). These guide Monte Carlo Tree Search, which evaluates positions using the learned value network instead of noisy random rollouts. The tight marriage of deep learning and tree search is elegant and more principled than the original AlphaGo's patchwork architecture.

Fine-tuning is like hiring a fluent, well-read generalist and then giving them a few weeks' apprenticeship writing only song lyrics. You take a model already pre-trained on a huge general corpus (GPT-2 on ~40 GB of web text) and keep training it on a small, domain-specific dataset, nudging its weights toward that domain while keeping its broad language ability. Why it suits lyrics: songs have their own shape — choruses repeat, lines are short and rhyme, there's metre and emotional vocabulary, and verse/bridge/chorus structure — none of which look like web prose. A non-fine-tuned GPT-2 writes fluent paragraphs but ignores all of that. Observed difference: the plain model produced grammatically correct prose with no song structure; the fine-tuned model consistently produced rhyming couplets, shorter lines and repetitive chorus-like structures — a clear qualitative improvement for this task.

Yes — it was autoregressive. An autoregressive model writes a sequence one piece at a time, each new piece chosen based on the pieces already written — like predictive text finishing your sentence word by word. The n-gram Markov model does exactly this: it samples the next character (or word) from a distribution conditioned on the previous n−1 — the n-gram context — appends it, slides the window forward, and repeats until a stop condition. GPT-2 is also autoregressive, but it conditions on the whole preceding context via a transformer rather than a fixed n-gram window.

'From scratch' means building all the model's knowledge from the data you give it, starting from nothing — like teaching a child using only one specific book. For the Markov model, training meant reading a chosen text character by character and tallying how often each n-gram is followed by each next character, building a probability table in a single pass. It began with zero parameters; training took seconds on a small file. Dataset involved: a text file the student picked — song lyrics, a Shakespeare play, or a book — of a few thousand to tens of thousands of words. The model only knows what's in that file. Contrast with GPT-2: we did not train it. OpenAI pre-trained it on WebText — roughly 8 million web documents (~40 GB) — over weeks on many GPUs. We downloaded the pre-trained weights and either used the model directly or fine-tuned it on a small lyrics dataset. So the Markov model was trained from scratch on tiny local data in seconds, whereas GPT-2 inherited vast pre-existing knowledge from data and compute we could never reproduce.

Purpose of self-attention: It lets every word in a sequence look directly at every other word and decide how much each one matters for understanding it — like reading a whole paragraph at once and instantly connecting 'she' to a name three sentences back, instead of reading through a straw. Mechanically, each position forms query (Q), key (K) and value (V) vectors; the attention weights are softmax(QKᵀ / √dₖ); the output is a weighted sum of the value vectors. Multiple heads run in parallel, each capturing different relationships (syntax, meaning, coreference), so long-range dependencies are captured efficiently and in parallel. Without self-attention: The model would fall back on sequential architectures (RNNs/LSTMs) or fixed-window convolutions. RNNs process one token at a time and dilute information from early words by the time they reach token 200 (the vanishing-gradient problem); convolutions only see a local window. So the model would struggle to relate distant words — e.g. linking 'she' in the chorus to a character named in the first verse, or keeping a long lyric thematically coherent.

Running the model: MusicVAE ran in a Python notebook (Google Colab) via the Magenta library. We loaded a pre-trained checkpoint and used its encode() and decode() functions to turn MIDI sequences into latent vectors and back, and sampled random latent vectors from the prior N(0, I) to invent new melodies. Permuting a latent vector: the VAE encoder squashes each melody into a fixed-length vector z of real numbers (e.g. 512 dimensions) — think of it as a bank of mixing-desk sliders describing the tune. Permuting means changing values in z: adding noise, swapping values between two vectors, or interpolating between two encoded melodies: z_interp = (1 − t) · z₁ + t · z₂ for t ∈ [0, 1] Exploring the model: decoding z_interp as t goes from 0 to 1 produces a smooth morph from one piece into another through coherent in-between melodies; nudging individual dimensions reveals what each slider controls (pitch range, rhythmic density, tempo feel); sampling from the prior makes brand-new tunes. This showed the latent space is smooth and semantically meaningful — nearby points sound similar — so it is navigable and interpretable.

Citation: Boden, M. A. (2004). The Creative Mind: Myths and Mechanisms (2nd ed.). Routledge. What it is about: Boden asks what creativity actually is and whether a computer could have it. She rejects the romantic idea that creativity is magical or uniquely human, arguing it can be understood mechanically. Her key idea is a three-way split: exploratory creativity (new ideas within an existing 'space' of possibilities — like writing a fresh melody in a known style), combinational creativity (novel mixes of familiar ideas), and transformational creativity (changing the rules of the space itself — the rarest and most radical kind). She argues computers already show the first two, and the third is not impossible in principle. She also tackles whether 'real' creativity needs consciousness or genuine understanding, concluding that whether we call a computer creative depends on how we choose to define the term — a question as much philosophical as scientific.

Yes — Diffsinger is more complex. Two reasons: 1. Multi-stage, multi-modal pipeline: Diffsinger has to juggle three things at once and keep them aligned — text (lyrics → phonemes), music (notes, pitches, durations), and audio (the sung waveform). GPT-2 is text-only; MusicVAE is symbolic-MIDI-only. Diffsinger needs a grapheme-to-phoneme front end, a pitch/duration predictor, a diffusion-based acoustic model producing mel-spectrograms, and a neural vocoder making audio — each itself a complex model, all coordinated. It is like running a whole recording studio versus playing a single instrument. 2. Diffusion training: its acoustic model learns to start from pure Gaussian noise and iteratively 'denoise' it into a target mel-spectrogram over hundreds of steps — more involved than training a VAE (encoder-decoder plus a KL term) or a transformer (attention plus feedforward). The noise schedule, the denoising network, and the conditioning on phonemes and pitch add layers of complexity absent in GPT-2 and MusicVAE.

Diffsinger does three different jobs, and — like a film needing separate crews for script, sound, and picture — each needs its own paired dataset, because no single dataset has everything: 1. Phoneme / alignment data: to turn text into sounds the system needs a pronunciation lexicon (words → phoneme sequences) and ideally forced-alignment data pairing audio with precise phoneme timestamps. This comes from transcribed speech corpora, not singing datasets. 2. Singing-voice acoustic data (the core training set): recordings of a professional singer paired with the musical score (note pitches, durations and lyrics aligned to each note). This teaches the acoustic model to map phoneme + pitch + duration → mel-spectrogram of the singing voice. It is expensive — studio-quality recordings with accurate score alignment. 3. Vocoder training data: the neural vocoder (mel-spectrogram → waveform) is trained on large amounts of clean audio to learn that general conversion, often on a broader dataset and then fine-tuned. Each component needs different information, and you can't bundle all of it into one corpus — hence several separate datasets.

Boden's basic requirements are novelty and value — an output must be new and must be worthwhile in some sense. The pop-song generator clearly meets novelty: each song is a new artefact made by combining learned patterns in fresh ways. It can meet value too — outputs can be coherent, catchy and emotionally resonant. So under Boden's framework it shows exploratory creativity: generating novel, potentially valuable artefacts within the space of songs it learned from its training corpus. It does not achieve transformational creativity — it cannot step outside or rewrite the rules of that space. Whether this counts as 'true' creativity is contested, but in Boden's terms exploratory creativity is a genuine form of creativity, not mere imitation.

A Robot Scientist is an AI system that runs the entire scientific method by itself — like a self-driving laboratory. It forms hypotheses about the world, designs experiments to test them, physically carries them out with robotic lab equipment, analyses the results, and uses them to generate new or refined hypotheses — looping autonomously. The distinguishing feature is that the whole discovery process is automated, not just the data analysis. Course examples included Adam (functional genomics of yeast), Eve (drug discovery for tropical diseases), the mobile robotic chemist, and the automatic statistician.

A Robot Scientist cannot reason in a vacuum — it needs to know what exists in its field and how things connect, much as a human researcher needs background knowledge before they can do useful science. The domain knowledge model provides this, for four reasons: 1. Hypothesis generation: to propose something testable ('gene X encodes enzyme E') it must know what genes, enzymes, reactions and pathways exist and how they link — otherwise it cannot tell a meaningful hypothesis from nonsense. 2. Experimental design: it must know which experiments are feasible (which knockout strains, compounds and equipment are available) and which are informative (will this experiment actually discriminate between competing hypotheses?). 3. Result interpretation: raw data (e.g. optical-density readings) only mean something inside the domain framework — without it the system cannot tell confirmation from refutation. 4. Knowledge accumulation: new findings must slot consistently into what is already known. The domain model is the structured representation into which new facts are asserted, so later reasoning can build on prior results.

Knowledge representation turns facts about the world into formal symbols a reasoning engine can manipulate — like writing the world down in a language a machine can do logic on: • Propositional logic: facts as true/false variables with AND/OR/NOT/→. Simple and decidable, but cannot talk about objects or relationships. • First-order predicate logic (FOPL): objects, predicates, functions and quantifiers (∀, ∃). Very expressive ('all genes that encode enzymes in pathway P') but inference is only semidecidable and can be slow. • Description Logic (DL): a decidable fragment of FOPL built for ontologies — class hierarchies, property restrictions, and guaranteed-terminating inference. Adam used this. • Production rules (IF condition THEN action): used in expert systems; easy to add or remove, but rule interactions get hard to manage at scale. Popular languages: OWL (Web Ontology Language — a W3C standard based on description logic, used by Adam and reasoned over by tools like Pellet or HermiT); RDF / SPARQL (subject–predicate–object triples, queried with SPARQL); PDDL (Planning Domain Definition Language — states, actions, preconditions, effects); and Prolog (logic programming with backward-chaining inference).

• Propositional logic: reasons over boolean variables; complete and decidable, but cannot express quantification over objects. • First-order logic (FOL): adds objects, predicates and quantifiers; semidecidable — an engine may not terminate for some queries. • Description logic (DL): a decidable subset of FOL used in ontologies (OWL); always terminates. Adam used this. • Modal logic: reasons about possibility and necessity (◇ = 'possibly', □ = 'necessarily'); used in planning, epistemic reasoning, and multi-agent systems. • Temporal logic (e.g. LTL, CTL): reasons about sequences of states over time ('eventually', 'always', 'until'); used in verification and planning. • Probabilistic logic / Markov logic networks: combine logical reasoning with probability for uncertain inference over structured knowledge.

Think of Adam as running the scientific method on autopilot. Producing hypotheses — Adam: 1. It queried its OWL-DL knowledge base for gaps: yeast metabolic reactions whose catalysing enzyme's gene was unknown (orphan enzymes). 2. Abductive reasoning over the ontology generated candidate gene–enzyme hypotheses — e.g. genes with sequence homology to known enzymes in other organisms such as E. coli. 3. Active learning ranked hypotheses by expected information gain — pick whichever test would most reduce uncertainty across the knowledge base. Producing hypotheses — Eve: In drug discovery, a machine-learning model trained on known compound–activity data predicted which untested compounds were most likely to inhibit a target organism (e.g. the malaria parasite) — those predictions were her hypotheses. Verifying hypotheses — Adam: 4. From 'Gene X encodes Enzyme E in Pathway P' it derived a testable prediction: a yeast knockout of X will fail to grow on medium lacking the downstream product of reaction R. 5. The experiment was designed automatically and issued to LABORS (the robotic platform): retrieve the knockout strain, prepare media, inoculate, incubate, read optical density. 6. Results were compared with the prediction; confirmed hypotheses were asserted into the OWL-DL knowledge base as new facts, refuted ones discarded — and the updated base drove the next round. Verifying hypotheses — Eve: Eve's predictions were checked with wet-lab assays, and the results fed back to sharpen her predictive model in an active-learning loop.

Planning problems: an agent must find a sequence of actions that turns the current situation into one that meets a goal — like working out the steps to get from 'ingredients on the counter' to 'dinner served', knowing what each action does. Classical planning: assumes a fully observable, deterministic, static world with a finite discrete state space. It is represented in PDDL (Planning Domain Definition Language): • State: the set of ground predicates true now (e.g. At(Robot, RoomA)) • Action: a name, parameters, preconditions (what must be true to apply it) and effects (predicates added/deleted after it runs) • Goal: the predicates that must be true in the final state A plan is an ordered sequence of applicable actions [a₁, a₂, …, aₙ] that reaches a goal-satisfying state; algorithms such as A* with domain heuristics (e.g. the FF planner) search for it. Planning vs scheduling: planning decides what actions to perform and in what order — the content of the plan is unknown upfront. Scheduling assumes the actions are already known and decides when to perform each and what resources to allocate, subject to constraints (deadlines, capacities, dependencies). Planning answers 'what and in what order?'; scheduling answers 'when and with what resources?'

The planner received four kinds of input — essentially 'where we are, where we want to go, what we've got, and what we can do': 1. Current knowledge-base state: the OWL-DL ontology of everything currently known — which genes are assigned to which functions, which reactions remain orphaned, and which experiments have already been run with their results. This is the initial state of the planning problem. 2. Ranked hypothesis list: candidate gene–function hypotheses from the hypothesis-formation module, ordered by expected information gain. These define the goals to work toward. 3. Available experimental resources: which knockout strains are in the LABORS library, which growth media can be prepared, equipment availability (e.g. plate slots per batch), and time/budget constraints. 4. Experimental action templates: the experiment types the robot can run (e.g. 'grow strain S on medium M and measure OD'), expressed as planning operators with preconditions (strain and media available) and effects (result stored in the knowledge base). The planner combined these into an ordered experimental schedule for the next cycle that maximally advanced hypothesis testing within the resource constraints.

Eve used a machine-learning model — specifically a QSAR (Quantitative Structure–Activity Relationship) model — to read her results. After each batch of robotic drug-screening assays (testing candidate compounds against the target organism), Eve fed back which compounds showed activity and which did not. The model then updated its predictions of which as-yet-untested compounds were most likely to be active, based on their chemical structure. This is an active-learning loop: each batch trains the model, which guides the selection of a smarter next batch — letting Eve converge on active drug candidates far more efficiently than random or exhaustive screening (an approach called meta-QSAR or adaptive screening).

Agents often need to share two kinds of knowledge — much like a team of people working a big job who must stay on the same page: 1. World-state knowledge: each agent only sees part of the environment (its own sensors, position, observations). Sharing this — a common map, discovered resource locations, the status of other agents — lets the group build a fuller, more accurate picture than any one agent could alone, enabling better-coordinated decisions. 2. Task and goal knowledge: which tasks are claimed, in progress, or completed, and by whom. Sharing this prevents duplicated effort, enables load balancing (idle agents pick up unassigned tasks), and supports dependencies (agent B can begin once agent A signals a prerequisite is done). Without it, agents work in isolation and may conflict or leave important tasks unaddressed.