Reinforcement learning (RL) is a paradigm in which an agent learns a policy — a mapping from states to actions — by interacting with an environment. At each step it observes the current state, selects an action, and receives a scalar reward signal. Its objective is to maximise the expected cumulative discounted reward over time. Crucially, it receives no labelled 'correct' actions — only evaluative feedback on how good its choices were. Contrast with supervised learning: In supervised learning a labelled dataset is provided upfront, with a correct output for every input. The model minimises the error between its predictions and these labels. RL has no such dataset — the agent must generate its own experience by acting in the world, and the reward may be delayed by many time steps (the credit-assignment problem). Additionally, in RL the agent's actions influence the data it sees next, whereas in supervised learning the training data is fixed and independent of the model.

Genetic algorithms (GAs) are search and optimisation methods inspired by biological evolution. A population of candidate solutions (each encoded as a 'genome') is maintained. At each generation: (1) individuals are evaluated by a fitness function, (2) fitter individuals are more likely to be selected as parents, (3) selected parents are combined via crossover and perturbed via mutation to produce offspring, and (4) offspring replace the least fit individuals. Over many generations the population tends to converge toward increasingly good solutions — without requiring an explicit formula for the optimum. Two typical applications: 1. Engineering design — evolving antenna shapes, aerodynamic profiles, or structural layouts where the search space of possible designs is too vast to enumerate exhaustively. 2. Combinatorial optimisation — finding near-optimal solutions to problems such as the Travelling Salesman Problem, job-shop scheduling, or timetabling, where exact methods are computationally infeasible.

AI planning requires a system to reason about what is true in the world, what actions are available, and how those actions change the world — in order to find a sequence of actions that achieves a goal. Knowledge representation provides the formal language in which all of this is expressed, and without it a planner cannot function. Specifically, knowledge representation in planning must encode: 1. World state — a description of what is currently true (e.g. 'the robot is in room A', 'the door is closed'). In classical STRIPS/PDDL planning this is a set of ground predicates; in richer systems it may be a first-order logic or ontology. 2. Actions — each action has a name, preconditions (what must be true for the action to be applicable) and effects (what becomes true or false after the action is executed). Without this the planner cannot determine which actions are legal in a given state. 3. Goal — a description of the desired end state. The planner searches for a sequence of applicable actions that transforms the initial state into one satisfying the goal. The expressive power of the representation determines what kinds of problems can be modelled: propositional logic is simple but limited; first-order logic is expressive but harder to reason with efficiently. PDDL (Planning Domain Definition Language) is the standard representation used in classical AI planning competitions.

Each candidate solution in the GA represents a complete city layout — for example, a grid in which each cell is assigned a zone type (residential, commercial, park, road, utility infrastructure). The fitness function evaluates each layout by running simulations: a traffic model measures flow and congestion, an energy model estimates consumption, and a liveability score combines green-space coverage with proximity of residents to services. The GA starts with a population of random layouts. At each generation the better-scoring layouts are selected to reproduce: crossover swaps zones or sub-regions between two parent layouts to create children that inherit features of both, and mutation randomly reassigns a zone or reroutes a road segment to introduce variation. Over many generations the population converges toward city designs that score well across all objectives, revealing trade-offs (e.g. maximising green space vs. minimising commute distances) that human planners can then evaluate.

1. Representation / Encoding The city footprint is divided into an M×N grid of equal-sized cells. Each cell is encoded as a categorical value from a fixed vocabulary: {Residential, Commercial, Park, Road, Utility, Empty}. The genome of one individual is the full M×N matrix. Alternatively, a graph encoding could represent zones as nodes and roads as weighted edges, allowing variable topology — but the grid is simpler to apply standard operators to. 2. Population initialisation Generate P random layouts (e.g. P = 200) subject to basic hard constraints: total road coverage ≥ minimum required for connectivity; residential zones ≥ target population capacity. Random initialisation ensures broad initial diversity. 3. Fitness function Each layout is evaluated by four sub-scores, combined as a weighted sum: • Traffic score: simulate vehicle flows using a traffic model (e.g. BPR function); penalise congestion hotspots and long average commute times. • Energy score: estimate total energy use based on building density, distance from utilities, and transport distances; reward layouts where renewable generation sites are close to demand centres. • Green space score: fraction of total area classified as Park, with bonus for accessibility (residential cells within walking distance of a park). • Liveability score: average distance from each residential cell to the nearest school, hospital, and commercial zone — lower is better. fitness = w₁·traffic + w₂·energy + w₃·green + w₄·liveability 4. Selection Tournament selection: randomly sample k=5 individuals from the population and select the one with the highest fitness as a parent. Repeat to select a second parent. Tournament selection controls selection pressure without requiring fitness scaling. 5. Crossover Two-point crossover on the flattened genome: choose two random indices and swap the sub-sequence between them between the two parents to produce two children. Alternatively, uniform crossover assigns each cell independently from either parent with probability 0.5. After crossover, check and repair hard constraints (e.g. re-add roads if connectivity is broken). 6. Mutation With a small probability (e.g. 0.01 per cell), randomly reassign a cell's zone type. A smarter mutation operator could swap a residential cell adjacent to a congested road for a Park cell — domain-specific mutations can speed convergence. 7. Replacement and termination Use elitism: always carry the top 5% of individuals unchanged into the next generation to prevent losing the best solutions. Replace the rest with offspring. Terminate after a fixed number of generations (e.g. 500) or when the best fitness has not improved for 50 consecutive generations. The best individual found is the proposed city layout.

Alternative: Multi-objective Deep Reinforcement Learning (MORL). Approach: Frame city design as a sequential decision process. At each step an RL agent chooses where to place the next zone element (residential block, park, road segment, utility). The environment is a city simulator that returns rewards based on how the placement affects traffic, energy, green space, and liveability. A deep Q-network or policy-gradient agent learns which placements lead to the best long-term composite reward. Pareto-front methods can be used to explore trade-offs between the four objectives without collapsing them to a single scalar. Comparison: Similarities — Both are black-box optimisers that require no closed-form gradient of the objective, and both use a simulation-based fitness/reward signal. Differences: 1. Population vs. single agent — GA maintains a diverse population exploring many regions of the design space simultaneously, which is beneficial for a one-shot static design problem. RL uses a single agent that improves incrementally; it is better suited when the design decisions must be made sequentially or must adapt to real-time feedback (e.g. adjusting a live city as population grows). 2. Sample efficiency — GAs evaluate hundreds of full city layouts per generation; each evaluation is expensive. RL can learn incrementally from partial episodes and may be more sample-efficient for large design spaces, but requires careful reward shaping to handle the long time horizons involved in city design.

Q-learning is a model-free, value-based reinforcement learning algorithm. It learns an action-value function Q(s, a) — the expected cumulative discounted reward of taking action a in state s and then following the optimal policy. Updates are made using the temporal-difference rule: Q(s,a) ← Q(s,a) + α [r + γ · max_a' Q(s', a') − Q(s,a)] The policy follows directly: always select the action with the highest Q-value. What distinguishes it: • From supervised learning — no labelled dataset of correct actions is provided; the agent learns from delayed, evaluative reward signals generated by its own interactions. • From unsupervised learning — there is a reward signal guiding learning toward a goal; unsupervised learning has no such signal. • From model-based RL — Q-learning needs no model of the environment's transition probabilities; it learns purely from sampled experience (model-free). • From on-policy RL (e.g. SARSA) — Q-learning is off-policy: it learns about the optimal greedy policy even while following a different (e.g. ε-greedy) exploration policy, because the update uses max_a' Q(s', a') rather than the action actually taken.

A generative model learns the underlying probability distribution of training data and can sample new data points from that distribution — it models how the data was generated rather than simply mapping inputs to class labels. Two examples where generative models are used: 1. Image synthesis — GANs (Generative Adversarial Networks) and diffusion models learn the distribution of photographic images and generate new, realistic images that do not exist in the training set (e.g. StyleGAN producing synthetic human faces). 2. Natural language generation — large language models such as GPT learn the distribution of text and generate coherent new passages, used in chatbots, code generation, and creative writing tools. Task that does NOT use a generative model: Image classification — a CNN trained to label photos as 'cat' or 'dog' is a discriminative model. It learns a direct mapping from input pixels to class probabilities P(class | image) without modelling the distribution of images themselves. It cannot generate new cat images — it can only classify ones it is given.

A rule-based AI system (expert system) consists of four main components: 1. Knowledge base — stores domain knowledge as a collection of IF-THEN rules (e.g. 'IF gene X is deleted AND compound Y is absent THEN growth = False') and a set of known facts. In the Robot Scientist case study this was encoded as an OWL-DL ontology of biological relationships. 2. Working memory (fact base) — holds the current set of asserted facts representing the known state of the world at any point during reasoning. As the system draws conclusions, new facts are added here. 3. Inference engine — applies the rules in the knowledge base to the facts in working memory to derive new conclusions. Two strategies: forward chaining (data-driven — start from known facts and fire applicable rules until no more apply) and backward chaining (goal-driven — start from the goal and work backwards to find supporting facts). The Robot Scientist used forward chaining over its OWL-DL ontology. 4. Explanation / transparency facility — because the system's reasoning is explicit and symbolic, it can trace and present the chain of rules that led to any conclusion. This is a major advantage over neural networks — the system can justify its outputs.

The problem is sequential and adaptive: which content to show next depends on the student's current engagement state, and decisions made now affect engagement minutes later — exactly the structure an MDP captures. The system must make a series of content decisions over the course of a session, where the effect of showing a video vs. a quiz is not immediately clear but becomes apparent in the student's subsequent EEG readings. Q-learning is appropriate because there is no prior model of how different students respond to different content types — that relationship must be learned from experience (model-free). As the system interacts with students across sessions, it updates its Q-values and learns which content actions tend to sustain or restore high engagement for a given state, without needing a hand-crafted model of student psychology.

State (S): The state captures all information needed to decide which content to show. Since the EEG system provides accurate concentration and engagement levels, the state vector includes: • Current engagement level (e.g. low / medium / high, from EEG) • Current concentration level (from EEG) • Time elapsed in the session (e.g. 0–30 min / 30–60 min / 60–90 min) • Current content type being shown (video / quiz / exercise / reading) • Recent trend in engagement (rising / stable / falling — computed from last N readings) • Student performance on recent quiz items (if any) Action (A): The actions are the content interventions the system can apply: • Continue current activity • Switch to a short video lecture • Present an interactive quiz • Show a worked example / explanation • Suggest a short break (2 minutes) • Switch to a gamified exercise Reward (R): Reward reflects the true objective — sustained learning engagement: • +1 for each time step in which engagement remains high • +2 for a correct quiz answer (evidence of learning) • −1 for each time step with low engagement • −3 if the student pauses, switches away, or abandons the session Transition function (T): T(s, a, s') = probability of moving to state s' after taking action a in state s. This is unknown and must be learned from experience — how different students' EEG readings evolve after each content type is shown. Discount factor (γ): A value such as γ = 0.95 ensures the agent values sustained engagement throughout the session, not just immediate spikes. Episode: One complete study session (e.g. 90 minutes). A new episode begins each session.

Role: Q-learning maintains and updates Q(s, a) — the estimated value of taking each content action in each engagement state. It learns the optimal content policy: which activity to present in each state to maximise cumulative engagement reward over the session. After sufficient training across many student sessions, the Q-table (or neural network approximation, if the state space is large) captures effective personalised content sequencing strategies for each engagement profile. Inputs to the Q-learning update at each time step: • Current state s — the full state vector from EEG readings, session time, current content type, and recent performance • Action taken a — which content intervention was applied • Reward received r — engagement score for this time step • Next state s' — updated EEG readings and context after the action Outputs: • Updated Q-values Q(s, a) for the (state, action) pair just experienced • The policy π(s) = argmax_a Q(s, a) — the recommended content action for the current state, which is passed to the content delivery system.

Alternative: Supervised learning recommendation system (collaborative filtering + content-based filtering). How it would work: Collect historical data from many previous students — for each session record the state (EEG engagement levels, session time, content type) and the engagement outcome after each content switch. Train a supervised model (e.g. gradient-boosted trees or a neural network) to predict: 'given this current state, which content type produced the highest engagement in similar students?' At runtime: the system observes the current EEG state, queries the trained model to predict which action will yield the best engagement, and presents that content — without any online learning during the session. Comparison to Q-learning: The recommender requires a large historical dataset labelled with engagement outcomes, whereas Q-learning can start with no data and improve online. However, the recommender is faster to deploy (no exploration phase) and more stable (no risk of initially poor decisions while the Q-table is empty). A hybrid approach — pre-train on historical data, then fine-tune with Q-learning online — would combine the best of both.

A neural network is a computational model loosely inspired by the structure of the biological brain. It consists of layers of interconnected nodes (neurons): an input layer, one or more hidden layers, and an output layer. Each connection has a learnable weight; each neuron applies an activation function to a weighted sum of its inputs. During training, backpropagation adjusts the weights to minimise a loss function, enabling the network to learn complex mappings from inputs to outputs from examples. Example: A convolutional neural network (CNN) used in a medical imaging AI system that receives X-ray images as input and outputs a probability distribution over diagnoses (e.g. pneumonia / normal). The CNN learns to detect progressively complex visual features — edges → textures → anatomical structures — that distinguish healthy from abnormal tissue, without hand-coded rules.

Bio-inspired computing is a field that designs algorithms and computational systems by abstracting mechanisms observed in natural biological processes. The principle is that evolution and biology have produced highly effective solutions to complex problems over billions of years, and capturing those mechanisms algorithmically can yield powerful optimisation and problem-solving methods. Example — Ant Colony Optimisation (ACO): inspired by the pheromone-laying behaviour of real ants searching for food. Artificial ants traverse a graph, depositing virtual pheromone on edges in proportion to solution quality. Shorter, better paths accumulate more pheromone and attract more ants, gradually concentrating the search on high-quality routes. ACO is widely used for vehicle routing, network routing, and scheduling problems.

Example: STRIPS / PDDL (Planning Domain Definition Language) classical planning. How a plan is represented: A plan is an ordered sequence of ground actions: [action₁, action₂, …, actionₙ]. Each action is defined by: • Name and parameters — e.g. MoveRobot(RoomA, RoomB) • Preconditions — a conjunction of predicates that must be true in the current world state for the action to be applicable (e.g. RobotAt(RoomA) ∧ DoorOpen(RoomA, RoomB)) • Effects — predicates that become true or false after the action executes (e.g. ¬RobotAt(RoomA) ∧ RobotAt(RoomB)) The world state is a set of true ground predicates. The planner searches for an action sequence that transforms the initial state into one satisfying the goal condition (e.g. RobotAt(RoomC) ∧ PackageDelivered). Applied to the virtual pop star: a high-level plan might be [GenerateLyrics(theme='summer', style='pop'), ComposeMelody(lyrics, key='C_major'), ArrangeInstrumentation(melody), SynthesiseVoice(lyrics, melody), MixTrack(voice, instrumentation)] — each step with preconditions (previous step complete) and effects (output artefact produced).

Stage 1 — Lyrics generation Technique: Large Language Model (LLM) — e.g. GPT-4 style transformer. Input: a style/theme prompt (e.g. 'upbeat pop song about new beginnings, 3 verses and a chorus, rhyming ABAB'). Output: structured lyrics text with verse/chorus/bridge labels. Justification: Transformers are state-of-the-art for natural language generation. They can capture long-range dependencies (rhyme schemes, thematic coherence across verses) and have been trained on vast corpora of song lyrics, making them well-suited to generating metrically consistent, thematically appropriate text. Stage 2 — Music composition and arrangement Technique: Variational Autoencoder (VAE) or GAN-based music generation (e.g. MuseGAN / MusicVAE). Input: the lyrics structure (number of syllables per line, stress pattern) and style tags (tempo, genre, key, mood). Output: a MIDI score — melody aligned to lyric syllables, chord progression, drum pattern, and arrangement (instruments). Justification: VAEs learn a continuous latent space of musical styles, enabling smooth control over tempo and feel. GANs produce convincingly realistic musical sequences. Both handle the structured, sequential nature of music. Stage 3 — Singing voice synthesis Technique: Neural singing voice synthesis — e.g. a VITS or SongCreator-style model (neural vocoder + acoustic model). Input: lyrics text + melody/rhythm from Stage 2 (MIDI with note–syllable alignment). Output: a realistic audio waveform of the virtual pop star's singing voice. Justification: Modern neural vocoders (WaveNet, HiFi-GAN) can synthesise high-fidelity audio indistinguishable from human singing. The model can be conditioned on a specific voice identity, enabling a consistent virtual pop star persona across songs.

Data flow through the pipeline: User → System: The user provides a natural-language brief via a web interface: theme, genre, mood, approximate duration, and optionally a sample melody or reference track. 1. LLM Lyrics Generator Input: user brief (text prompt) Output: structured lyrics document — verse/chorus/bridge sections with line breaks, annotated with syllable counts per line and rhyme scheme → Passed to: Stage 2 2. Music Composition Model (VAE/GAN) Input: lyrics structure (syllable counts, stress patterns, section layout) + style tags extracted from the user brief (tempo BPM, key, genre, mood) Output: a multi-track MIDI file — melody track with note–syllable alignment, harmony/chord track, bass track, drum/percussion track, and tempo/time signature map → Passed to: Stage 3 (melody + lyrics) and audio rendering (instrumentation) 3. Singing Voice Synthesiser Input: lyrics text (word/phoneme sequence) + melody MIDI (note pitch and duration aligned to each syllable) Output: a high-fidelity WAV audio file of the virtual pop star's vocal performance → Passed to: final mixing 4. Mixing / Mastering (audio production software or a neural mastering model) Input: vocal WAV + rendered instrumental audio (from MIDI via software synthesiser) Output: a mixed and mastered stereo MP3/WAV — the completed song → Delivered to the user via the web interface

The user accesses a browser-based web application. They are presented with a simple form containing plain-English fields: a text box for the song theme ('a breakup song, hopeful not sad'), dropdowns for genre, tempo feel (slow / mid / upbeat), song length, and an optional voice style selector (e.g. 'bright soprano', 'warm alto'). They click 'Generate Song'. A progress indicator shows the pipeline running (Lyrics → Music → Voice → Mix). After approximately one to two minutes a built-in audio player appears with the completed song. The user can play it immediately, download it as an MP3, or click 'Regenerate' to produce a different version with the same settings. No technical knowledge, Python, or command-line interaction is required at any point.

Model 1 — Boden's Taxonomy of Creativity What it is: Margaret Boden (2004) identifies three types of creativity: • Exploratory — generating novel outputs by searching within an existing conceptual space • Combinational — combining familiar ideas in new ways • Transformational — changing the rules of the conceptual space itself Applied to the song generator: The lyrics LLM and music VAE both operate primarily at the exploratory level — they generate novel songs by sampling from the distribution of existing human-created music and lyrics encoded during training. They explore a vast conceptual space of possible songs without stepping outside it. The system also exhibits combinational creativity: it can blend genre styles (e.g. jazz harmonics + hip-hop rhythm + pop melody) in ways that individual training examples did not, and combines the output of three subsystems that were trained independently. The system does not exhibit transformational creativity — it cannot invent a new musical genre, a new lyrical form, or a new vocal technique that did not exist in its training data. The conceptual space is fixed by what humans have already created. Overall Boden assessment: the system is a sophisticated exploratory and combinational creative system, but not transformationally creative. --- Model 2 — Skinnerian vs. Popperian Creativity (after Dennett / Boden) What it is: Skinnerian creativity produces novel outputs through stimulus–response without any internal model or self-evaluation — outputs are generated and selected by an external environment (like an animal reinforced by trial and error). Popperian creativity involves an agent that has an internal model of the world and can pre-evaluate ideas before acting — it generates hypotheses internally, tests them mentally, and only outputs those it 'believes' are good. Applied to the song generator: The system is primarily Skinnerian: each subsystem maps its inputs to outputs probabilistically based on learned statistical patterns — the LLM predicts likely next tokens; the VAE samples from its prior. Neither has an internal model of 'what makes a good song' that it uses to evaluate and revise ideas before outputting them. The system does not internally compose ten melody options, evaluate them against aesthetic criteria, and present the best — it generates one sample and delivers it. A more Popperian system would include an evaluation module (e.g. a trained quality discriminator or a human-in-the-loop feedback loop) that can judge candidate outputs and trigger regeneration if quality is insufficient — bringing the system closer to how a human songwriter drafts, critiques, and revises.