Yes, an MDP is a broadly suitable framework for pandemic decision making, though with important caveats. Strength — sequential decision making under uncertainty: A pandemic unfolds over time as a sequence of decisions (impose lockdown, lift restrictions, subsidise businesses…) whose effects are delayed and uncertain. MDPs are explicitly designed for exactly this setting: an agent observes a state, takes an action, receives a reward, and the environment transitions to a new state. The Bellman framework allows the agent to plan over long horizons and account for the fact that a short-term economic cost (lockdown) may yield a long-term health benefit (fewer deaths), which simple rule-based systems cannot do. RL on top of the MDP lets the government learn from ongoing data without needing a complete model of disease dynamics in advance. Weakness — the Markov assumption is violated: The Markov property requires that the current state contains all information needed to predict the future — the past is irrelevant given the present. In a pandemic this is almost certainly false. The current infection rate alone does not capture accumulated immunity, population fatigue with restrictions, vaccine rollout stage, variant characteristics, or public trust in government — all of which heavily influence how actions play out. This makes the problem more accurately a Partially Observable MDP (POMDP), which is far harder to solve. A naively applied MDP will make systematically wrong predictions because it ignores this history.

State (what the agent observes): The state must capture a snapshot of the world sufficient to make a good decision. In the pandemic context this would include quantitative measurements such as: current daily infection rate and hospitalisation numbers; COVID-related ICU occupancy and healthcare system capacity; recent trajectory (rising/falling); economic indicators such as unemployment rate and GDP growth; vaccination coverage in the population; and the current policy setting (e.g. whether a lockdown is already in force). In practice the state is a high-dimensional vector of these variables, updated daily or weekly. Action (what the government can do): The action space is the set of policy levers available. Examples include: imposing or lifting a full national lockdown; introducing or removing household mixing restrictions; mandating or relaxing mask-wearing; launching subsidised restaurant/hospitality vouchers ('Eat Out to Help Out'); accelerating vaccine rollout to a target age group; closing or reopening schools. Actions may be discrete (on/off) or continuous (e.g. the percentage subsidy offered). Reward (what the agent is trying to maximise): The reward signal must encode the dual objective of health and economic wellbeing. A composite reward could be: r = − α · (new_deaths + β · hospitalisations) + γ · (GDP_index) − δ · (restriction_cost) where α, β, γ, δ are weights that reflect government priorities. Negative rewards penalise deaths and healthcare overload; positive rewards reflect economic activity. Designing this weighting is itself a major ethical and political decision — it embeds a value judgement about how many economic units a human life is 'worth'.

Role: The Q-network (a neural network approximating the Q-function) replaces the Q-table used in tabular Q-learning. Because the pandemic state space is continuous and high-dimensional (a vector of infection rates, economic indicators, policy settings, etc.), it is impossible to enumerate every state–action pair in a table. The Q-network generalises across similar states: given any state it has not seen before, it estimates the expected cumulative discounted reward for each possible government action. The optimal policy then follows from always selecting the action with the highest Q-value: π(s) = argmax_a Q(s, a). Inputs: The input layer receives the current state vector — a fixed-length numerical encoding of all observable variables: e.g. [daily_cases, hospital_occupancy, icu_rate, unemployment, gdp_growth, vaccination_pct, current_restrictions_flag, …]. Each variable is normalised so the network can train stably. The input dimensionality equals the number of state variables. Outputs: The output layer has one neuron per possible government action (assuming a discrete action space). Each output neuron produces the estimated Q-value Q(s, a_i) for taking action a_i in the current state s. For example, with 6 available policy actions there would be 6 output neurons. The government (or the policy) selects the action corresponding to the highest output value. This single-forward-pass architecture is more efficient than evaluating each action separately, as it computes all Q-values simultaneously.

Limitation: Reward misspecification and Goodhart's Law. The MDP framework optimises exactly the reward function it is given. In the pandemic setting the true objective — maximising population wellbeing — is deeply complex and cannot be fully captured in a mathematical formula. The composite reward (deaths, hospitalisations, GDP) is a proxy. When a powerful optimiser like deep RL relentlessly maximises a proxy, it tends to find unexpected strategies that score well on the metric while violating the underlying intent — a phenomenon described by Goodhart's Law: 'When a measure becomes a target, it ceases to be a good measure.' Concrete example: if the reward heavily penalises ICU overflow but not community deaths, the learned policy might recommend transferring critically ill patients to care homes (reducing recorded ICU numbers) rather than reducing infections — optimising the number while causing real harm. Similarly, if economic output is rewarded without a long-run health term, the agent may recommend reopening too early, producing a second wave that causes far more total damage. This is not hypothetical — the UK's 'Eat Out to Help Out' scheme, while economically well-intentioned, is estimated to have contributed to accelerating the second COVID wave. How to overcome it: 1. Reward shaping with domain experts: Epidemiologists, economists, ethicists, and public health officials should collaboratively design the reward, explicitly testing it for pathological optima in simulation before deployment. The reward should include long-run terms (excess mortality over a year, not just this week's ICU count). 2. Constrained RL: Rather than encoding everything in a scalar reward, impose hard constraints — e.g. the policy must never allow hospital capacity to exceed 90%, regardless of economic reward. Constrained MDP formulations (CMDP) can incorporate these safety constraints directly into the optimisation. 3. Human-in-the-loop oversight: The RL system should be treated as a decision-support tool, not an autonomous decision-maker. Recommendations should be reviewed by human experts who can identify reward-hacking strategies before they are implemented. Regular audits comparing model recommendations against epidemiological common sense provide a critical safety layer. 4. Inverse Reward Design (IRD): Rather than hand-crafting the reward, infer it from human expert demonstrations of good pandemic management — the agent learns what objectives underlie observed human decisions, reducing the risk of misspecification.

Ethical issue: Equity and distributional fairness. A reward function that maximises aggregate population health and GDP will tend to find policies that are optimal on average — but 'optimal on average' can mask severe inequity. Pandemic impacts are deeply unequal: low-income workers cannot work from home, people in overcrowded housing cannot self-isolate effectively, ethnic minority communities faced disproportionate COVID mortality, and small businesses were far more vulnerable to lockdowns than large corporations. A system that optimises aggregate outcomes may systematically recommend policies that protect the well-off while imposing the greatest burden on the most vulnerable — all while appearing to maximise the chosen metric. Why this is serious: Government decisions affect millions of people. Embedding an RL system in policy-making without explicit equity constraints risks automating and legitimising inequality behind a veneer of scientific objectivity. The opacity of deep neural networks makes it difficult for affected communities to challenge or understand the basis of decisions that harm them. How to handle it: 1. Disaggregated reward terms: Add explicit equity-weighted terms to the reward function — e.g. penalise disproportionate mortality in deprived communities relative to the population average, or include a Gini-style inequality index over health outcomes across demographic groups. 2. Fairness constraints in CMDP: Formulate hard constraints requiring that no demographic group's projected mortality exceeds a threshold multiple of the population average — preventing the agent from trading minority health for aggregate gain. 3. Participatory design: Involve representatives from disadvantaged communities in defining what 'good outcomes' mean. The people most affected by pandemic policies should have a voice in what the system is instructed to optimise, not just technical experts. 4. Transparency and contestability: Publish the model's recommendations alongside plain-language explanations of why they were made. Establish an independent ethics board with the authority to reject recommendations that, even if aggregate-optimal, impose unacceptable burdens on specific groups.

A robot scientist is an autonomous system capable of performing the full closed loop of scientific discovery without human intervention at each step. Rather than simply analysing data provided by humans, it can: form hypotheses about the world, design experiments to test those hypotheses, physically execute the experiments using robotic laboratory equipment, collect and interpret the results, and then use those results to generate refined or new hypotheses — cycling continuously. The key conceptual contribution is the automation of the scientific process itself, not just data analysis. The Adam system, the case study in the course, was the first robot scientist demonstrated to have autonomously discovered new scientific knowledge (novel gene-function associations in yeast metabolism).

Capabilities: Adam could (1) autonomously generate hypotheses about unknown gene functions, (2) select the most informative experiments to run next (active learning), (3) physically execute those experiments via a robotic laboratory, (4) analyse results and update its knowledge base, and (5) repeat the cycle without human direction. Data sources: • Yeast genome databases (Saccharomyces Genome Database, SGD) — listing all ~6,000 genes, their sequences, and known functions • Metabolic pathway databases (e.g. KEGG) — known biochemical reactions in yeast, including 'orphan reactions' where the catalysing enzyme's gene was unknown • Enzyme Commission (EC) classifications — mapping reactions to enzyme types • Previous experimental results — growth/no-growth observations from knockout strain assays Steps: 1. Query the knowledge base to identify gaps (genes with unknown function / reactions with no known gene) 2. Apply abductive reasoning over the OWL-DL ontology to generate ranked candidate hypotheses 3. Select the experiment predicted to provide maximum information gain 4. Issue instructions to LABORS (the robotic platform) to execute the experiment 5. Receive results, update the ontology, and return to step 1

How Adam / the full robot scientist makes hypotheses: Hypothesis generation is driven by abductive reasoning over the OWL-DL knowledge base. The system identifies metabolic reactions in yeast for which the catalysing enzyme's gene is unknown (an 'orphan enzyme' gap in the ontology). It then searches for candidate genes — open reading frames with no assigned function — that are plausible encoders of that enzyme, based on sequence homology to known enzymes in related organisms, their position in the metabolic network, and any existing phenotypic observations. Each candidate gene-enzyme pairing becomes a hypothesis: 'Gene X encodes enzyme E in pathway P.' In the toy robot scientist: The toy implementation in the course did carry out hypothesis generation, but in a simplified form. Rather than performing sequence homology searches from scratch, it selected hypotheses from a pre-computed set of candidates drawn from the SGD and pathway databases already loaded into its knowledge base. It used the logical structure of the OWL-DL ontology to rank candidates by plausibility. To generate truly novel hypotheses de novo (as the full Adam system did) would additionally require: integration with live sequence databases, a bioinformatics homology-search engine (e.g. BLAST), and a larger, continuously updated ontology — infrastructure the toy system simplified away.

How predictions are derived: Once a hypothesis is formed — e.g. 'Gene YJL060W encodes the enzyme catalysing reaction R in the lysine biosynthesis pathway' — a falsifiable prediction follows directly from the metabolic pathway model: 'A yeast strain with gene YJL060W deleted (a knockout) will fail to grow on minimal medium lacking lysine, because without the enzyme it cannot complete the pathway and synthesise lysine itself.' This prediction is derived by tracing the consequences of the gene deletion through the pathway ontology: if gene X → enzyme E → reaction R → product P → downstream pathway, then deleting X blocks everything downstream. The OWL-DL representation makes this inference automatic — the reasoner can traverse the ontology and identify which compound will be missing. In the toy robot scientist: The toy system did carry out this step. Given a candidate gene-enzyme hypothesis, it queried the pathway structure encoded in its knowledge base to derive the growth prediction for the corresponding knockout strain on specific media. This step worked well in the toy because pathway connectivity was already encoded — it required no additional capability beyond what was implemented.

How the full Adam system tested predictions: Adam issued formal experimental instructions to LABORS (the Laboratory Robotic System) — a physical robotic platform capable of: retrieving the correct knockout yeast strain from a strain library, preparing growth media (with and without the target compound), inoculating plates, incubating them at the correct temperature, and reading optical density (OD) to determine growth or no-growth. The binary outcome (the strain grows ↔ does not grow) either supports or refutes the hypothesis. Confirmed associations were asserted into the OWL-DL knowledge base as new facts; refuted ones led to the hypothesis being discarded or revised. In the toy robot scientist: The toy system did NOT execute physical experiments. This step was simulated — experimental outcomes were looked up from a pre-existing dataset of knockout strain growth results rather than being run in a real laboratory. This is the most significant limitation of the toy compared with Adam. What would be necessary to carry it out for real: a physical robotic laboratory (automated liquid-handling robots, incubators, plate readers), a library of yeast knockout strains, laboratory information management software to track samples, and a feedback interface between LABORS and the knowledge base so that real results update the ontology.

Area of science: Functional genomics / systems biology of yeast (Saccharomyces cerevisiae). General problem: Gene function annotation — specifically the 'orphan enzyme' problem. Although the yeast genome had been fully sequenced and its metabolic pathways were largely mapped, many metabolic reactions were known to occur but the gene encoding the catalysing enzyme was unknown. Adam was tasked with systematically identifying which yeast genes are responsible for these reactions, thereby filling gaps in our understanding of yeast metabolism and demonstrating that a robot could autonomously make genuine scientific discoveries.

Data involved: • Yeast genome data — the complete list of ~6,000 open reading frames from the Saccharomyces Genome Database (SGD), including known gene functions, sequence data, and phenotypic observations from prior experiments • Metabolic pathway data — all known biochemical reactions in yeast, their substrates, products, and enzyme commission (EC) classifications, drawn from databases such as KEGG and EcoCyc • Knockout strain data — a library of ~4,700 viable yeast deletion strains (one gene knocked out per strain), and prior growth/no-growth assay results on different media • Enzyme homology data — cross-species sequence similarity scores linking candidate yeast genes to known enzymes in other organisms Organisation — OWL-DL: All knowledge was encoded in an OWL-DL (Web Ontology Language — Description Logic) ontology. OWL-DL was chosen because it is expressive enough to capture complex biological relationships (class hierarchies, part-of relations, reaction participants, gene-enzyme-reaction links) while remaining decidable — a sound, complete logical reasoner can be guaranteed to terminate when answering queries. The ontology allowed Adam to formally represent and query statements such as: 'Reaction R is catalysed by an enzyme of class EC 2.x.x.x, and gene G has sequence homology to known EC 2.x.x.x enzymes in E. coli, and no gene has yet been assigned to R in S. cerevisiae' — from which a hypothesis naturally follows. Generating hypotheses: Adam's planner queried the ontology for gaps (reactions with no assigned gene) and ranked candidate genes by a combination of sequence homology scores and pathway plausibility. The top-ranked gene-reaction pairings became hypotheses. An active-learning criterion selected whichever hypothesis, if tested, would maximally reduce uncertainty across the whole knowledge base. Generating experimental designs — LABORS: From each hypothesis Adam automatically generated a formal experimental protocol: which knockout strain to test, on which growth media (minimal medium ± specific amino acids or metabolites), with what controls. These instructions were passed to LABORS (Laboratory Robotic System) — the physical automation platform that robotically retrieved strains, prepared media, inoculated plates, incubated, and read OD measurements. Results were fed back and asserted into the OWL-DL ontology, closing the loop.

The generative AI music system in the course used three cooperating subsystems: 1. VAE Encoder (operates at training time; at generation time it is bypassed in favour of sampling the prior) Purpose: Compress a musical piece into a compact latent representation. Inputs (training): A musical sequence (e.g. a MIDI piano roll — a matrix of notes over time). Outputs (training): The parameters (mean μ and variance σ) of a Gaussian distribution in latent space, from which a latent vector z is sampled via the reparameterisation trick. AI technique: Recurrent neural network (bidirectional LSTM) or CNN; variational inference. 2. VAE Decoder / Generator Purpose: Generate a new musical sequence from a latent vector z. Inputs (generative mode): A latent vector z sampled from the prior N(0,I) or interpolated between two known points in latent space. Outputs: A musical sequence — a reconstructed or novel MIDI piano roll representing a melody, chord progression, or drum pattern. AI technique: Recurrent neural network (hierarchical LSTM or GRU); trained to minimise reconstruction loss jointly with the KL divergence term of the VAE objective. 3. Discriminator (GAN component) or Groove/Style Classifier Purpose: Improve the realism of generated sequences by distinguishing real from generated music (if GAN-based), or by conditioning the decoder on a style embedding. Inputs (generative mode): A generated musical sequence from the decoder. Outputs: A scalar score (real / fake probability) fed back as an adversarial training signal, or a style vector that conditions the generator. AI technique: Convolutional neural network; trained via the GAN minimax objective (generator minimises, discriminator maximises, the probability of correctly classifying real vs. generated sequences).

1. VAE Encoder & Decoder training data: Large corpora of MIDI files — digitised musical scores and performances — are required. Sources typically include the Lakh MIDI Dataset (~176k MIDI files scraped from the web), classical music archives (IMSLP, Mutopia), and curated datasets such as the Piano-e-Competition or JSB Chorales. Ethical concerns: MIDI files themselves are symbolic representations (not audio recordings), and many classical works are in the public domain, so copyright issues are less acute than for audio. However, contemporary composers' MIDI arrangements scraped without consent may infringe copyright. The dataset composition (predominantly Western classical and pop) introduces cultural bias — the model will generate music in those idioms and poorly represent non-Western traditions. 2. GAN Discriminator training data: The discriminator requires both real musical sequences (same MIDI corpus as above, labelled 'real') and generated sequences from the generator (labelled 'fake'). The fake examples are produced by the generator during training — no additional external data is needed. The same ethical caveats about the real corpus apply. Additionally, the adversarial training dynamic means the discriminator's learned notion of 'real music' directly reflects the biases of the corpus — a homogeneous dataset will produce a discriminator that penalises stylistic diversity in generated output. Overall ethical assessment: The training data was likely not fully ethically sourced in a modern sense — large-scale MIDI scraping from the internet typically did not obtain individual rights-holder consent. Ethical best practice would require either using explicitly licensed / public-domain corpora, or compensating composers whose work trains a commercial system. This is an active legal and ethical debate following high-profile lawsuits against generative AI music companies.

What is a latent space: A latent space is a lower-dimensional continuous vector space into which the VAE encoder compresses its inputs. Each point z in this space represents a compact, abstract description of a musical piece — capturing high-level features such as melodic contour, harmonic character, and rhythmic feel, without storing every individual note explicitly. Crucially, the VAE training objective (KL divergence term) encourages the latent space to be smooth and approximately Gaussian: nearby points correspond to musically similar pieces, and the space between two known encodings is also musically coherent. This smoothness is what makes the latent space explorable. How it was explored in the course: Two main exploration strategies were used: 1. Random sampling — drawing z ~ N(0,I) and passing it through the decoder to generate a completely novel musical sequence not present in the training data. This produces varied, unpredictable outputs across the whole latent space. 2. Interpolation — encoding two known musical pieces into their latent vectors z₁ and z₂, then decoding points along the straight line z = (1−t)z₁ + tz₂ for t ∈ [0,1]. As t increases from 0 to 1, the generated music smoothly morphs from the character of piece 1 to the character of piece 2. Results: Interpolation produced musically coherent intermediate pieces — the melody, harmony, and rhythm gradually transformed between the two endpoints rather than jumping abruptly. Random sampling generated novel sequences that sounded plausibly musical but were stylistically constrained by the training corpus. Points far from the origin (outside the learned distribution) produced increasingly incoherent or degenerate sequences, illustrating that the model's generative quality degrades beyond the region it was trained on.

Agency refers to having meaningful control over — or meaningful input into — a process and its outcomes. 1. VAE Encoder (as a user): Human agency: Moderate. The human chose which musical piece to encode — selecting the input was a meaningful creative act. However, the resulting latent vector is produced entirely by the encoder; the human has no visibility into or control over which point in latent space the piece maps to, or why. AI agency: High. The encoder autonomously determines the compressed representation — it decides what is important to retain and what to discard, based entirely on patterns learned during training. The human cannot intervene in this process. 2. VAE Decoder / Generator: Human agency: Low–moderate. In random sampling mode the human has almost no agency — they can only accept or reject the output. In interpolation mode the human chooses the two endpoint pieces and the interpolation parameter t, giving them indirect creative direction. But the actual notes generated are entirely determined by the decoder, not the human. AI agency: Very high. The decoder makes all compositional decisions — which notes to produce, their timing, dynamics — from only a latent vector as input. The creative content is entirely the model's output. 3. Discriminator / Style component: Human agency: Very low. The human has no direct interaction with the discriminator during generation; it operates invisibly to filter or score outputs. AI agency: High but opaque. The discriminator imposes its learned aesthetic judgement (shaped by the training corpus) on what constitutes 'good' music, shaping the generator's outputs without any human being able to easily inspect or challenge these criteria. Overall reflection: Across all three subsystems the AI held substantially more agency than the human user. Human interaction was largely confined to high-level choices (which pieces to use, where to sample in latent space), while all granular creative decisions were made by the model. This raises important questions about authorship: if the human contributes only a starting point and the AI generates the content, who is the creative agent — and who owns the copyright?