Natural Language Processing (NLP) is a subfield of artificial intelligence and computational linguistics concerned with enabling computers to process, understand, interpret and generate human (natural) language — both text and speech. It combines linguistic rules with statistical and machine-learning methods. Its significance in data science is that a very large share of real-world data is unstructured text — reviews, social media, emails, support tickets and documents. NLP turns that unstructured language into structured features and insight (sentiment, topics, named entities, summaries), making it analysable, and it powers core applications such as classification, search, machine translation, chatbots, question answering and information extraction.

Regularisation adds a penalty on the model weights to the loss function to discourage overly large coefficients and reduce overfitting. L1 (Lasso) uses a penalty proportional to λ·Σ|wᵢ|, the sum of the absolute weights. It encourages sparsity, driving some coefficients exactly to zero, so it doubles as feature selection. L2 (Ridge) uses a penalty proportional to λ·Σwᵢ², the sum of squared weights. It shrinks coefficients smoothly toward (but rarely exactly) zero and handles correlated features / multicollinearity well. The regularisation parameter λ (sometimes expressed as C = 1/λ) controls the strength. If λ is too large (too much regularisation) the coefficients are shrunk too aggressively, the model becomes too simple, and it underfits — high bias and poor performance on both training and test data. If λ is too small (too little regularisation) the penalty barely constrains the model, so it fits noise and overfits — high variance, good training accuracy but poor generalisation. λ is tuned (e.g. by cross-validation) to find the bias–variance sweet spot.

The broad options are to delete rows/columns with missing values (only safe when very few and missing at random) or to impute — estimate sensible replacements. With data missing throughout a large set, imputation is preferred. Two methods: 1. Mean / median / mode imputation — pandas .fillna() or sklearn SimpleImputer(strategy='mean'|'median'|'most_frequent'). Each missing value is replaced with the column statistic. Limitations: it distorts the variance and distribution, ignores correlations between features, and the mean is sensitive to outliers (use the median for skewed data); every gap in a column gets the same value, which can bias estimates. 2. KNN imputation — sklearn KNNImputer. A missing value is filled using the (weighted) mean of its k nearest neighbours measured over the other features. Limitations: it is computationally expensive on large datasets, sensitive to the choice of k and requires feature scaling, degrades in high dimensions (the curse of dimensionality), and assumes that similar rows really do have similar values. (Iterative/regression imputation — IterativeImputer / MICE — is also acceptable. It is worth stating the missingness assumption: MCAR, MAR or MNAR.)

Cleaning: strip HTML/markup, URLs, handles and control characters; handle emojis; deduplicate; fix encoding. Language detection: identify each review's language (e.g. langdetect, fastText, langid) so it can be routed correctly. Translation (if applicable): translate non-target languages into one common language (e.g. English) with a machine-translation model/API — or skip translation and use a multilingual model instead. Tokenisation: split into sentences/words with a language-aware tokeniser (some languages such as Chinese or Japanese need segmentation rather than whitespace splitting). Normalisation: lowercasing, punctuation removal, stop-word removal, stemming/lemmatisation. Representation: encode to features — bag-of-words/TF-IDF, or (multilingual) word/sentence embeddings.

Sentiment analysis: classify each review as positive/negative/neutral (or fine-grained) to gauge satisfaction with the new feature, using a lexicon-based or ML/transformer classifier. Aspect-based sentiment analysis: attach sentiment to specific aspects (price, delivery, UI, the new feature) for actionable detail. Topic modelling: unsupervised discovery of recurring themes (LDA, NMF, or clustering of embeddings) to learn what users discuss. Named entity recognition: extract product names, brands, locations and features mentioned. Text classification: route feedback into categories (bug / praise / feature request). Summarisation: condense large volumes of feedback into digestible summaries.

Multilingual data means we need language detection plus either translation or a multilingual transformer (e.g. mBERT, XLM-R) so all feedback is handled consistently; the translation-versus-multilingual-model trade-off (cost, accuracy, drift) should be justified. Short, informal, noisy social text (slang, emojis, misspellings) favours robust preprocessing and contextual embeddings, which tolerate informal language far better than plain bag-of-words. A large, unlabelled dataset suits unsupervised topic modelling/clustering to explore themes without labels, while pre-trained sentiment/NER models avoid expensive manual labelling. Finally, the business goal of improving the shopping experience means aspect-based sentiment plus NER pinpoint which products/features drive positive or negative reactions, directly informing product decisions.

Overall correctness — Accuracy = (TP+TN)/total (note it is misleading under class imbalance). Class-level correctness — Precision, Recall and their harmonic mean F1. Generalisation / overfitting control — a train/validation/test split and k-fold cross-validation, comparing training versus test error. Ranking / threshold behaviour — the ROC curve and AUC (or PR-AUC for imbalanced data). (Other valid choices include the confusion matrix, log-loss, and for regression RMSE / MAE / R².)

Precision = 1 and recall = 1 means a perfect classifier on the positive class: every item it predicts positive really is positive, and it finds every actual positive. It follows that the number of false positives = 0 (precision = TP/(TP+FP) = 1) and the number of false negatives = 0 (recall = TP/(TP+FN) = 1). Trade-offs: moving the decision threshold trades precision against recall. Spam detection favours high precision — a false positive (a legitimate email sent to the spam folder) is costly, so you tolerate missing some spam (lower recall). Medical diagnosis or screening favours high recall — a false negative (missing a real disease) is dangerous, so you tolerate extra false positives, which simply trigger further tests. The right operating point depends on the relative cost of false positives versus false negatives. Class imbalance: when the positive class is rare, a model that mostly predicts the majority class can show high accuracy yet poor recall on the minority, and precision/recall become very sensitive because each false positive or false negative shifts them a lot. Ways to address imbalance include resampling (oversampling the minority, e.g. SMOTE, or undersampling the majority), class weighting / cost-sensitive learning, threshold tuning, using imbalance-aware metrics (F1, PR-AUC) instead of accuracy, and collecting more minority-class data.

Confusion matrix (rows = actual, columns = predicted Sports/Politics/Technology/Health): Sports: 50, 2, 3, 0 (row total 55) Politics: 5, 45, 2, 3 (row total 55) Technology: 3, 2, 40, 5 (row total 50) Health: 2, 3, 5, 40 (row total 50) Column totals: 60, 52, 50, 48 (grand total 210). For each class, TP = diagonal, FP = column total − TP, FN = row total − TP, with Precision = TP/(TP+FP) and Recall = TP/(TP+FN): Sports: TP=50, FP=10, FN=5 → P = 50/60 = 0.833, R = 50/55 = 0.909. Politics: TP=45, FP=7, FN=10 → P = 45/52 = 0.865, R = 45/55 = 0.818. Technology: TP=40, FP=10, FN=10 → P = 40/50 = 0.800, R = 40/50 = 0.800. Health: TP=40, FP=8, FN=10 → P = 40/48 = 0.833, R = 40/50 = 0.800. F-measure per class, F1 = 2PR/(P+R): Sports = 0.870; Politics = 0.841; Technology = 0.800; Health = 0.816. Macro averages (mean of the per-class values): macro precision = (0.833+0.865+0.800+0.833)/4 = 0.833; macro recall = (0.909+0.818+0.800+0.800)/4 = 0.832. Micro averages (pool all classes): ΣTP = 50+45+40+40 = 175 and total predictions = total actuals = 210, so micro precision = micro recall = 175/210 = 0.833. For single-label multiclass classification, micro-precision = micro-recall = micro-F1 = accuracy = 175/210 ≈ 83.3%. Comment: the classifier performs fairly evenly (about 80–83% across classes), strongest on Sports and weakest on Technology/Health, with the main confusions between Health and Technology (5 each way) and Politics predicted as Sports (5). Because the classes are roughly balanced, the macro and micro averages agree. A simple way to improve performance is to gather more and better training data for the weaker, most-confused classes, add discriminating features, and tune the decision boundary between Health and Technology, whose vocabularies overlap most.

The sentence is: “The CEO of Acme Corp., John Doe, announced a merger with Beta Inc. on January 15, 2025.” 1. Abbreviation periods: the ‘.’ in ‘Corp.’ and ‘Inc.’ is part of the token, not a sentence boundary, so a naïve splitter ends the sentence in the wrong place. 2. Multi-token organisations: ‘Acme Corp.’ and ‘Beta Inc.’ are single ORG entities spanning two tokens each and must be grouped rather than split into separate words. 3. Multi-token person name: ‘John Doe’ is one PERSON entity made of two capitalised tokens that must be linked together. 4. Dates spanning tokens with internal punctuation: ‘January 15, 2025’ is one DATE entity but contains a comma, which must not be treated as a list/clause separator. 5. Appositive commas: the commas around ‘John Doe’ set off an appositive; punctuation must be separated correctly while NER still links ‘John Doe’ to the ‘CEO’ role. 6. Role versus entity and capitalisation cues: ‘CEO’ is a title, not a named entity, yet it is capitalised, so NER relying on capitalisation can misfire and must disambiguate roles from organisations and people. (Any six well-justified difficulties earn the marks.)