Introduction to Large Language Models
— the interactive lab

Zero-shot sentiment vs. the supervised classifiers you just built

Prior lecture: TF-IDF features + Naive Bayes / SVM / logistic regression, labeled training data, F1/AUC. Now: no training data at all — describe the task in words, and run it on the same 30-review Amazon sample from the word-vector exercise.

The entire “model” — a system prompt

def classify(text):
    ans = chat(prompt=text, system=SYSTEM, temperature=0.0).upper()
    if "POSITIVE" in ans: return "POSITIVE"
    if "NEGATIVE" in ans: return "NEGATIVE"
    return "UNKNOWN"

for i, row in tqdm(enumerate(sample.itertuples()), total=len(sample)):
    predictions.append(classify(row.text))

print(classification_report(clean["gold"], clean["pred"],
                            labels=["POSITIVE","NEGATIVE"]))

Same input, same prompt — can you trust a single answer?

A deliberately ambiguous review

Majority voting — the easy reliability upgrade

Ten harder reviews, five votes each at temperature=0. Watch the one case where a perfectly unanimous vote is still wrong.

def classify_vote(text, n=5, temperature=0):
    labels = []
    for _ in range(n):
        ans = chat(prompt=text, system=SYSTEM, temperature=temperature).strip().upper()
        hits = re.findall(r"POSITIVE|NEGATIVE", ans)
        labels.append(hits[0] if hits else "NEGATIVE")
    return Counter(labels).most_common(1)[0][0], labels

Topic labeling — interpretation, not clustering

The LLM doesn’t replace LDA — it replaces the human who used to name LDA’s clusters. First the naive version: keywords only.

The production fix: keywords + representative documents + a self-rating

Give the model what a human analyst would get — a few documents near the cluster centroid — and permission to report low coherence.

LABEL_TMPL = (
  "Below are the top keywords and a few representative documents from one cluster.\n"
  "Decide on a 2-4 word label and a one-sentence description.\n"
  "Also rate the cluster's coherence: high / medium / low.\n"
  "If keywords and documents do not share a clear common theme, set coherence to 'low' "
  "and say so. It is acceptable to report low coherence — do not invent a theme.\n\n"
  "TOP KEYWORDS: {kw}\n\nREPRESENTATIVE DOCUMENTS:\n- {docs}"
)

Aspect-based sentiment, structured for a pipeline

Prior lecture: lexicon sentiment — sum the positive words, subtract the negative ones. It misses negation, context, sarcasm. The LLM extracts {aspect: sentiment} pairs directly, as JSON.

Harder: aspects the review never names

This reviewer never says “size,” “waterproofing,” “durability,” or “warranty” — they describe them obliquely. A bag-of-words extractor can’t catch these without a hand-built ontology. Click each extracted aspect to see its evidence in the text.

The review

implicit_prompt = (
  "Extract aspect-sentiment pairs from the product review. "
  "Aspect names should reflect what the customer is actually talking about, "
  "even if they don't use that exact word. "
  "Return JSON: {aspects: [{aspect, sentiment, evidence}]}\n\nREVIEW:\n" + implicit_review
)

Three eras of “similar”: TF-IDF → MiniLM → LLM-era embeddings

One query, seven documents, three methods. The query shares zero words with the two documents that actually answer it.

Query

# 1) TF-IDF cosine
v = TfidfVectorizer().fit(docs + [query])
tfidf_scores = cosine_similarity(v.transform([query]), v.transform(docs))[0]

# 2) Sentence-transformer (BERT-era, 384-dim)
st = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
st_scores = cosine_similarity([st.encode(query)], st.encode(docs))[0]

# 3) LLM-era embedding via Ollama (768-dim)
def embed(text):
    return ollama.embeddings(model="nomic-embed-text", prompt=text)["embedding"]
ollama_scores = [cos(embed(query), embed(d)) for d in docs]

The agreement trap

A tempting pipeline: run the LLM N times, the runs agree, Cohen’s κ ≈ 1.0, declare the labels reliable. But a model can be consistently wrong — Bojić et al. report LLMs beating humans on inter-rater reliability, which measures agreement, not accuracy.

“Just give each review a 0–10 score” — a common business ask

Three reviews — a rave, a genuinely mixed one, a rant. Two tests, both at temperature=0, five runs per condition (every condition returned five identical scores — that’s the stability temperature 0 buys; the question is what the stable number means).

Test 1 — same reviews, two rubrics

Test 2 — same reviews, two reviewer personas

If the score measured the review, the persona shouldn’t matter. Watch the mixed review.