Generative AI and Prompt Engineering

Generative AI architecture and use cases

A generative model predicts what comes next, over and over. Everything it does – writing, coding, summarising – is that one operation.

• Pre-training: the model reads an enormous amount of text and learns to predict the next piece of it. This is where the knowledge comes from.
• Fine-tuning and alignment: further training on examples of helpful, safe answers. This is where the behaviour comes from.
• Inference: what happens when you send a prompt. The model generates one token at a time, each one conditioned on everything before it.
• It has no database and does no lookup. If you need facts from your own documents, you must supply them – that is Module 7.

Large Language Models and transformers

What “large” means, and the differences that matter when you pick one.

• Parameters are the learned weights. More parameters generally means more capability and more cost per call.
• Hosted models (Claude, GPT, Gemini) are called over an API – strongest, no infrastructure, your data leaves your network.
• Open-weight models (Llama, Mistral, Qwen) can run on your own servers – full control, lower unit cost, more work and usually a capability gap.
• Model choice is a testable decision: run the same evaluation set against two models and compare cost, latency and quality. Do not choose by reputation.
• Models are versioned and get deprecated. Pin the exact model id in configuration, never hard-code it in twenty places.

Tokens, context windows and limits

The unit of everything: what you are billed for, what the limit is measured in, and the reason long documents fail.

• A token is a chunk of text – roughly 4 characters or 0.75 of an English word. Indian names, code and non-English text use more tokens than you expect.
• The context window is the total tokens of prompt plus answer the model can hold at once.
• Everything counts: system instructions, conversation history, retrieved documents and the answer.
• Cost is per token, and input and output are usually priced differently.
• Exceed the window and the oldest content is dropped or the call fails – which is why a long chat suddenly “forgets” the beginning.

def estimate_tokens(text: str) -> int:
    """Rough but good enough for a budget: about 4 characters per token."""
    return max(1, len(text) // 4)

policy = open("policy.txt", encoding="utf-8").read()
print(estimate_tokens(policy), "tokens just for the document")

# a quick monthly cost estimate before anyone builds anything
calls_per_day, in_tok, out_tok = 2000, 1500, 400
price_in, price_out = 3.0 / 1_000_000, 15.0 / 1_000_000      # dollars per token, check current pricing
daily = calls_per_day * (in_tok * price_in + out_tok * price_out)
print(f"about ${daily:.2f} a day, ${daily * 30:.0f} a month")

System, user and developer instructions

Prompts have layers, and the layers have different authority.

• The system prompt sets the role, rules and tone. It is set by your application, not by the end user.
• The user message is the request – and in a real product it contains untrusted input.
• Assistant messages are the model’s previous replies; sending them back is what creates memory.
• Rules that must hold always belong in the system prompt. Repeating them in every user message is a sign the design is wrong.
• Never trust the system prompt as a security boundary. It is instruction, not enforcement – see prompt injection in Module 9.

messages = [
    {"role": "system", "content":
        "You are a QA assistant for an insurance application. "
        "Answer only from the context provided. If the context does not contain the answer, "
        "reply exactly: NOT_IN_CONTEXT. Never invent a policy number."},
    {"role": "user", "content": "What is the claim limit for plan Gold?"},
]

Zero-shot, few-shot and structured prompting

The cheapest quality improvement available: show the model what good looks like.

• Zero-shot: just ask. Fine for simple, common tasks.
• Few-shot: include two or three worked examples in the prompt. Accuracy and format consistency jump immediately.
• Examples teach format as strongly as content – pick examples that match the edge cases you care about.
• Be specific about length, audience and what to leave out. “Write test cases” and “write 5 negative test cases for this field, one line each, no explanation” are different jobs.
• Put the long reference material before the instruction, and the instruction last.

Write negative test cases for the field below.

Example input:  Mobile number, required, 10 digits, Indian numbers only
Example output:
- empty value
- 9 digits
- 11 digits
- letters in the field
- leading zero
- +91 prefix included

Now do the same for: Policy number, required, format JSTA-00000, 5 digits after the dash

Chain-of-thought and task decomposition

Hard tasks get better when the model is allowed to work in steps – and when you stop asking for everything at once.

• Asking the model to work through a problem step by step improves multi-step reasoning noticeably.
• Modern models do much of this on their own; the bigger win now is decomposition in your own code.
• Split a big job into separate calls: extract the requirements, then generate cases, then review them. Each step is checkable.
• Smaller steps are cheaper to retry and far easier to test than one giant prompt.
• Do not show raw reasoning to end users – it is working, not an answer, and it can be wrong even when the answer is right.

Structured JSON output and schema validation

The step that turns a chat toy into something you can put in a pipeline.

• Ask for JSON and validate it. A model that returns JSON 97% of the time breaks your pipeline 3% of the time.
• Most APIs now support a response schema or JSON mode – use it, then still validate.
• Strip markdown fences before parsing; models like wrapping JSON in triple backticks.
• Retry once with the validation error appended to the prompt. That simple loop fixes most failures.

import json
from jsonschema import validate, ValidationError

SCHEMA = {
    "type": "object",
    "required": ["cases"],
    "properties": {"cases": {"type": "array", "minItems": 1, "items": {
        "type": "object",
        "required": ["id", "title", "steps", "expected", "type"],
        "properties": {
            "id": {"type": "string", "pattern": "^TC-[0-9]{3}$"},
            "title": {"type": "string", "minLength": 8},
            "steps": {"type": "array", "minItems": 1, "items": {"type": "string"}},
            "expected": {"type": "string", "minLength": 5},
            "type": {"enum": ["positive", "negative", "boundary"]},
        }}}},
}

def parse(raw: str):
    data = json.loads(raw)          # fails loudly if it is not JSON at all
    validate(data, SCHEMA)          # fails loudly if a field is missing or malformed
    return data["cases"]

try:
    cases = parse(model_output)
    print(len(cases), "valid cases")
except (json.JSONDecodeError, ValidationError) as err:
    print("retry with a stricter prompt:", err)

Hallucinations, grounding and factual verification

Confident, fluent and wrong. The defect type that is unique to this technology.

• A hallucination is output that is plausible and false. Fluency is not evidence of correctness.
• Grounding means supplying the source material and instructing the model to answer only from it.
• Require citations: every claim points at the chunk it came from. Then an automated check can verify the quote exists.
• Give the model a way out – an explicit “say NOT_IN_CONTEXT if it is not here” instruction – otherwise it will fill the gap.
• Verify anything checkable in code: numbers, dates, ids, totals. Do not ask the model to confirm its own answer.

Text, image, code and document generation

What generative models are genuinely good at, and where they stop.

• Text: summarising, rewriting, translating, drafting. Strongest when the source material is in the prompt.
• Code: boilerplate, test scaffolding, conversions, explaining unfamiliar code. Always review – the confident wrong answer compiles too.
• Images: illustrations and mock-ups. Not acceptable where accuracy matters, and licensing and likeness rules apply.
• Documents: extracting fields from PDFs and scans, drafting reports from structured data.
• Weak spots that have not gone away: arithmetic over many numbers, counting, very long documents, and anything needing information after the training cut-off.

Comparing hosted and open-weight models

A decision with cost, privacy and quality consequences – so it deserves evidence.

• Hosted: best quality, no servers, pay per token, data leaves your network, model can change under you.
• Open-weight, self-hosted: data stays inside, fixed infrastructure cost, you own the GPUs and the upgrades, quality usually a step behind.
• Compare on four axes: quality on your evaluation set, cost per 1,000 calls, latency at p95, and what the contract says about your data.
• Keep the model behind an interface in your code so swapping it is a configuration change, not a rewrite.
• Re-run the comparison when a new version ships. “It got better” is a claim, not a result.

Project: a prompt-based assistant for requirements, summaries and reports

1. Pick one real requirement. Write a zero-shot prompt for test cases and save the output.
2. Rewrite it with a system role, two worked examples and an explicit JSON schema. Save that output too.
3. Score both by hand: usable cases, wrong cases, missing scenarios. Write the three numbers down.
4. Add the validation code from this module so malformed output is caught and retried once.
5. Run the finished prompt on five different requirements and record where it was weakest. That weakness list is what Module 9 teaches you to measure automatically.