Data + AI practice labs
These labs follow the size and shape of small portfolio projects:
llm-data-profiling-toolsnowflake-ai-integrationai-data-quality-agent
Those examples set the level: small enough to finish, concrete enough to explain, and real enough for portfolio proof.
Each lab has a small version that avoids paid services and credentials. Do that first.
Use one lab as your course project, or run several after Lesson 3.4 as extra practice.
For career positioning, pair this page with Job pathways + AISOFT offerings.
For instructors, pair this page with Teaching agentic engineering. The labs are not prompt challenges. They are practice in briefing, planning, reviewing, proving, and handing off agentic work.
Coverage by work area
| Work area | Labs |
|---|---|
| SDLC + DevOps | Lab 14, Lab 25, Lab 26, Lab 27, Lab 30 |
| Data engineering | Lab 2, Lab 4, Lab 5, Lab 9 |
| Big data + streaming | Lab 16, Lab 17 |
| Analytics + BI | Lab 1, Lab 6, Lab 7 |
| Data quality + governance | Lab 3, Lab 8, Lab 9 |
| Data science | Lab 18, Lab 19 |
| ML + MLOps | Lab 20, Lab 21, Lab 22 |
| AI app engineering | Lab 11, Lab 23, Lab 24 |
| Backend/internal tools | Lab 10, Lab 11, Lab 22 |
| Agentic workflows | Lab 14, Lab 15, Lab 28, Lab 29, Lab 30 |
| Fresh-grad portfolio | Lab 12, Lab 13 |
Pathways by background
| Pathway | Best for | Start with | Then try |
|---|---|---|---|
| Data engineering | ETL, Snowflake, dbt, pipelines, platform work | Lab 2 · Snowflake AI integration | Lab 4 · Pipeline run explainer, Lab 5 · dbt test generator |
| Big data + streaming | Spark, Kafka, batch/stream jobs, lakehouse work | Lab 16 · Spark job tuning assistant | Lab 17 · Streaming data monitor |
| Data analytics | BI, reporting, SQL, dashboards, business analysis | Lab 1 · LLM data profiling tool | Lab 6 · KPI narrative analyst, Lab 7 · Dashboard QA assistant |
| Data quality + governance | QA, data governance, Collibra-style stewardship, lineage | Lab 3 · AI data quality agent | Lab 8 · PII policy scanner, Lab 9 · Data contract checker |
| Data science | notebooks, experiments, exploratory analysis, stakeholder findings | Lab 18 · Notebook insight reviewer | Lab 19 · Experiment report generator |
| ML + MLOps | model training, evaluation, deployment, monitoring | Lab 20 · Model eval harness | Lab 21 · Feature drift monitor, Lab 22 · ML inference API |
| AI app engineering | RAG, chat apps, tool calling, prompt/version control | Lab 23 · RAG answer evaluator | Lab 24 · Tool-calling assistant |
| SDLC + DevOps | requirements, tickets, CI, tests, releases, incidents | Lab 25 · Story-to-test planner | Lab 26 · CI failure explainer, Lab 27 · Release note generator |
| Agentic workflows | multi-agent work, memory, orchestration, handoff | Lab 28 · Multi-agent task board | Lab 29 · Agent memory curator, Lab 30 · Handoff packet generator |
| Backend/app engineering | APIs, CLIs, internal tools, services | Lab 10 · API log triage agent | Lab 11 · Support ticket routing service |
| Fresh-grad portfolio | New engineers who need concrete GitHub projects | Lab 12 · CSV cleaning assistant | Lab 13 · Resume project README improver |
| Team lead / manager | Standards, review gates, team adoption | Lab 14 · Agentic PR reviewer | Lab 15 · Team runbook generator |
Full versions can add Snowflake, LLM APIs, deployment, or alerts after the local behavior works.
Fresh graduate setup
Use this setup before Lab 12 or Lab 13 if you do not already have a repo.
mkdir fresh-grad-agentic-lab
cd fresh-grad-agentic-lab
git init
mkdir -p data output tests artifacts
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip pytest
cat > main.py <<'EOF'
import argparse
def main():
parser = argparse.ArgumentParser(description="Fresh graduate agentic lab")
parser.add_argument("--name", default="learner")
args = parser.parse_args()
print(f"hello, {args.name}")
if __name__ == "__main__":
main()
EOF
cat > tests/test_main.py <<'EOF'
import subprocess
import sys
def test_help_runs():
result = subprocess.run(
[sys.executable, "main.py", "--help"],
check=False,
capture_output=True,
text=True,
)
assert result.returncode == 0
assert "Fresh graduate agentic lab" in result.stdout
EOF
touch README.md
git status --short
If you are on Windows PowerShell, activate the environment with:
.venv\Scripts\Activate.ps1
Your first proof command can be simple:
python main.py --help
python -m pytest
After the first agent-assisted change, always run:
git status --short
git diff
Then centralize the lesson in AGENTS.md or CLAUDE.md so the next session starts with the command, folder layout, and rule the agent just learned.
Standard lab command block
Use this rhythm for every lab, regardless of pathway:
If git status, git diff, or python -m pytest is new to you, do Beginner prep first. The labs assume you can open a terminal, work in a repo, and run one Python or Node command.
# create or open the lab repo
mkdir agentic-lab
cd agentic-lab
git init
# start your selected agent
claude
# or: codex
# or: gemini
# after each slice, inspect and verify
git status --short
git diff
python -m pytest
# or: npm test
# or: make test
# capture proof only after the check passes
git add .
git commit -m "feat: complete lab slice"
For data labs, also keep one command that proves the artifact works on a fixture:
python main.py --input data/sample.csv --out outputs/result.md
ls -lh outputs/
cat outputs/result.md
Standard lab artifact package
Every lab should leave a small package in the repo:
lab-artifacts/
├── brief.md
├── plan.md
├── proof.md
├── decisions.md
└── next-slice.md
The learner writes these in plain English:
brief.md: goal, constraints, inputs, outputs, done check.plan.md: first slice only, with files and checks.proof.md: command output, screenshot notes, test results, or generated files.decisions.md: what changed and why.next-slice.md: the next instruction a future agent should receive.
Quick chooser
If someone says:
- “I know SQL but not much Python”. Start with Lab 6.
- “I work in Snowflake”. Start with Lab 2.
- “I do data quality or governance”. Start with Lab 3 or Lab 8.
- “I work with Spark, Kafka, or lakehouse jobs”. Start with Lab 16 or Lab 17.
- “I do data science or notebooks”. Start with Lab 18.
- “I train or deploy models”. Start with Lab 20 or Lab 22.
- “I am building RAG or AI apps”. Start with Lab 23 or Lab 24.
- “I work across SDLC, CI, releases, or incidents”. Start with Lab 25, Lab 26, or Lab 27.
- “I want agentic workflow practice”. Start with Lab 28 or Lab 30.
- “I am a fresh graduate”. Start with Lab 12.
- “I build APIs”. Start with Lab 10.
- “I manage a team adopting AI”. Start with Lab 14 or Lab 15.
Lab 1 · LLM data profiling tool
Goal: Given table metadata and a few sample rows, produce a data dictionary and suggested validation rules.
Who it is for: data analysts, data engineers, QA analysts, fresh graduates who know Python basics.
Small version: Use a local CSV instead of Snowflake.
Full version: Connect to Snowflake and profile a real table.
Suggested structure
llm-data-profiling-tool/
├── main.py
├── data/
│ └── customer_transactions.csv
├── profiler/
│ ├── schema_extractor.py
│ └── sample_reader.py
├── llm/
│ ├── prompt_templates.py
│ └── profiler_client.py
├── output/
│ ├── dictionary_writer.py
│ └── rule_exporter.py
└── tests/
└── test_rule_exporter.py
Step-by-step
- Create a tiny CSV with 8-10 rows and obvious issues: nulls, invalid dates, negative amounts, duplicate IDs.
- Ask the agent to inspect the CSV and propose a project plan.
- Build
sample_reader.pyto load rows and infer basic types. - Build
schema_extractor.pyto summarize columns, null rates, examples, and suspicious values. - Write one prompt template that asks an LLM to explain business meaning and propose validation rules.
- Add an offline mode that returns deterministic sample rules without calling an LLM.
- Export a Markdown data dictionary.
- Export validation rules as YAML or JSON.
- Add a test for one rule export.
- Run the tool and save the generated output.
Practice proof
python main.py --csv data/customer_transactions.csv --output output/profile.md
python main.py --csv data/customer_transactions.csv --rules output/rules.yaml
Done means:
- the Markdown dictionary exists,
- the rules file exists,
- at least one test passes,
- the output flags at least one real issue in the sample data.
Lab 2 · Snowflake AI integration
Goal: Run a query, summarize results, classify text rows, or narrate a trend.
Who it is for: data engineers, analytics engineers, Snowflake users, experienced engineers learning governed AI workflows.
Small version: Use local CSV fixtures.
Full version: Use Snowflake with approved credentials.
Suggested structure
snowflake-ai-integration/
├── main.py
├── config/
│ └── config.example.yaml
├── connectors/
│ ├── local_client.py
│ └── snowflake_client.py
├── pipelines/
│ ├── insight_generator.py
│ ├── classifier.py
│ └── trend_narrator.py
├── prompts/
│ └── prompt_templates.py
└── tests/
└── test_classifier.py
Step-by-step
- Start with local CSV mode so the project works without credentials.
- Add a CLI with three commands:
insight,classify, andtrend. - Implement
insightagainst a local CSV: summarize row count, columns, and notable values. - Implement
classify: assign each text row to one of the supplied labels. - Implement
trend: group a date/value series and narrate direction. - Add prompt templates for the LLM version.
- Add Snowflake connection code behind a config flag.
- Add dry-run mode that prints the query and planned action before running it.
- Add one test for label validation or trend calculation.
- Write a README with safe credential setup.
Practice proof
python main.py insight --csv data/kpi_daily.csv --context "Daily KPIs"
python main.py classify --csv data/support_tickets.csv --text-field description --labels billing technical account
python main.py trend --csv data/orders.csv --date-col order_date --value-col revenue --metric "Daily Revenue"
Done means:
- all three commands run on local fixtures,
- Snowflake credentials are never committed,
- the tool has dry-run mode,
- at least one behavior has a test.
Lab 3 · AI data quality agent
Goal: Detect data quality issues, classify impact, and generate a human-readable scorecard.
Who it is for: QA analysts, data quality analysts, data engineers, governance teams.
Small version: Run checks on a local CSV.
Full version: Add Snowflake and alert routing.
Suggested structure
ai-data-quality-agent/
├── main.py
├── config/
│ └── config.example.yaml
├── profiler/
│ ├── schema_validator.py
│ ├── statistical_checks.py
│ └── lineage_mapper.py
├── agent/
│ ├── impact_classifier.py
│ └── prompt_templates.py
├── output/
│ ├── scorecard_generator.py
│ └── alert_router.py
└── tests/
└── test_statistical_checks.py
Step-by-step
- Create a CSV with expected schema and a second CSV with broken rows.
- Build schema checks: missing column, new column, wrong type.
- Build statistical checks: null spike, duplicate IDs, out-of-range values.
- Build an issue object with
name,severity,evidence, andsuggested_owner. - Add an impact classifier: Low, Medium, High.
- Generate a Markdown scorecard.
- Add alert routing as a stub that writes to
output/alerts.json. - Add one test for each check family.
- Ask the agent to run a no-slop pass.
- Ship the result as a repo with sample output.
Practice proof
python main.py --csv data/orders_bad.csv --expected-schema data/orders_schema.yaml --out output/scorecard.md
Done means:
- the scorecard lists each detected issue,
- every issue includes evidence,
- at least one high-impact issue is classified correctly,
- the test suite catches a broken check.
Lab 4 · Pipeline run explainer
Pathway: Data engineering
Goal: Turn raw pipeline logs into a readable incident summary with failed step, likely cause, and next action.
Small version: Use sample log files in data/logs/.
Full version: Pull logs from Airflow, Dagster, dbt Cloud, GitHub Actions, or Snowflake task history.
Suggested structure
pipeline-run-explainer/
├── main.py
├── data/logs/
├── parser/
│ └── log_parser.py
├── analyzer/
│ ├── failure_classifier.py
│ └── summary_writer.py
└── tests/
└── test_log_parser.py
Step-by-step
- Collect three fake logs: success, SQL failure, timeout.
- Parse timestamps, step names, status, and error blocks.
- Classify failures into
sql_error,dependency_down,timeout, orunknown. - Generate a Markdown incident summary.
- Add a
--sinceor--run-idoption. - Add one test for log parsing.
Practice proof
python main.py --log data/logs/failed_sql.log --out output/incident.md
Done means the incident summary names the failed step, evidence, likely cause, and next action.
Lab 5 · dbt test generator
Pathway: Data engineering
Goal: Read a dbt model SQL file and propose useful schema.yml tests.
Small version: Parse local .sql files.
Full version: Inspect a dbt project and write candidate tests into a review file.
Step-by-step
- Add two sample dbt model SQL files.
- Extract selected columns and simple transformations.
- Infer candidate tests:
not_null,unique,accepted_values, relationships. - Write proposed YAML to
output/proposed_schema.yml. - Add a review note explaining why each test was proposed.
- Add one test for YAML generation.
Practice proof
python main.py --model models/fct_orders.sql --out output/proposed_schema.yml
Done means the generated YAML is valid and every proposed test has a reason.
Lab 6 · KPI narrative analyst
Pathway: Data analytics
Goal: Turn a weekly KPI CSV into a plain-English business summary.
Small version: Use local CSVs.
Full version: Connect to a BI export or Snowflake query.
Step-by-step
- Create a KPI CSV with date, metric, segment, value.
- Compute week-over-week change.
- Flag biggest movers.
- Generate a short executive summary.
- Generate a second analyst note with caveats.
- Add a test for percent-change calculation.
Practice proof
python main.py --csv data/weekly_kpis.csv --metric revenue --out output/kpi_summary.md
Done means the summary includes top movement, segment, numeric evidence, and caveat.
Lab 7 · Dashboard QA assistant
Pathway: Data analytics
Goal: Compare dashboard numbers against source extracts and flag mismatches.
Small version: Compare two CSVs: source.csv and dashboard.csv.
Full version: Connect to BI export, semantic layer, or warehouse query.
Step-by-step
- Create source and dashboard CSVs with matching metric names.
- Join by metric/date/segment.
- Compute absolute and percentage difference.
- Flag mismatches above tolerance.
- Generate a QA report.
- Add a test for tolerance behavior.
Practice proof
python main.py --source data/source.csv --dashboard data/dashboard.csv --tolerance 0.01 --out output/dashboard_qa.md
Done means the report separates pass, warning, and fail metrics.
Lab 8 · PII policy scanner
Pathway: Data quality + governance
Goal: Scan column names and sample values for possible PII, then generate a stewardship review file.
Small version: Use CSV headers and sample rows.
Full version: Connect to Snowflake information schema or a catalog export.
Step-by-step
- Create a CSV with fields like email, phone, customer_name, notes.
- Add pattern checks for email, phone, SSN-like values.
- Add name-based checks for sensitive columns.
- Classify risk as Low, Medium, High.
- Generate a stewardship review Markdown file.
- Add allowlist/false-positive config.
Practice proof
python main.py --csv data/customers.csv --out output/pii_review.md
Done means every flagged field has evidence and a suggested handling policy.
Lab 9 · Data contract checker
Pathway: Data quality + governance
Goal: Compare an incoming file/table against a declared contract.
Small version: YAML contract plus CSV file.
Full version: Validate warehouse tables before pipeline runs.
Step-by-step
- Write a YAML contract: columns, types, required fields, allowed values.
- Load an incoming CSV.
- Check missing columns, extra columns, nulls, and invalid values.
- Produce a pass/fail report.
- Add an exit code:
0for pass, non-zero for fail. - Add tests for one passing and one failing file.
Practice proof
python main.py --contract contracts/orders.yaml --csv data/orders_incoming.csv
Done means the checker can block a bad file before it reaches a pipeline.
Lab 10 · API log triage agent
Pathway: Backend/app engineering
Goal: Summarize API logs and group failures by endpoint, status code, and likely cause.
Small version: Use local JSONL logs.
Full version: Pull logs from CloudWatch, Datadog, GCP Logging, or app files.
Step-by-step
- Create sample JSONL logs with status, route, latency, message.
- Group errors by route and status.
- Detect latency spikes.
- Generate a triage report.
- Add a suggested owner field based on route prefix.
- Add one test for grouping.
Practice proof
python main.py --logs data/api_logs.jsonl --out output/triage.md
Done means the report names top failing routes and gives evidence.
Lab 11 · Support ticket routing service
Pathway: Backend/app engineering
Goal: Classify support tickets and route them to the right queue.
Small version: CLI reads CSV and writes routed CSV.
Full version: Add a small FastAPI endpoint.
Step-by-step
- Create a ticket CSV with subject, description, customer tier.
- Define route labels: billing, technical, account, bug, feature.
- Build deterministic keyword routing first.
- Add optional LLM routing behind a flag.
- Write output CSV with label and confidence.
- Add a test for at least three ticket examples.
Practice proof
python main.py --tickets data/tickets.csv --out output/routed_tickets.csv
Done means every ticket has a route, reason, and confidence.
Lab 12 · CSV cleaning assistant
Pathway: Fresh-grad portfolio
Goal: Build a friendly CLI that cleans a messy CSV and writes a cleaned file plus report.
Small version: Local CSV only.
Full version: Add a simple web UI.
Step-by-step
- Create a messy CSV: extra spaces, mixed casing, bad dates, duplicate rows.
- Trim strings and normalize column names.
- Parse dates and report failed rows.
- Remove duplicates.
- Write
clean.csvandcleaning_report.md. - Add one test for duplicate removal.
Practice proof
python main.py --csv data/messy_customers.csv --out output/clean_customers.csv --report output/cleaning_report.md
Done means the cleaned file and report are both created.
Lab 13 · Resume project README improver
Pathway: Fresh-grad portfolio
Goal: Turn a rough project README into a stronger portfolio README with setup, demo, screenshots, and proof.
Small version: Markdown in, Markdown out.
Full version: Add GitHub repo inspection.
Step-by-step
- Create a rough README.
- Parse existing headings.
- Detect missing sections: setup, usage, proof, limitations.
- Generate an improved README draft.
- Add a checklist of what still needs human input.
- Add one test for missing-section detection.
Practice proof
python main.py --readme README_rough.md --out README_improved.md
Done means the improved README is clearer but does not invent fake claims.
Lab 14 · Agentic PR reviewer
Pathway: Team lead / manager
Goal: Review a diff against team standards and produce actionable findings.
Small version: Read a saved .diff file.
Full version: Pull PR diff from GitHub.
Step-by-step
- Create or save a small diff file.
- Define review categories: bug risk, missing tests, scope creep, unclear naming.
- Parse changed files.
- Produce review findings with file, line, severity, and suggestion.
- Add a no-findings path.
- Add one test for detecting TODO/fake done.
Practice proof
python main.py --diff data/sample.diff --out output/review.md
Done means findings are specific, grounded, and not generic advice.
Lab 15 · Team runbook generator
Pathway: Team lead / manager
Goal: Convert scattered notes into a repeatable runbook for a recurring engineering task.
Small version: Local notes folder.
Full version: Pull from wiki/docs and create a PR.
Step-by-step
- Create three messy notes about deploy, rollback, and verification.
- Extract steps, commands, owners, and warnings.
- Generate a clean runbook.
- Add a verification checklist.
- Add a “when to stop and ask” section.
- Add one test that required sections exist.
Practice proof
python main.py --notes data/deploy_notes/ --out output/deploy_runbook.md
Done means the runbook is usable by someone who did not write the notes.
Lab 16 · Spark job tuning assistant
Pathway: Big data + streaming
Goal: Read Spark job metrics and explain likely bottlenecks, skew, shuffle cost, and tuning options.
Small version: Use sample Spark event summaries as JSON.
Full version: Parse Spark event logs or Databricks job exports.
Step-by-step
- Create three sample job summaries: healthy, skewed join, excessive shuffle.
- Parse stage duration, input rows, shuffle read/write, spill, and task skew.
- Detect likely bottlenecks with deterministic rules first.
- Generate a tuning report with evidence and safe next experiments.
- Add a confidence field and “needs human review” flag.
- Add one test for skew detection.
Practice proof
python main.py --metrics data/spark_job_skew.json --out output/spark_tuning.md
Done means the report names the bottleneck, cites metrics, and proposes one safe experiment.
Lab 17 · Streaming data monitor
Pathway: Big data + streaming
Goal: Detect lag, schema changes, and bad event spikes in a simulated stream.
Small version: Read timestamped JSONL events from a local folder.
Full version: Connect to Kafka, Kinesis, Pub/Sub, or Snowflake streams.
Step-by-step
- Create JSONL files for normal events, delayed events, and schema drift.
- Track event time versus processing time.
- Detect missing required fields and new unexpected fields.
- Generate an alert summary with severity and evidence.
- Add a replay command for the affected time window.
- Add tests for lag and schema drift.
Practice proof
python main.py --events data/stream/ --contract contracts/events.yaml --out output/stream_monitor.md
Done means lag, schema drift, and bad event spikes are separated in the report.
Lab 18 · Notebook insight reviewer
Pathway: Data science
Goal: Review a notebook export for unclear assumptions, weak charts, missing caveats, and unsupported conclusions.
Small version: Read a Markdown or .ipynb export.
Full version: Inspect a notebook, generated figures, and source data profile.
Step-by-step
- Create a small notebook export with a chart, conclusion, and caveat.
- Parse headings, code cells, chart captions, and markdown conclusions.
- Check whether each conclusion has numeric evidence nearby.
- Flag missing caveats, unclear filters, and chart-label issues.
- Generate a review report.
- Add one test for unsupported conclusion detection.
Practice proof
python main.py --notebook notebooks/customer_churn.md --out output/notebook_review.md
Done means the review separates evidence issues from style suggestions.
Lab 19 · Experiment report generator
Pathway: Data science
Goal: Turn experiment metrics into a stakeholder-ready report with winner, tradeoffs, and next experiment.
Small version: Use local CSV metrics from A/B or model experiments.
Full version: Pull from MLflow, W&B, Optuna, or warehouse tables.
Step-by-step
- Create an experiment CSV with variant, metric, segment, and confidence columns.
- Compute winner by primary metric.
- Identify segments where the winner is weaker.
- Generate a short report with decision, risks, and next experiment.
- Add a “not enough evidence” path.
- Add tests for winner and no-winner cases.
Practice proof
python main.py --metrics data/experiment_results.csv --primary conversion_rate --out output/experiment_report.md
Done means the report makes a decision only when the evidence supports it.
Lab 20 · Model eval harness
Pathway: ML + MLOps
Goal: Build a repeatable eval harness for a classifier, extractor, or LLM response task.
Small version: Evaluate canned predictions against a labeled CSV.
Full version: Run a model or LLM client and write versioned eval results.
Step-by-step
- Create a labeled dataset with input, expected output, and difficulty.
- Write exact-match and rubric-style scoring functions.
- Add per-category metrics.
- Save results with model name, prompt version, and timestamp.
- Generate an eval summary with failure examples.
- Add tests for scoring behavior.
Practice proof
python main.py --cases data/eval_cases.csv --predictions data/predictions.csv --out output/eval_report.md
Done means the eval can compare two model or prompt versions without changing code.
Lab 21 · Feature drift monitor
Pathway: ML + MLOps
Goal: Compare training and production feature distributions and flag drift.
Small version: Compare two CSV snapshots.
Full version: Read from feature store, warehouse, or model monitoring export.
Step-by-step
- Create training and production feature CSVs.
- Compute null rate, mean/median, category frequency, and range changes.
- Flag features above drift thresholds.
- Generate a model-owner report with severity and suggested action.
- Add config for thresholds.
- Add tests for numeric and categorical drift.
Practice proof
python main.py --train data/train_features.csv --prod data/prod_features.csv --out output/drift_report.md
Done means drift findings include evidence and avoid claiming model impact without proof.
Lab 22 · ML inference API
Pathway: ML + MLOps
Goal: Wrap a simple model behind an API with validation, logging, and a testable prediction contract.
Small version: FastAPI app with a fake or scikit-learn model.
Full version: Add Docker, model version metadata, and deployment checks.
Step-by-step
- Define request and response schemas.
- Add input validation and clear error messages.
- Load a model artifact or deterministic stub.
- Return prediction, confidence, and model version.
- Log request metadata without sensitive fields.
- Add API tests for success and validation failure.
Practice proof
uvicorn app.main:app --reload
pytest
Done means the API has a stable contract and tests protect it.
Lab 23 · RAG answer evaluator
Pathway: AI app engineering
Goal: Evaluate retrieved context and answer quality for a small RAG system.
Small version: Use local Markdown docs and canned answers.
Full version: Connect to a vector database and run live retrieval.
Step-by-step
- Create five small documents and ten questions.
- Store expected source document IDs for each question.
- Evaluate retrieval hit rate.
- Score answers for groundedness and citation coverage.
- Generate a failure report with examples.
- Add tests for citation extraction.
Practice proof
python main.py --questions data/questions.csv --answers data/answers.csv --out output/rag_eval.md
Done means the report separates retrieval failures from answer-generation failures.
Lab 24 · Tool-calling assistant
Pathway: AI app engineering
Goal: Build an assistant that chooses between safe tools and records why each tool was called.
Small version: Local tools for calculator, CSV lookup, and note search.
Full version: Add authenticated business tools behind explicit approval.
Step-by-step
- Define three local tool schemas.
- Create user requests that require zero, one, and multiple tools.
- Add a router that selects the tool and validates arguments.
- Log tool call, reason, result, and user-visible summary.
- Add a refusal path for unsupported or risky requests.
- Add tests for routing and argument validation.
Practice proof
python main.py --request "Look up customer C-100 and summarize open balance"
Done means the assistant uses tools only when needed and leaves an audit trail.
Lab 25 · Story-to-test planner
Pathway: SDLC + DevOps
Goal: Turn a product story into acceptance criteria and test cases before implementation.
Small version: Markdown story in, test plan out.
Full version: Pull tickets from Jira, Linear, GitHub Issues, or Azure DevOps.
Step-by-step
- Write three sample stories: clear, vague, and risky.
- Extract user, goal, constraints, and missing information.
- Generate acceptance criteria.
- Generate unit, integration, and manual test ideas.
- Add a “questions before build” section.
- Add tests for missing-information detection.
Practice proof
python main.py --story data/story.md --out output/test_plan.md
Done means vague stories produce questions instead of invented requirements.
Lab 26 · CI failure explainer
Pathway: SDLC + DevOps
Goal: Summarize CI logs into failure cause, affected test, likely owner, and next command.
Small version: Use saved GitHub Actions logs.
Full version: Pull logs from GitHub Actions, GitLab CI, Jenkins, or Buildkite.
Step-by-step
- Save logs for lint failure, test failure, dependency failure, and timeout.
- Parse command, exit code, failing file, and error block.
- Classify failure type.
- Generate a concise fix summary.
- Add next command to reproduce locally.
- Add tests for log parsing.
Practice proof
python main.py --log data/ci_failed_test.log --out output/ci_summary.md
Done means a developer can reproduce the failure from the summary.
Lab 27 · Release note generator
Pathway: SDLC + DevOps
Goal: Turn commits, PR titles, or issue notes into release notes with risk and verification sections.
Small version: Read a local changelog input file.
Full version: Pull merged PRs from GitHub and draft a release PR.
Step-by-step
- Create sample PR titles and commit messages.
- Group changes by feature, fix, internal, and docs.
- Identify migration or rollout notes.
- Generate user-facing and internal release notes.
- Add verification evidence placeholders.
- Add tests for grouping behavior.
Practice proof
python main.py --changes data/merged_prs.md --out output/release_notes.md
Done means the notes are useful without overstating what shipped.
Lab 28 · Multi-agent task board
Pathway: Agentic workflows
Goal: Break a project into agent-sized tasks with dependencies, owners, and review gates.
Small version: Markdown brief in, task board out.
Full version: Create GitHub issues or Linear tickets with labels.
Step-by-step
- Write a project brief with three independent workstreams.
- Extract deliverables, dependencies, and risks.
- Split work into tasks suitable for parallel agents.
- Add review gate and proof requirement for each task.
- Generate a Markdown board.
- Add a validation check for missing proof.
Practice proof
python main.py --brief data/project_brief.md --out output/task_board.md
Done means tasks can be claimed independently without losing the overall goal.
Lab 29 · Agent memory curator
Pathway: Agentic workflows
Goal: Convert scattered project notes into concise agent memory files.
Small version: Local notes folder to AGENTS.md plus topic files.
Full version: Pull from docs, issues, and previous handoffs.
Step-by-step
- Create messy notes about architecture, commands, decisions, and open questions.
- Extract durable facts from temporary chatter.
- Write
AGENTS.mdwith commands, standards, and boundaries. - Write separate notes for architecture and deployment.
- Add a stale-fact warning section.
- Add a check that required sections exist.
Practice proof
python main.py --notes data/project_notes/ --out output/memory/
Done means a new agent can start work without re-discovering basic project facts.
Lab 30 · Handoff packet generator
Pathway: Agentic workflows
Goal: Create a handoff packet from git status, recent commits, open tasks, and verification notes.
Small version: Read local text fixtures.
Full version: Inspect a real repo and open PR.
Step-by-step
- Create fixtures for git status, commit log, TODOs, and test output.
- Summarize what changed.
- Separate done, in progress, blocked, and next actions.
- Include verification commands and results.
- Add risks and files touched.
- Add tests for section completeness.
Practice proof
python main.py --repo-fixture data/repo_state/ --out output/handoff.md
Done means another person or agent can continue without guessing the state.
How to use these with the 32 lessons
| Course point | What to do with a lab |
|---|---|
| Lesson 2.1 | Pick two lab candidates and write done checks. |
| Lesson 3.1 | Choose one lab and cut scope to the small version. |
| Lesson 3.2 | Ask for a plan for the first slice only. |
| Lesson 3.3 | Build one module at a time. |
| Lesson 3.4 | Ship local proof: command output, sample files, README. |
| Lesson 4.2 | Add tests/checks for the critical behavior. |
| Lesson 4.3 | Add project instructions and safe credential rules. |
| Lesson 4.4 | Run no-slop against generated code. |
| Lesson 5.2 | Attach proof before saying shipped. |
| Lesson 6.3 | Add handoff notes and team conventions. |
| Lesson 6.9 | Add cost and observability notes where the lab has API, model, or data runs. |
| Lesson 6.10 | Add data boundary and credential safety notes. |
| Lesson 6.11 | Package the lab so another learner can run it. |
Instructor notes
For fresh graduates, start with local CSV mode. Snowflake and LLM credentials add too many failure modes for the first pass.
For experienced engineers, keep the same lab but raise the bar:
- config validation,
- dry-run mode,
- tests,
- structured output,
- credential safety,
- README with reproducible commands.
For live workshops, choose the shared room exercise by audience:
- mixed or beginner cohort: Lab 1 or Lab 12,
- engineering leaders: Lab 14, Lab 25, or Lab 30,
- data platform cohort: Lab 4, Lab 9, or Lab 16,
- AI product cohort: Lab 20, Lab 23, or Lab 24.
Use the smallest version first. The room should see one complete pass from brief to proof before splitting into deeper pathway work.