Skip to content

Mock Tasks and Timed Workflows

What This Is

A mock task is a packet that simulates the shape of a real AI competition or production sprint: a problem statement, a data card, an evaluation spec, and a time budget. The point of running mocks is not entertainment. It is to rehearse the decisions you will need to make fast under real constraints — baseline first, which split, which metric, when to stop, what to submit.

This topic gives you the packet format and two fully worked packets to practice against. It is the artifact-producing half of the IOAI Competition Surface — the competition surface teaches the habits, the mock packets let you exercise them.

When You Use It

  • you are preparing for IOAI, a Kaggle comp, or a production take-home
  • you can execute on a narrow task but freeze when the clock starts
  • you want to calibrate how fast you actually are on an unfamiliar problem
  • you need a repeatable drill that exposes decision gaps, not coding gaps

Mock Packet Contract

Every mock packet, whether you build one or run someone else's, has the same four parts:

section what it contains
problem one paragraph; names the task, the deployment story, and the scoring metric
data card sample counts, fields, known weak slices, known leakage traps
evaluation the single headline metric, the validation split, submission format and cap
time budget total clock; suggested phase allocation; when to commit the first submission

If a packet is missing any of these four, the first job is to infer it. The inference itself is an IOAI muscle — grader packets are routinely under-specified.

The Timed Workflow Skeleton

A 90-minute packet usually breaks down like this:

  0–10   Read problem, data card, eval spec. Decide on split and metric.
 10–25   Inspect data. Build a sklearn-level baseline. Submit it.
 25–55   One real modeling iteration. Validate on the chosen split.
 55–75   One more iteration OR one ablation that defends the current best.
 75–85   Error analysis on the worst slice. Decide ship vs. iterate.
 85–90   Final submission. Write the 4-sentence decision note.

A 180-minute packet doubles the middle two phases. Packets longer than that are rarely time-bound — the decision becomes about stopping, not starting.

Phase Discipline

Three rules that separate a working mock from a useless one:

  1. Submit a baseline inside the first quarter of the clock. The baseline sets a floor and confirms the scoring pipeline works. Teams that skip this lose an hour to "my code almost works" and submit nothing.
  2. Validate on the split that matches the deployment story, not the split that gives the best number. See Splitter Choice Under Ambiguity.
  3. Every iteration must move a measured number. If the last change did not move validation, do not add another change. Back out and re-diagnose.

Packet A — Tabular Churn Classification

A fully worked packet. Run against it before reading the reference move below.

Problem

A subscription business wants a model that flags customers likely to cancel in the next 30 days. Predictions are used by a retention team whose review budget is 100 outreach calls per day.

Data Card

  • 120,000 rows, one row per customer-month
  • fields: customer_id, signup_date, plan, billing_failures_30d, support_tickets_30d, sessions_30d, avg_session_minutes_30d, months_active, churn_next_30d (label)
  • timespan: 24 months of history
  • known weak slice: customers with months_active < 2 (high natural churn, sparse behavior)
  • known leakage trap: support_tickets_30d includes tickets filed after a cancellation is queued

Evaluation

  • metric: precision@100 per day (top-100 highest-score predictions, measured against actual cancellations)
  • split: last 3 months of data held out as test; submit predictions on the final month only
  • submission: CSV of customer_id, score; single submission per day

Time Budget

90 minutes.

Reference Move (Read After Your Own Attempt)

Decision trail that defends a strong mock run:

  • the split is time-ordered. Any CV inside training data must also be time-ordered. Random KFold is not valid here.
  • the leakage trap on support_tickets_30d must be neutralized: either drop the field, or construct a lag-1 version using only tickets from the prior month.
  • the eval is precision@100, not accuracy or AUC. Ranking quality matters; calibration does not. Logistic regression + LightGBM both work; pick by validation precision@100.
  • months_active < 2 is the named weak slice. Report precision@100 on that slice separately. If the slice is near zero, the model is useless on new customers — disclose that in the decision note.
  • a one-hour baseline that earns the ship decision: LightGBM on engineered features (billing-failure rate trend, session-decay ratio over last 3 months, ticket lag-1), time-ordered 3-fold CV, early stopping on precision@100.

Four-Sentence Decision Note Template

Model: {which family, why}. Validation split: {what split, why it matches deployment}. Weak slice check: {which slice, what number you got, whether you recommend shipping anyway}. What would change my mind: {one concrete next evidence}.

Packet B — Vision Fine-Grained Classification Under Shift

A harder packet that forces a PEFT vs. full fine-tune vs. prompt-only-adapter decision.

Problem

Classify 20 subspecies of moth from camera-trap images. The training set was collected at night with IR illumination; the test set is a mix of day and night with visible-spectrum photos. A field biologist will review up to 50 low-confidence predictions per week.

Data Card

  • 12,000 training images, 5 images to 1200 per class (long-tailed)
  • 2,000 held-out test images with a 60/40 day/night split
  • labels include the class and an imaging_mode ∈ {IR_night, visible_day} field
  • known weak slice: the rarest 5 classes (<50 training images each)
  • known leakage trap: some images were taken sequentially by a trap and contain the same moth from different angles — trap_id leak if not split by trap

Evaluation

  • metric: macro-F1 across all 20 classes, plus a separate reported macro-F1 on the visible_day slice
  • split: group-by-trap_id, time-ordered, last three months of traps held out as test
  • submission: CSV of image_id, predicted_class
  • compute cap: a single T4 GPU for 90 minutes total wall time, including training and inference

Time Budget

90 minutes, compute cap included.

Reference Move (Read After Your Own Attempt)

  • the imaging-mode shift is the real task. A model trained only on night/IR images that tests on day/visible images will fail, regardless of architecture. Day images should be upsampled or augmented into the training distribution.
  • full fine-tuning a ResNet-50 or ViT-B under a 90-minute T4 budget is possible but tight. LoRA on a pretrained encoder (see PEFT and LoRA) is the recipe that fits the compute cap with room to spare.
  • the weak slice is the rarest 5 classes and the visible_day slice. Macro-F1 weights classes equally, so ignoring rare classes is expensive; a class-balanced sampler or focal loss is justified.
  • augmentation shape matters more than augmentation strength — rotations and brightness shifts simulate the day/visible shift; random erasing and cutout do not. See Augmentation Choice.
  • a working fast recipe: LoRA (r=8) on a pretrained DINOv2 ViT-B, class-balanced sampler, colour-jitter augmentation with a strong brightness component, 3 epochs, report macro-F1 overall and on the day slice.

Four-Sentence Decision Note Template

Adaptation depth: {frozen / LoRA / full fine-tune, why}. Augmentation shape: {which augmentations, which shift they simulate}. Rare-class handling: {loss or sampler change, why}. What would change my mind: {e.g., "day-slice macro-F1 < 0.3 means the augmentation did not close the shift"}.

What To Inspect While The Clock Runs

  • time-of-first-submission — if it is past the quarter mark, your baseline is too ambitious
  • metric on the actual deployment split — not on the one that was easy to build
  • weak-slice number — headline metric without slice numbers is not a decision
  • compute used vs. compute cap — wall time is a constraint, not a suggestion
  • decision note length — four sentences; more than that means you are explaining, not deciding

Failure Pattern

The common failure is optimization drift: the clock is running, you iterate on the model, but you never ran the eval pipeline on a real submission, so when you finally try, the format is wrong and you submit nothing. The fix is rule 1 above — baseline submission first, always, even if it is worse than random.

The second failure is metric amnesia: teams optimize the thing they usually optimize (accuracy, AUC) instead of the thing the grader scores. A 45-minute tune on the wrong metric is indistinguishable from a wasted hour. Re-read the eval spec at the 30-minute mark.

Common Mistakes

  • reading the problem once and starting to code — re-read at minute 10 and minute 30
  • picking the split by which one looks best — pick the deployment-shaped split and report its number
  • writing three features, one new model, two new regularizers in a single iteration — one move per iteration
  • skipping error analysis because the clock is tight — five minutes of error analysis is worth thirty minutes of blind retries
  • over-investing in a flashy architecture before a sklearn baseline exists
  • trying to fit into the compute cap by cutting epochs instead of cutting model size or batch size

Building Your Own Packet

Useful drill: write a mock packet for a teammate, then run their packet. The skill of authoring a packet sharpens your eye on the packet contract more than any single run.

A packet is strong if:

  • the problem statement mentions a deployment story that constrains the split
  • the data card names at least one weak slice and one leakage trap
  • the evaluation is a single metric with an explicit review budget or top-k constraint
  • the time budget is tight enough that the student must stop, not finish

Practice

  1. Run Packet A start to finish in 90 minutes. Commit a submission, then commit the four-sentence note.
  2. Run Packet B start to finish in 90 minutes. Defend the adaptation-depth decision against a peer.
  3. Rerun Packet A with the time budget halved (45 minutes). You will have to cut real scope. Decide what to cut before the clock starts.
  4. Author a third mock packet in a domain you know (audio, text, tabular). Have a peer run it. Compare their decision note to your solution.
  5. After each packet, fill a what-I-cut and what-I-got-right list. The patterns across four packets are more useful than any single run.

Runnable Example

Mock packets are workflow drills. The runnable work is whatever pipeline you build while the clock ticks — there is no single "run this in the browser" artifact. The Mock Tasks and Timed Workflows track is the matching capstone.

Longer Connection

This topic is the operational half of IOAI Competition Surface. Before a packet:

Mid-packet:

After a packet: