# Create Tasks – Context for New Chats

## What This Is

Two ways to create tasks from sjreport data:

1. **CreateTasks** (original) – Single Lambda: fetch sjreport → transform → **in-process** batch upsert (1k/batch) → update_store_issue_counts → invoke StartProcessQueue → success email. Use for normal/small runs. **Times out at ~10k tasks (15 min limit).**

2. **CreateTasksBulk + Step Function** – For 34k+ tasks: **You invoke the CreateTasksBulk Lambda first.** It fetches sjreport, transforms, writes batches to S3, then **starts** the Step Function. The state machine does **not** run CreateTasksBulk as a step; CreateTasksBulk is the **trigger** that starts the state machine with the batch list.

---

## Flow (Bulk Path)

**Step 1 – You invoke CreateTasksBulk (Lambda), not the state machine:**

- Invoke **CreateTasksBulk** with `{"week_start_date":"YYYY-MM-DD"}` (e.g. from AWS Lambda console or an API).
- CreateTasksBulk: fetches sjreport → transforms to task rows → uploads batches (5k rows/file) to S3 → **starts the Step Function** with `{ items: [{ s3_bucket, s3_key, week_start_date }, ...], week_start_date }`.

**Step 2 – The Step Function runs (started by CreateTasksBulk):**

- **PrepareBatches** (CreateTasksListBatches): if input already has `items` (from CreateTasksBulk), passes them through; otherwise lists S3 `batches/{date}/` and returns `{ items, week_start_date }`.
- **InsertBatches** (Map): for each item, invokes InsertTasksBatch (read S3 file, upsert to Supabase).
- **Finalize** (CreateTasksFinalize): update_store_issue_counts, StartProcessQueue, success email.

**Important:** The state machine’s first step is **PrepareBatches**, not CreateTasksBulk. For bulk runs you must **invoke the CreateTasksBulk Lambda first**; that Lambda then starts the state machine with the correct `items`. If you start the state machine from the AWS console with only `{"week_start_date":"..."}` and have not run CreateTasksBulk for that date, PrepareBatches will list S3 and get no batches (empty `items`), so InsertBatches runs 0 times.

---

## Key Files

| Path | Purpose |
|------|--------|
| `create_tasks/app.py` | **Original** in-process create tasks (unchanged). Triggered by BatchUpdateSjreport etc. |
| `create_tasks_bulk/app.py` | Bulk entry: fetch+transform, S3 upload, start Step Function. Invoke with `{"week_start_date":"YYYY-MM-DD"}`. |
| `create_tasks_list_batches/app.py` | Lists S3 keys under `batches/{week_start_date}/`; returns `{ items, week_start_date }`. Used as first step so state machine can be started from console with only `week_start_date`. |
| `insert_tasks_batch/app.py` | Reads one S3 JSON file, upserts to Supabase `task` (same on_conflict as create_tasks). |
| `create_tasks_finalize/app.py` | After all batches: update_store_issue_counts, invoke StartProcessQueue, success email. Has **local** `email_service.py` (do not rely on layer for it). |
| `create_tasks_workflow.asl.json` | Step Function definition (reference; actual definition is inline in template with !Sub). |
| `template.yaml` | CreateTasksStagingBucket (S3, lifecycle 1d on `batches/`), CreateTasks, CreateTasksBulk, CreateTasksListBatches, InsertTasksBatch, CreateTasksFinalize, CreateTasksWorkflow (AWS::StepFunctions::StateMachine). |

---

## How to Run

- **Normal (small) runs:** Invoke **CreateTasks** with `{"week_start_date":"YYYY-MM-DD"}` (e.g. from BatchUpdateSjreport or API).
- **Bulk (34k+ tasks):** Invoke the **CreateTasksBulk Lambda** with `{"week_start_date":"YYYY-MM-DD"}`. Do **not** start the state machine from the console for bulk runs. CreateTasksBulk will upload to S3 and start the Step Function with the batch list; the state machine will then run PrepareBatches (which passes through those items) → InsertBatches → Finalize.
- **State machine from console (e.g. re-run inserts only):** Start execution of **datafyNew-CreateTasks** with `{"week_start_date":"YYYY-MM-DD"}`. PrepareBatches will list S3 for that date; if no batches exist (e.g. you never ran CreateTasksBulk for that date), `items` is empty so InsertBatches runs 0 times. Run CreateTasksBulk for that date first if you want inserts.

---

## Gotchas

- **Empty `items`:** If you start the state machine with only `week_start_date` and haven’t run CreateTasksBulk for that date, PrepareBatches lists S3 and returns `items: []`. InsertBatches runs 0 times; only Finalize runs. For bulk runs, always invoke **CreateTasksBulk Lambda** first so it starts the state machine with `items` (PrepareBatches then passes them through). That’s expected.
- **Finalize ImportModuleError:** CreateTasksFinalize must have a **local** `email_service.py` (copy from create_tasks). Do not rely on the Lambda layer for `email_service` in this function.
- **Layer:** Other new Lambdas (create_tasks_bulk, insert_tasks_batch, create_tasks_list_batches) use CommonPythonLayer for supabase/pandas etc.; their `requirements.txt` are comment-only (deps from layer). CreateTasksFinalize and InsertTasksBatch have local email_service only where used (Finalize uses it; InsertTasksBatch does not).

---

## Template Snippets (for reference)

- **CreateTasks:** Original permissions (lambda Invoke StartProcessQueue, SSM, Gmail params). No S3, no Step Function.
- **CreateTasksBulk:** Env CREATE_TASKS_STAGING_BUCKET, CREATE_TASKS_STATE_MACHINE_ARN, CREATE_TASKS_FINALIZE_ARN. Policies: S3 PutObject, states:StartExecution, lambda Invoke (Finalize), SSM, Gmail.
- **CreateTasksWorkflow:** AWS::StepFunctions::StateMachine; DefinitionString with !Sub for CreateTasksListBatchesArn, InsertTasksBatchArn, CreateTasksFinalizeArn. Role has lambda:InvokeFunction on those three.
- **S3 lifecycle:** Rule with `Prefix: batches/`, `ExpirationInDays: 1` (top-level Prefix for cfn-lint). CreateTasksBulk uploads under `batches/{week_start_date}/{YYYYmmdd_HHMMSS}/` so each run has a unique path and multiple runs for the same date do not overwrite each other.

---

## Changelog / NewKnowledgeBase

- CHANGELOG.md has an entry for 2026-02-10 (Create Tasks bulk path, Step Function, PrepareBatches, Finalize local email_service).
- NewKnowledgeBase.md has a short note on “Step Functions + S3 for Lambda timeout avoidance”.