# Project plan — datafy-lambda-new

Evidence date: 2026-10-17.

Source repository: `git@bitbucket.org:overdrive/datafy-lambda-new.git`.

Inspected revision: `main` at `9519bea` (2026-07-20, "update documents"). `git ls-remote` HEAD matched that commit. History before the tip was not walked.

This plan describes what the repository contains. It does not decide what the product should become.

There was no `silicon_ark/datafy-lambda-new/` directory before this document set. The repository already contains `readme.md`, `CHANGELOG.md`, `NewKnowledgeBase.md`, `docs/`, and stack READMEs. Those were read and incorporated. Where they disagree with the templates or handlers, the disagreement is recorded.

Related Silicon Ark sets, not this repository: `datafy_web`, `datafy_rekognition`, `datafy_powersync`, `datafy-supabase-functions`.

## Product

**Inferred.** This repository is the AWS Lambda side of Datafy, a retail store-execution system. It imports weekly Spinning Jenny (SJ) report spreadsheets and promo/KaChing spreadsheets into Supabase, builds store tasks from those rows, refreshes store issue counts, backs up and clears old rows, emails operators, writes a combined weekly report and a KaChing daily report, and asks Amazon Bedrock vision models whether a shelf photo shows a specific product.

The root `readme.md` title is "Tools and Lambdas for Datafy" and names the SJ report, call sheet, store file, and task-parse steps. Those steps are described as paths under `datafy_lambda\file_processing\`. Those JavaScript files are not in this repository. Provenance: **Documented** path list, **not implemented** here.

Who the customer is, and which Supabase project the deployed Lambdas use, is **Unknown / Requires Confirmation**. Three different hosts appear in documentation across Datafy repos (see Integrations in the PRD).

## Branch reality

Default branch at inspection: `main`, tip `9519bea`.

`CHANGELOG.md` records implementation notes from 2026-02-10 through 2026-04-23 (Bedrock retry/batch, report service-role key, Read Files failure handling, Create Tasks bulk). The tip commit message is "update documents" on 2026-07-20, after the last changelog date. Which files that commit changed, and whether later code exists only on the scan host, was not established from a single-commit checkout.

The onboarding issue's top-level inventory (captured 2026-09-28) listed `sonar-project.properties`. That file is not in the tree at `9519bea`. There is no `bitbucket-pipelines.yml`.

## Scope in the repository

In scope of what is present:

- Four SAM application folders under `lambda/`: `datafy_new` (stack name `datafyNew`), `bedrock_analysis` (stack name `datafy-bedrock-analysis`), `processes` (stack name `processes`), `Rekognition` (stack name `Rekognition`).
- Python 3.9 handlers, a shared Python layer, Step Functions for bulk task insert and for Bedrock retry/batch, EventBridge schedules, API Gateway routes, S3 buckets, VPC attachment, SSM reads, Gmail SMTP notifications, and Supabase PostgREST/RPC/storage calls.
- Operational notes: root readme, changelog, knowledge base, and `docs/` for Supabase migration, private-subnet Lambda, RDS proxy, SMS auth hook, and CloudWatch Logs Insights.

Out of scope of the code that was found, or not deployed by the default template:

- The Flutter web client (`datafy_web`) and the separate `datafy_rekognition` training repository.
- Supabase SQL, RLS policies, and the RPC bodies. This repo calls them. It does not define them.
- JavaScript file processors named in the root readme.
- Rekognition Custom Labels functions as a deployable stack. `lambda/Rekognition/template.yaml` creates buckets, SSM parameters, and an IAM role. The functions and state machines live in `template-copy.yaml`, and the daily/weekly EventBridge rules in that copy are commented out.
- An SMS-sending Lambda. `docs/supabase-auth-hook-send-sms-lambda.md` describes how to wire GoTrue to one. No SMS handler is in the tree.
- A CI pipeline. None is committed.

## System architecture

```text
Operators / datafy_web / EventBridge
  → API Gateway (no authorizer in the templates; CORS *)
      datafyNew stack, stage prod
        POST /trigger-read-files
        POST /trigger-read-promos-kaching
        POST /generate-combined-week-report
      bedrock-analysis stack, stage prod
        POST /analyze-image
        POST /analyze-image-test
  → Lambda (Python 3.9) in VPC vpc-0ada1e5366ad5c132
      private subnets, NAT egress (template description)
      SSM /supabase/* and /gmail/*
      Supabase PostgREST, RPC, Storage
      Gmail SMTP (email_service)
  → Step Functions
      datafyNew-CreateTasks (S3 batch insert)
      {Stack}-BedrockRetry and {Stack}-BedrockBatch
  → S3
      CreateTasks staging bucket (batches/, 1-day lifecycle)
      datafy-bedrock-artifacts (default parameter)
      processes TrainingS3Bucket (XML object-created)
      Rekognition source and training buckets
  → Amazon Bedrock (eu-west-1 IAM; inference profile eu.anthropic.claude-sonnet-4-6)
```

Account id hardcoded in IAM resource ARNs: `587594388832`. Region in those ARNs: `eu-west-1`. Provenance: **Implemented** in `lambda/datafy_new/template.yaml` and `lambda/bedrock_analysis/template.yaml`.

## Major components

| Component | Stack | Role |
|---|---|---|
| `ReadFiles` + `TriggerReadFiles` | datafyNew | Import one unprocessed SJ spreadsheet from Supabase Storage |
| `ReadPromosKaching` + `TriggerReadPromosKaching` | datafyNew | Import one unprocessed promo/KaChing spreadsheet |
| `BatchUpdateSjreport` | datafyNew | After import, start bulk task creation and mark the upload processed |
| `CreateTasks` | datafyNew | In-process task upsert. Still deployed. The import chain calls the bulk path |
| `CreateTasksBulk` + workflow | datafyNew | Fetch SJ rows, stage JSON on S3, Map-insert `task`, then finalize |
| `StartProcessQueue` and backup/clear chain | datafyNew | Customer ids, archive, store queue, CSV backup, delete old rows |
| `UpdateStoreIssues`, `UpdateTaskIssues` | datafyNew | Store issue RPCs and task-status RPC. No API or schedule in the template |
| `CreateCombinedWeeklyReport` | datafyNew | Paginated RPC to Excel in Storage bucket `Reports`; email only on cron |
| `CreateKachingDailyReport` | datafyNew | Daily KaChing RPC to Excel; cron only |
| `BedrockImageAnalysis` | bedrock-analysis | Shelf check via Claude; writes `task` / `product_images` |
| `BedrockImageAnalysisTest` | bedrock-analysis | Dry-run model call. Default model Qwen. No DB or S3 writes |
| `BedrockRetryWorkflow` | bedrock-analysis | Daily 22:00 SAST retry of incomplete GenAI for tasks fixed that SAST day |
| `BedrockBatchWorkflow` | bedrock-analysis | Same window, grouped by `product_id`, one model call per group |
| `ConvertTrainingImages` | processes | On `.xml` upload, crop the matching `.jpg` by Pascal VOC boxes |
| Rekognition functions | not in `template.yaml` | Dataset, training, and inference handlers under `lambda/Rekognition/functions/` |

## SJ import chain (implemented)

1. `POST /trigger-read-files` sends a start email and asynchronously invokes `ReadFiles`.
2. `ReadFiles` takes the next `sjreport_uploads` row with `processed` equal to the string `FALSE` and `times_ran` less than 5.
3. It downloads the Excel object, fuzzy-matches columns, requires `date` and `store_code` on at least one visible sheet, and upserts rows.
4. It runs product and customer RPCs. Duplicate-key `23505` on `insert_distinct_products` is ignored.
5. On failure it still sets `processed` true, logs, and emails.
6. It invokes a **hardcoded** Lambda ARN for `BatchUpdateSjreport`.
7. `BatchUpdateSjreport` does not run its own product-id update (that call is commented out). It asynchronously invokes `CreateTasksBulk`, then marks the upload processed.
8. `CreateTasksBulk` writes 5,000-row JSON files to S3 and starts the state machine.
9. The state machine inserts `task` rows, then `CreateTasksFinalize` calls `update_store_issue_counts` and `StartProcessQueue`.
10. `StartProcessQueue` updates task customer ids, archives old tasks, rebuilds `store_processing_queue`, and starts the backup/clear chain.

Provenance: **Implemented** in the handlers and `lambda/datafy_new/template.yaml`. The intended order is also **Documented** in `CREATE_TASKS_CONTEXT.md`, `start_process_queue/FLOWCHART.md`, and `lambda/datafy_new/migration.md`.

## Milestones visible in the changelog

These are recorded changes, not a forward roadmap. No future milestone list exists in the repo.

| When | What landed (changelog) |
|---|---|
| 2026-02-10 to 2026-02-13 | Bulk Create Tasks via S3 and Step Functions; timestamped batch prefixes |
| 2026-02-20 to 2026-02-27 | Bedrock prompt and model changes (Sonnet 4.6, exact product, five positions, EXIF); Read Files marks failed imports processed |
| 2026-03-04 to 2026-03-12 | Bedrock token logging; weekly report uses `/supabase/service_role` |
| 2026-04-21 to 2026-04-23 | Bedrock retry and batch Step Functions, Qwen test API, fallback profile, SAST 22:00 schedule |

## Implementation plan (as the repo stands)

There is no separate implementation plan. The closest planning documents are:

- `lambda/datafy_new/migration.md` and `lambda/bedrock_analysis/migration.md` — inventory for moving stacks to a new AWS account and a new Supabase project. They state that no code changes are included in those files.
- `docs/migrate-supabase-cloud-to-self-hosted.md` and `docs/migrate-supabase-delta-sync.md` — migration runbooks.
- `docs/private-subnet-lambda-sam.md` — how the Lambdas attach to the private subnets.

Whether that migration has been executed is **Unknown / Requires Confirmation**. The templates still default to VPC `vpc-0ada1e5366ad5c132` and account `587594388832`.

## Known dependencies

- AWS SAM CLI, Docker (documented for `sam build --use-container`), Python 3.9, profile name `datafy` in the root readme's `sam sync` example.
- Supabase Python client, pandas, openpyxl, rapidfuzz, requests, numpy (layer `lambda/datafy_new/lambda_layer/requirements.txt`).
- SSM parameters listed in `migration.md`: `/supabase/url`, `/supabase/anon`, `/supabase/service_role` (weekly report), `/gmail/username`, `/gmail/app_password`, `/gmail/to_emails`, `/gmail/to_emails_client`.
- Postgres RPCs named in the handlers (see the PRD and business rules). The function bodies are not in this repo.
- Amazon Bedrock model access for Claude Sonnet 4.6 / 4.5 and, for the test route, Qwen.
- Root `package.json` depends on `@azure/msal-node`, `axios`, and `xlsx`. No JavaScript application uses them in this tree.

## Current status

**Implemented** and documented as the active import and GenAI path: the `datafyNew` and `bedrock-analysis` templates.

**Present but not on the default deploy path:** `sj_read_files`, root `lambda/trigger_read_files` (no email; expects `READ_FILES_ARN`), `template copy.yaml`, Rekognition `template-copy.yaml`, `processes` `ConvertXmlToJson` (commented out), Read Promos one-minute schedule (commented out).

**Starter / stale text:** `lambda/datafy_new/README.md` still describes a Hello World SAM app. `lambda/datafy_new/tests/` is the SAM API Gateway sample (`from read_files import app`). Tests were not executed.

**Unknown / Requires Confirmation:** production API URLs, whether `datafy_web`'s `Stage/analyze-image` host is this Bedrock stack (this template's stage name is `prod`), which Supabase URL is in SSM, and whether the Rekognition copy-template was ever the deployed stack.