# Project plan — datafy_lambda

Evidence date: 2026-10-17.

Source repository: `git@bitbucket.org:overdrive/datafy_lambda.git`.

Inspected revision: `main` at `e985575` (2025-03-05, Dirk, "adding new lambda functions"). Full SHA `e9855759523cabec306577a106d5cd30e4311d0c`. `origin/HEAD` is `main`. No other branches were present.

The scan-host path `~/projects/sonar_scan/repos/datafy_lambda` was not on this machine. The tree was read from that git revision. A full working-tree checkout of every blob fails on Windows because committed `.aws-sam` paths exceed the path-length limit. Source files were read with a sparse checkout that excluded `.aws-sam` and `node_modules`. `git ls-tree` reports 81,689 files at this revision; 213 of those are outside `.aws-sam`.

This plan describes what the repository contains. It does not decide what the product should become.

There was no `silicon_ark/datafy_lambda/` directory before this document set. The source repository has `readme.md` and several SAM starter READMEs. Those were read. Where they disagree with the handlers, the disagreement is recorded.

Secret values that are committed in templates, scripts, and `.env` files are not copied here.

## Product

**Documented** in root `readme.md`: "Tools and Lambdas for Datafy". The same file describes a monthly call sheet, a weekly Spinning Jenny (SJ) report uploaded by a client, and scripts that build tasks per store per user after an upload from a frontend.

**Implemented** at this revision is a set of local Node scripts, four AWS SAM applications, SQL scripts, and one Supabase Edge Function. They read spreadsheets, write Postgres tables (`sjreport`, `task`, `store`, `store_product`, `app_user`, `kaching`), and run Amazon Rekognition custom labels against product images.

**Inferred.** Datafy here is a retail store-execution pipeline: import shelf and stock reports, turn issue rows into tasks, and label product photos. No product brief, owner, or success metric is in the repository.

Who operates the frontend named in the readme is **Unknown / Requires Confirmation**. That frontend is not in this repository.

## Related Silicon Ark sets

These directories already exist in Silicon Ark and were not produced by this inspection. They are separate repositories. This repository does not name them.

| Silicon Ark directory | What that set says it is | Relation to this tree |
|---|---|---|
| `datafy-lambda-new` | Later SAM repository (`datafy-lambda-new.git`, tip 2026-07-20). Its readme still points at `datafy_lambda\file_processing\` paths | Those JavaScript files are in this repository. Whether the later repository replaced this one in production is **Unknown / Requires Confirmation** |
| `datafy_rekognition` | Separate repo, tip 2024-06-03, SAM app `DatafyRek` plus a Rhodes training image set | This repository has its own `lambda/Rekognition` SAM app. They are not the same tree |
| `datafy-supabase-functions` | SQL kit for store metrics, RLS, and upload routing. Names clients `datafy_ai` and `datafy_web` | This repository has an older schema dump and a `Scripts/` folder. Whether they target the same database is **Unknown / Requires Confirmation** |
| `datafy_powersync` | Self-hosted PowerSync kit, database name `powersync_datafy` | Not referenced by this repository |
| `datafy_web` | Named by the other sets | Not referenced by this repository |

## Branch reality

| Line | Tip | Date | What it is |
|---|---|---|---|
| `main` (default; `origin/HEAD`) | `e985575` | 2025-03-05 | Only branch. 11 commits from 2024-05-17 to 2025-03-05 |

No tags were inspected. Nothing in git is dated after 2025-03-05. Whether any stack is still deployed is **Unknown / Requires Confirmation**.

| Commit | Date | Author | Subject |
|---|---|---|---|
| `126a83d` | 2024-05-17 | Jaco Brits | Initial commit (`.gitignore` only) |
| `dbcc141` | 2024-05-17 | Jaco Brits | Node parsers under `read_callsheet`, `read_sjreport`, `users` |
| `6e1e557` | 2024-05-19 | Jaco Brits | More `read_callsheet` listing scripts |
| `03d2dda` | 2024-07-22 | Jaco Brits | Moves those scripts under `old/` and `file_processing/`, adds `read_edge_function` and the schema dump |
| `e32e095` | 2024-07-22 | Jaco Brits | Root readme import notes |
| `0ef17b5` | 2024-10-01 | Dirk | Adds `lambda/datafy` and `lambda/Rekognition` |
| `266f807` | 2024-10-29 | Dirk | Adds `update_store_issues` and schedule comments on `lambda/datafy` |
| `bdb5d6f` | 2024-12-13 | Dirk | Adds `lambda/processes`; updates datafy and Rekognition handlers |
| `ddc2986` | 2024-12-18 | Dirk | "send date to create task function" on `lambda/datafy` |
| `1677b33` | 2025-01-06 | Dirk | Adds `Scripts/` |
| `e985575` | 2025-03-05 | Dirk | Adds `lambda/datafy_new` (and its `.aws-sam` tree) |

The onboarding inventory captured on 2026-09-28 listed `sonar-project.properties` and `sonarqube_scan.bat` at the repository root. Neither file is in git at `e985575`. There is no `bitbucket-pipelines.yml`.

## Objectives

No objectives file exists. The behaviour the commits introduced:

1. May 2024 — Local Node scripts that download public Supabase storage spreadsheets and insert `callsheet`, `sjreport`, `user`, and `file_upload` rows.
2. July 2024 — Those scripts are copied under `old/` and `file_processing/`. A Supabase CLI project `read_edge_function` is added with a remote schema dump and an edge function `read_sj_report`. The root readme describes the import order.
3. October 2024 — SAM stack `datafy` polls `sjreport_uploads` every minute, upserts spreadsheet rows, and invokes `CreateTasks`. SAM stack `Rekognition` starts a custom-labels model, analyses images, and stops the model.
4. December 2024 — SAM stack `processes` crops training images from Pascal VOC XML uploaded to S3. `CreateTasks` is changed to receive a week-start date; the file at HEAD of `lambda/datafy/create_tasks/app.py` still uses a fixed date (see the PRD).
5. January 2025 — SQL scripts for products, KaChing, upload triggers, store issue counts, and report functions.
6. March 2025 — SAM stack `datafyNew` repeats the datafy functions, reads Supabase settings from SSM, adds `ReadPromosKaching`, prefixes SPAR store codes, and calls `update_product_table`.

## Scope

### In scope of `main` at `e985575`

- Root readme and root `package.json` (`@azure/msal-node`, `axios`, `xlsx`). No root script imports `msal`.
- `file_processing/` — current Node importers (`read_xlxs` 1.0.0).
- `old/` — earlier copies of those importers, including a OneDrive download script.
- `read_edge_function/` — Supabase CLI config, one edge function, and schema dump `20240627123825_remote_schema.sql`.
- `Scripts/` — SQL intended to be applied to Postgres. Not a migration runner.
- `lambda/datafy` — SAM stack name `datafy`.
- `lambda/datafy_new` — SAM stack name `datafyNew`.
- `lambda/processes` — SAM stack name `processes`.
- `lambda/Rekognition` — SAM stack name `Rekognition`.
- Committed `.aws-sam` build output under `lambda/datafy_new` and `lambda/Rekognition`.

### Out of scope / not in this repository

- The frontend that the readme says uploads files.
- The live Supabase project, its current schema, and whether `https://alb.datafyapp.co.za` is the same project as the host in the readme. **Unknown / Requires Confirmation**.
- Proof that any SAM stack was deployed, or that a stack is still running.
- A pipeline, Sonar project file, Dockerfile, or custom domain resource.
- IAM, SSM parameter values, and the Rekognition project itself. The templates only grant permission to read parameters and call Rekognition.

## System shape

```text
Spreadsheet in Supabase Storage bucket "uploads"
        |  storage.objects insert
        v
sjreport_files_to_process()          (Scripts/Upload trigger.sql)
        |  insert sjreport_uploads where path starts sjreport
        v
EventBridge rate(1 minute)
        v
ReadFiles (stack datafy and stack datafyNew)
        |  upsert table named by the storage path
        |  async invoke CreateTasks with week_start_date
        v
CreateTasks
        |  RPC get_sjreport_data_pages
        |  upsert task
        |  RPC update_store_issue_counts   (datafyNew only)
        v
Postgres (Supabase)

Parallel, also rate(1 minute): ReadPromosKaching (datafyNew only)
        reads promo_kaching_uploads, upserts the path's table

S3 bucket datafy-prod-training-images  (.xml created)
        v
ConvertTrainingImages (stack processes)
        crops JPEG boxes to bucket datafy-prod-training

Step Functions (stack Rekognition), trigger commented out
        StartModel -> AnalyzeImage -> StopModel
        reads product_images, writes product_ml and storage bucket "ml"
```

`UpdateStoreIssues` is a function in both datafy templates and has no EventBridge rule and no invoke from the other handlers.

## Major components

| Component | Role at this revision |
|---|---|
| `file_processing` | Manual Node runs against Supabase using `.env` |
| `lambda/datafy` | Scheduled XLSX import and task build. Supabase URL and key are environment variables in the template |
| `lambda/datafy_new` | Same import plus KaChing promo import. Supabase URL and key come from SSM |
| `lambda/processes` | XML-driven image cropping for training |
| `lambda/Rekognition` | Start, detect, stop a Rekognition custom-labels model |
| `read_edge_function` | Schema snapshot (2024-06-27) and an edge function that lists storage and returns one `app_user` name |
| `Scripts` | Ad hoc SQL, including the upload trigger the Lambdas depend on |
| `old` | Superseded local scripts. Still in `main` |

## Milestones

There is no milestone plan in the repository. The dates above are commit dates, not an approved roadmap.

## Implementation plan

No forward plan is written in the repository. The newest application code is `lambda/datafy_new` (2025-03-05). `lambda/datafy` is still in the tree with its own stack name and a hardcoded task date.

## Known dependencies

- AWS Lambda, EventBridge, S3, SSM, Rekognition, Step Functions, CloudFormation/SAM, Application Insights. Account id and region `eu-west-1` are hardcoded in IAM resource ARNs.
- Supabase Postgres, Auth, and Storage.
- Python 3.9 (datafy, datafy_new, processes) and Python 3.10 (Rekognition).
- Node for `file_processing` (`@supabase/supabase-js`, `node-xlsx`, `date-fns`, `dotenv`).
- Pandas and openpyxl in the Lambda requirements.
- SAM CLI and Docker, documented in the starter READMEs.
- Local Postgres on port 54322 for the dump/restore commands in the readme.

## Current project status

**Implemented** in git, last change 2025-03-05. No open roadmap, no CI config, and no statement that work continues.

**Requires confirmation:** which stack is production (`datafy` vs `datafyNew`), whether `lambda/datafy` should still be deployed, and whether `datafy-lambda-new` has replaced this repository.