# Technical documentation — datafy-lambda-new

Evidence date: 2026-10-17. Source: `main` at `9519bea` (2026-07-20).

This describes the repository as inspected. It does not prescribe a redesign. Secret values are not copied. SSM parameter names are configuration keys, not secret values.

## Repository layout

```text
readme.md                      operational notes (Supabase client, SAM sync, SJ import narrative)
CHANGELOG.md                   change log through 2026-04-23
NewKnowledgeBase.md            implementation notes (Bedrock, layers, RPC, timeouts)
package.json                   Node dependencies; no application JS in the tree
docs/                          migration, VPC, SMS hook, Logs Insights, one-off scripts
lambda/datafy_new/             SAM stack datafyNew
lambda/bedrock_analysis/       SAM stack datafy-bedrock-analysis
lambda/processes/              SAM stack processes
lambda/Rekognition/            buckets in template.yaml; functions in template-copy.yaml
lambda/trigger_read_files/     unwired second trigger (no email)
```

There is no `bitbucket-pipelines.yml` and no `sonar-project.properties` at this revision.

## Stacks

| Folder | samconfig stack_name | Deploy template | Runtime |
|---|---|---|---|
| `lambda/datafy_new` | `datafyNew` | `template.yaml` | python3.9, x86_64 |
| `lambda/bedrock_analysis` | `datafy-bedrock-analysis` | `template.yaml` | python3.9, x86_64 |
| `lambda/processes` | `processes` | `template.yaml` | python3.9, x86_64 |
| `lambda/Rekognition` | `Rekognition` | `template.yaml` (buckets and role only) | function code is Python; not in this template |

`samconfig.toml` sets `confirm_changeset = true`, `resolve_s3 = true`, and `capabilities = CAPABILITY_IAM` (bedrock also `CAPABILITY_NAMED_IAM`). Region and profile are not in the toml. The root readme's sync example uses `--profile datafy`. IAM ARNs in the templates are `eu-west-1` and account `587594388832`.

`template copy.yaml` files are not the default SAM template.

## datafyNew globals

From `lambda/datafy_new/template.yaml`:

- Timeout 900 seconds, memory 1024 MB, ephemeral storage 1536 MB.
- VPC `vpc-0ada1e5366ad5c132`.
- Subnets `subnet-0aa14f7bb45db8b87`, `subnet-0b651a99800a9d99e`. Description says private subnets with NAT egress.
- Security group `LambdaSecurityGroup` created by the stack.
- Logging `LogFormat: JSON`.
- `InsertTasksBatch` overrides timeout to 300 and memory to 512.

Bedrock uses the same VPC defaults. `ListBedrockRetryTasks` and `ListBedrockBatchGroups` are timeout 120 and memory 512. `BedrockBatchAnalyze` memory is 2048. Processes globals are timeout 600 and memory 1024, and that template does not set the VPC.

## Layer

`CommonPythonLayer`, `BuildMethod: python3.9`, content `lambda/datafy_new/lambda_layer/`.

`requirements.txt`: `requests`, `supabase`, `numpy==1.26.4`, `pandas==2.2.2`, `openpyxl`, `rapidfuzz`.

Function-level requirements files are expected not to repeat those (root readme). `CreateTasksFinalize` carries a local `email_service.py` because the layer must not be relied on for that import (`CREATE_TASKS_CONTEXT.md`).

Rebuild after layer changes: `sam build --use-container --clean` then deploy. Single-function code sync, documented in the root readme:

```text
sam build ReadFiles --cached
sam sync --stack-name datafy-new -t template.yaml --resource-id ReadFiles --code --profile datafy
```

The samconfig stack name is `datafyNew`. The sync example uses `datafy-new`. Which name is deployed is **Unknown / Requires Confirmation**.

## HTTP APIs

`ReadFilesApi` and `BedrockAnalysisApi` are `AWS::Serverless::Api`, `StageName: prod`.

CORS: methods `POST,OPTIONS`; headers `Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token`; origin `*`.

No `Auth` property. The `Authorization` header is allowed and not checked by these templates.

| Route | Function | Returns |
|---|---|---|
| `POST /trigger-read-files` | `TriggerReadFiles` | 202 after async invoke |
| `POST /trigger-read-promos-kaching` | `TriggerReadPromosKaching` | 202 after async invoke |
| `POST /generate-combined-week-report` | `CreateCombinedWeeklyReport` | report result JSON (handler) |
| `POST /analyze-image` | `BedrockImageAnalysis` | analysis JSON or HTTP 500 |
| `POST /analyze-image-test` | `BedrockImageAnalysisTest` | model JSON, no persistence |

Deployed hostnames are stack outputs and are not committed. `datafy_web` calls `https://i78eyrnc87.execute-api.eu-west-1.amazonaws.com/Stage/analyze-image`. Stage name `Stage` does not match this template's `prod`. **Requires confirmation.**

## Schedules

| Rule | Expression | Input | Default |
|---|---|---|---|
| `CreateCombinedWeeklyReportDailySchedule` | `cron(0 3 * * ? *)` | `{"customer_id": 1,"is_cron": true}` | Always in template |
| `CreateKachingDailyReportDailySchedule` | `cron(0 3 * * ? *)` | `{"customer_id": 1}` | Always in template |
| `BedrockRetryScheduleRule` | `cron(0 20 * * ? *)` | `{}` | On when `EnableBedrockRetrySchedule=true` (default) |

03:00 UTC is 05:00 SAST (UTC+2, no DST). 20:00 UTC is 22:00 SAST. The Bedrock rule description states that. The report crons do not mention SAST.

Commented and therefore not deployed from these files: `ReadPromosKaching` `rate(1 minute)`; Rekognition daily `cron(0 2 * * ? *)` and Sunday `cron(0 1 ? * SUN *)` inside `template-copy.yaml`.

## Step Functions

### CreateTasksWorkflow

Definition is inline in `lambda/datafy_new/template.yaml` (a reference ASL file `create_tasks_workflow.asl.json` is mentioned in `CREATE_TASKS_CONTEXT.md`).

Order: `CreateTasksListBatches` → Map `InsertTasksBatch` → `CreateTasksFinalize`.

`CreateTasksBulk` starts the execution. It is not a state. Env `CREATE_TASKS_STATE_MACHINE_ARN`.

### BedrockRetryWorkflow and BedrockBatchWorkflow

List page → choice → Map (`ResultPath: null`, so outputs are discarded and the 256 KB state limit is avoided) → next page while `has_more`.

Concurrency parameters: `BedrockRetryMapMaxConcurrency` default 2 (max 100); `BedrockBatchMapMaxConcurrency` default 2. Batch chunk `BedrockBatchMaxShelfImages` default 6 (max 20). Batch page `BedrockBatchListPageSize` default 80 (max 500). Retry page env `RETRY_LIST_BATCH_SIZE` is `500`.

`run_date` on the execution input is preserved for later pages. `{}` means today in `Africa/Johannesburg`, computed in the list Lambdas with `zoneinfo`.

## Environment variables set by SAM

| Function | Variable | Source |
|---|---|---|
| `CreateTasksBulk` | `CREATE_TASKS_STAGING_BUCKET`, `CREATE_TASKS_STATE_MACHINE_ARN`, `CREATE_TASKS_FINALIZE_ARN` | bucket, state machine, finalize ARN |
| `CreateTasksListBatches` | `CREATE_TASKS_STAGING_BUCKET` | bucket |
| `CreateTasksFinalize` | `START_PROCESS_QUEUE_ARN` | `StartProcessQueue` |
| `BatchUpdateSjreport` | `CREATE_TASKS_BULK_ARN` | `CreateTasksBulk` |
| `StartProcessQueue` | `BACKUP_SJREPORT_ARN` | `BackupSjreport` |
| `BackupSjreport` | `CLEAR_SJREPORT_OLD_DATA_ARN` | `SJReportClearOldData` |
| `SJReportClearOldData` | `BACKUP_TASKS_ARN` | `BackupTasks` |
| `BackupTasks` | `CLEAR_TASKS_OLD_DATA_ARN` | `TasksClearOldData` |
| `TriggerReadFiles` | `READ_FILES_ARN` | `ReadFiles` |
| `TriggerReadPromosKaching` | `READ_PROMOS_KACHING_ARN` | `ReadPromosKaching` |
| `BedrockImageAnalysis` | `INFERENCE_PROFILE_ARN` | literal `eu.anthropic.claude-sonnet-4-6` |
| same | `BEDROCK_FALLBACK_INFERENCE_PROFILE_ARN` | parameter, default the same 4.6 profile |
| same | `GENAI_ARTIFACTS_S3_BUCKET` | `ArtifactsBucketName`, default `datafy-bedrock-artifacts` |
| `BedrockImageAnalysisTest` | `BEDROCK_TEST_MODEL_ID` | `qwen.qwen3-vl-235b-a22b` |
| `ListBedrockBatchGroups` | `BATCH_MAX_SHELF_IMAGES`, `BATCH_LIST_PAGE_SIZE` | parameters |

## Hardcoded invoke targets

These do not update when CloudFormation renames the function (`migration.md`):

- `ReadFiles` policy resource `arn:aws:lambda:eu-west-1:587594388832:function:datafyNew-BatchUpdateSjreport-utMCtdWkdNo8`
- `CreateTasks` policy resource `arn:aws:lambda:eu-west-1:587594388832:function:datafyNew-StartProcessQueue-r6W4pr83cx2y`. `create_tasks/app.py` invokes that same function name.

`read_files/app.py` `invoke_lambda_function` calls `FunctionName='datafyNew-BatchUpdateSjreport-utMCtdWkdNo8'` with `InvocationType=Event`. That string is not an environment variable. A renamed function fails this invoke even if the IAM list is updated to the new ARN.

## SSM and Supabase

| Parameter | Consumers |
|---|---|
| `/supabase/url` | Almost every data and Bedrock Lambda |
| `/supabase/anon` | Same, as the API key |
| `/supabase/service_role` | `CreateCombinedWeeklyReport` only |
| `/gmail/username`, `/gmail/app_password`, `/gmail/to_emails`, `/gmail/to_emails_client` | Functions that construct `email_service` |

Client construction is `create_client(url, key)`. `ClientOptions(http_client=...)` is **Documented** as unsupported on the supabase-py version they use.

`raw_sql` RPC with `SET statement_timeout = '0'` is used by several long functions. That depends on a Postgres function named `raw_sql` existing and being callable by the key. The function is not in this repo.

### Storage buckets used from code

| Bucket string | Use |
|---|---|
| `public/{name from payload}` | Download SJ and promo uploads |
| `Reports` | Weekly and KaChing Excel, `get_public_url` |
| `uploads/backups` | CSV backups of old SJ and task rows |
| Table name `report_files` | Weekly report metadata row |

S3 (AWS), separate from Supabase Storage:

| Bucket | Use |
|---|---|
| `CreateTasksStagingBucket` | Task JSON batches, 1-day lifecycle |
| `ArtifactsBucketName` | Bedrock prompt and images |
| `TrainingS3Bucket` | Processes XML trigger |
| Rekognition source and training buckets | Created by `lambda/Rekognition/template.yaml` |

## Bedrock IAM

Production and batch functions may `bedrock:InvokeModel` and `InvokeModelWithResponseStream` on:

- `foundation-model/anthropic.claude-sonnet-4-6`
- `foundation-model/anthropic.claude-sonnet-4-5`
- `inference-profile/*`

The test function adds `bedrock:Converse` and `foundation-model/qwen.*`.

Primary profile id in env is `eu.anthropic.claude-sonnet-4-6` (cross-region inference profile form). Region of the Lambda is `eu-west-1`.

Token log line shape: `BedrockUsage` JSON with `input_tokens`, `output_tokens`, and `model_id`. Queries are documented in `BEDROCK_SETUP.md` and `docs/Log-insights-queries.md`. The knowledge base cost formula still cites Claude Sonnet 4 rates. Pricing was not re-verified.

`HasArtifactsBucket` is true when `ArtifactsBucketName` is not `''`. The default is non-empty, so the S3 put policy is included unless overridden. The policy resources are the literal bucket name `datafy-bedrock-artifacts`, not `!Ref ArtifactsBucketName`. A different parameter value would set the env var and still grant IAM only on `datafy-bedrock-artifacts`. **Implemented** inconsistency.

## Processes function

Logical id `ConvertTrainingImages`, `CodeUri: get_training_images/`, handler `app.lambda_handler`. Event: S3 object created, suffix `.xml`, bucket `TrainingS3Bucket`. Policy `AmazonS3FullAccess`.

`ConvertXmlToJson` is commented out above it in the same template.

## Rekognition template.yaml

Resources actually declared:

- `SourceDataBucket`, `TrainingDataBucket`
- SSM parameters for source bucket, training bucket, training prefix, project name
- `RekognitionServiceRole`
- Application Insights resource group

Functions under `lambda/Rekognition/functions/` (`analyze_image`, `create_manifest`, `create_dataset`, `start_training`, `start_model`, `stop_model`, and the check/manage helpers) are wired in `template-copy.yaml` only. `sam deploy` in that directory, with the committed samconfig, uses `template.yaml`.

Supabase tables touched by that code include `product_images`, `product_ml`, `model_versions`, `models`, `datasets`.

## Auth

No Cognito, no IAM authorizer, no API key, no Supabase JWT verification on the HTTP handlers. Callers who reach API Gateway invoke the functions. The functions then use the anon or service-role key from SSM.

VPC placement is not an authentication control. API Gateway invokes the Lambda through the Lambda service API.

The SMS hook document describes Standard Webhooks signature verification for a GoTrue send-SMS hook. That Lambda is not in the tree. The document's example secret is a sample and is not recorded here.

## Email

`email_service.py` (copied per function, not a single package) reads Gmail SSM parameters and sends with SMTP SSL. Failures to send are printed and usually do not fail the Lambda, except where the caller does not catch them.

## Monitoring

- JSON Lambda logs.
- Application Insights auto-configuration on datafyNew and processes.
- `BedrockUsage` log lines for token sums.
- Upload rows store an `error` string (`sjreport_uploads`, `promo_kaching_uploads`).
- `tasks_create_logs` is written by the create-task handlers.
- `genai_error` on `product_images` for production analysis failures.

No third-party APM, no queue (SQS), and no DynamoDB table are defined in the active templates. The separate `datafy_rekognition` repository writes Rekognition labels to DynamoDB. This repo's Rekognition `analyze_image` path uses Supabase.

## Local development and tests

Documented commands (datafy README, still partly Hello World):

```text
sam build --use-container
sam deploy --guided
sam local invoke ReadFiles --event events/uploadEvent.json
sam local invoke CreateTasks --event events/uploadEvent.json
sam local invoke CreateTasksBulk --event events/uploadEvent.json
sam local start-api
```

`lambda/datafy_new/tests/unit/test_handler.py` builds an API Gateway proxy event and imports `from read_files import app`. It is the SAM sample. It was not run. Integration tests expect `AWS_SAM_STACK_NAME`.

Root `readme.md` also documents `pg_dump` against the Supabase pooler and `npx supabase db dump`, plus restore into local Postgres on port 54322. Those are operator commands. Credentials in that file were not copied into Silicon Ark.

## CI and quality gates

None committed. `package-lock.json` exists beside `package.json`. No pipeline installs it.

## Deployment defects already written down in-repo

From `lambda/datafy_new/migration.md` and the templates:

1. Literal Lambda ARNs for Batch Update and Start Process Queue.
2. Account id repeated in every SSM ARN.
3. VPC ids are template defaults, not lookups.
4. Bedrock artifact IAM is pinned to bucket name `datafy-bedrock-artifacts`.
5. Fallback inference profile default equals the primary profile, so the fallback does not change model unless the parameter is overridden.
6. `UpdateTaskIssues` passes a Python set as the RPC params (`{ date }`), which is not a named-argument object.

## docs/ index

| File | Subject |
|---|---|
| `docs/private-subnet-lambda-sam.md` | Running these Lambdas in the private subnets |
| `docs/connect-private-ec2-ssm.md` | SSM access to a private EC2 |
| `docs/supabase-aws-rds-proxy-setup.md` | RDS proxy in front of Postgres |
| `docs/supabase-auth-hook-send-sms-lambda.md` | GoTrue SMS hook to API Gateway. Names `https://supabase.datafy.co.za` |
| `docs/supabase-auth-container-logs.md` | Auth container logs |
| `docs/migrate-supabase-cloud-to-self-hosted.md` | Cloud to self-hosted move |
| `docs/migrate-supabase-delta-sync.md` | Delta sync |
| `docs/Log-insights-queries.md` | CloudWatch Insights queries |
| `docs/scripts/` | Delta retry and storage-key remap helpers |

Whether those runbooks match production today is **Unknown / Requires Confirmation**. The SAM defaults still point at the VPC and account above.