# Features — datafy_lambda Evidence date: 2026-10-17. Source: `git@bitbucket.org:overdrive/datafy_lambda.git`, `main` at `e985575` (2025-03-05). "Current" means present in the default templates or in a script that calls itself when loaded. A handler whose SAM resource is commented out is not current. There is no issue tracker in the repository, so nothing is marked planned unless a comment describes behaviour the code does not perform. ## Current features ### F-01 — Poll SJ uploads every minute `lambda/datafy/template.yaml` and `lambda/datafy_new/template.yaml` both declare `ReadFiles` with EventBridge `rate(1 minute)`. Status: **current** in both stacks. Confidence: high. ### F-02 — Import an SJ workbook into the folder's table `lambda/datafy_new/read_files/app.py` downloads one unprocessed `sjreport_uploads` row, reads every sheet of an `.xlsx` file, and upserts into the table named by `path_tokens[0]`. Status: **current**. Confidence: high. ### F-03 — SPAR store-code prefix Only `lambda/datafy_new/read_files/app.py`. If `channel` contains `SPAR`, `store_code` is prefixed with `S` before insert. Status: **current** on stack `datafyNew`. Absent from `lambda/datafy/read_files/app.py`. Confidence: high. ### F-04 — Hand the week-start date to CreateTasks `datafy_new` invokes `datafyNew-CreateTasks-ccMR48YAjXLP` with `week_start_date` set to the last Monday computed in that file. `lambda/datafy_new/create_tasks/app.py` reads `event['week_start_date']`. Status: **current** on `datafyNew`. Confidence: high. ### F-05 — Build task rows from SJ issues `lambda/datafy_new/create_tasks/app.py` pages `get_sjreport_data_pages` and upserts `task`. `Scripts/get_sjreport_data_pages.sql` excludes `NO ISSUE - ITEM SELLING`. Status: **current** in the handler. The SQL file may not match the columns the handler reads (`article`, `customer_id`, `dc_soh`). Confidence: medium on the live function, high on the Python. ### F-06 — Refresh store issue counts after tasks `lambda/datafy_new/create_tasks/app.py` calls RPC `update_store_issue_counts` after the upsert. `lambda/datafy_new/read_files/app.py` calls RPC `update_product_table` before the invoke. Status: **current** as calls. The function bodies are not in this repository under those names. Confidence: high that the calls exist, unknown that the functions exist. ### F-07 — Import promo / KaChing workbooks `ReadPromosKaching` in `lambda/datafy_new` only. Schedule `rate(1 minute)`. Queue table `promo_kaching_uploads`. Upsert on `row, file, sheetname`. Origin: `e985575`. Status: **current**. Confidence: high. ### F-08 — Older datafy stack still in the tree Stack name `datafy`. Same three functions as the first SAM app (`ReadFiles`, `CreateTasks`, `UpdateStoreIssues`). Supabase URL `https://alb.datafyapp.co.za` is an environment variable. `CreateTasks` uses the date `2024-12-23`. Status: **current** in git. Whether it is still deployed beside `datafyNew` is **Unknown / Requires Confirmation**. ### F-09 — Crop labelled training images `lambda/processes` function `ConvertTrainingImages`, S3 suffix `.xml`, bucket `datafy-prod-training-images`, output bucket `datafy-prod-training`. Origin: `bdb5d6f`. Status: **current**. Confidence: high. ### F-10 — Rekognition start / detect / stop `lambda/Rekognition/template.yaml` state machine and three functions. Analyze writes `product_ml` and storage bucket `ml`, and marks `product_images.processed`. Origin: `0ef17b5`, handler updates in `bdb5d6f`. Status: **current** as a state machine with no schedule. Confidence: high. ### F-11 — Manual Node importers `file_processing/` scripts listed in the PRD workflow WF-5. Package name `read_xlxs`. Origin: `03d2dda`. Status: **current** as files that run `main()` on load. Confidence: high. ### F-12 — Rep auth signup `file_processing/signup_users.js` creates Auth users and `app_user` rows from `group_by_users()`. Status: **current** as a script. The `pool` it uses is undefined in the file. Confidence: high. ### F-13 — Schema snapshot and edge function `read_edge_function/supabase/migrations/20240627123825_remote_schema.sql` and function `read_sj_report`. Status: **current** in the tree. The edge function does not import spreadsheets (the processing loop is commented out). Confidence: high. ### F-14 — SQL script library `Scripts/` added in `1677b33`. Includes upload trigger, KaChing table, product tables, store-count functions, and brand-total functions. Status: **current** as files. Nothing in the repo applies them automatically. Confidence: high. ### F-15 — Application Insights Each SAM template creates `ApplicationResourceGroup` and `ApplicationInsightsMonitoring` with auto configuration. Status: **current**. Confidence: high. ## Not deployed by the current templates ### F-16 — UpdateStoreIssues `lambda/datafy/update_store_issues/app.py` and `lambda/datafy_new/update_store_issues/app.py` page stores with `get_store_data_pages`, then call `get_issue_counts`, `update_store_issues`, and `update_total_counts`. No `Events` block and no `lambda.invoke` of this function was found. Status: **in tree, not triggered**. Confidence: high. ### F-17 — XML to Rekognition JSON `lambda/processes/convert_xml_to_json/app.py`. The resource `ConvertXmlToJson` is commented out in `template.yaml`. Status: **not deployed** by this template. Confidence: high. ### F-18 — Daily Rekognition schedule `DailyTriggerRule` in `lambda/Rekognition/template.yaml` is commented out (`cron(0 12 * * ? *)`). Status: **not deployed**. Confidence: high. ### F-19 — API Gateway hello route Described in the SAM starter READMEs and in `lambda/datafy/template copy.yaml` (`ReadFiles` as an `Api` event). The active `template.yaml` files use a schedule, and the API outputs are commented out. Status: **not current**. Confidence: high. ### F-20 — Node task insert `file_processing/3_parse_task.js` builds the row and leaves the insert commented out. Status: **incomplete**. Confidence: high. ### F-21 — Edge-function spreadsheet import `processFile` in `read_sj_report/index.ts` is not called. Status: **incomplete**. Confidence: high. ## Completed, kept as history ### F-22 — Early parsers, now under `old/` `old/callsheet`, `old/read_callsheet`, `old/read_sjreport`, `old/store`, `old/store_product`, `old/task`, `old/users`. Moved in `03d2dda`. `old/read_callsheet/parse_callsheet_onedrive.js` downloads one OneDrive share link. Status: **completed and retained**. The folder name `old` is the only deprecation signal. Confidence: high. ### F-23 — One-off SQL data fixes `Scripts/Update issue for feedback.sql` renames `NO ISSUE - IGNORE` to `NO ISSUE - ITEM SELLING` and sets `task.customer_id = 1` where `week_start_date = 2024-12-16`. `Scripts/sj-task-backup-restore.sql` copies `sjreport` and `task` into `*_backup_20241209`. Status: **scripts present**. Whether they were executed is **Unknown**. Confidence: high that the files exist. ## Planned or in development No TODO, roadmap, or ticket list was found. Comments that describe work not finished in the active template: - `lambda/datafy/create_tasks/app.py` and `read_files/app.py`: "Change to paramenter store". `datafy_new` does read SSM. The older stack still uses environment variables. - `lambda/datafy_new/read_files/app.py` and `read_promo_kaching/app.py` keep a commented query that selects `processed = TRUE` and `test = TRUE` "for testing only". - Rekognition daily schedule, processes XML-to-JSON function, and the edge-function file loop are commented out rather than marked as a future release. ## Recently changed At `e985575` (2025-03-05) the new tree is `lambda/datafy_new`: `ReadFiles`, `ReadPromosKaching`, `CreateTasks`, `UpdateStoreIssues`, SSM parameters, SPAR prefix, `update_product_table`, and a committed `.aws-sam` directory. The previous product change is `Scripts/` on 2025-01-06 (`1677b33`). ## Deprecated or removed - Paths `read_callsheet/`, `read_sjreport/`, and `users/` existed in May 2024 and are absent at HEAD. Their contents were copied under `old/` in `03d2dda`. `users/result.json` was added in `03d2dda` and is not in the HEAD tree. - No SAM function was deleted between `lambda/datafy` and `lambda/datafy_new`. `ReadPromosKaching` was added. Both stacks remain. ## Tests present `tests/unit/test_handler.py` under `datafy`, `datafy_new`, and `processes` is the SAM API Gateway sample. It imports `read_files.app` and expects a hello-world style response. It does not exercise the schedule handler. `lambda/Rekognition/tests/unit/test_buyer.py`, `test_seller.py`, and `test_checker.py` are the SAM stock-trading sample. `test_state_machine.py` is the sample integration test. `file_processing/package.json` script `test` prints an error and exits 1. Status: **starter tests, not product tests**. Confidence: high.