# Changelog

## [2026-04-23] - BedrockRetry schedule: 22:00 SAST daily (20:00 UTC)

### Changed
- **`lambda/bedrock_analysis/template.yaml`:** **`BedrockRetryScheduleRule`** **`cron(0 20 * * ? *)`** — 10pm **Africa/Johannesburg** (was **22:00 UTC**, ~midnight SAST).

### Documentation
- **`lambda/bedrock_analysis/README.md`**.

## [2026-04-23] - Bedrock retry list: document task-table-only (no `product_images` query)

### Documentation
- **`lambda/bedrock_analysis/list_retry_tasks/app.py`**, **`README.md`**, **`NewKnowledgeBase.md`:** Clarify **`ListBedrockRetryTasks`** uses **`task` only;** removed unused **`json`** import.

## [2026-04-23] - Bedrock retry list: `task_id` items only (no `image_id`)

### Changed
- **`lambda/bedrock_analysis/list_retry_tasks/app.py`:** Step Functions items are only **`{ "task_id": "..." }`**. Requires non-empty **`task.image_url`**. Removed per-**`product_images`** **`image_id`** rows (multi-shelf-per-task without **`task.image_url`** → use batch or API).

### Documentation
- **`lambda/bedrock_analysis/README.md`**, **`NewKnowledgeBase.md`**.

## [2026-04-23] - Step Functions: execution input `run_date` for past SAST days

### Fixed
- **`lambda/bedrock_analysis/template.yaml`:** **`BedrockRetry`** / **`BedrockBatch`** — replaced **PrepareInput** with **Choice** + **Seed** states so **`{"run_date":"YYYY-MM-DD"}`** in **Start execution** is passed to the first **List** Lambda (was discarded). Schedule/EventBridge **`{}`** unchanged (today SAST).

### Documentation
- **`lambda/bedrock_analysis/README.md`**.

## [2026-04-23] - Bedrock list Lambdas: explicit `fixed_at IS NOT NULL`

### Changed
- **`lambda/bedrock_analysis/list_retry_tasks/app.py`**, **`list_bedrock_batch_groups/app.py`:** `.not_.is_("fixed_at", "null")` before the SAST range filter.

## [2026-04-23] - Bedrock retry list: skip tasks with no `image_url` and no pending `product_images`

### Changed
- **`lambda/bedrock_analysis/list_retry_tasks/app.py`:** Exclude incomplete tasks when **`task.image_url`** is empty **unless** there is at least one **`product_images`** row with **`processed_genai = false`** for that task.

## [2026-04-23] - Bedrock: Qwen Converse smaller per-image cap + env override

### Changed
- **`lambda/bedrock_analysis/app/test_app.py`:** **`_run_qwen_converse`** uses **`BEDROCK_TEST_CONVERSE_MAX_IMAGE_BYTES`** (default **1 500 000**) instead of **`app.MAX_IMAGE_BYTES`** to avoid intermittent Qwen **`length limit exceeded`** on the full request.

## [2026-04-23] - Bedrock: test Lambda accepts root-level `task_id` (console test event)

### Fixed
- **`lambda/bedrock_analysis/app/test_app.py`:** Merge **`image_id`**, **`task_id`**, **`model_id`** from the event root when **`body`** is absent (AWS “Test” event vs API Gateway proxy shape).

## [2026-04-23] - Bedrock: test API defaults to Qwen when `model_id` omitted

### Changed
- **`lambda/bedrock_analysis/app/test_app.py`:** **`resolve_test_model_id`** — default **`qwen.qwen3-vl-235b-a22b`** (overridable with **`BEDROCK_TEST_MODEL_ID`** or body **`model_id`**).
- **`lambda/bedrock_analysis/template.yaml`:** Test Lambda env **`BEDROCK_TEST_MODEL_ID`**, IAM **`foundation-model/qwen.*`**.

## [2026-04-23] - Bedrock: test API uses dedicated `invoke_bedrock_test_model` (not prod analyzer)

### Changed
- **`lambda/bedrock_analysis/app/test_app.py`:** **`invoke_bedrock_test_model`** + **`resolve_test_model_id`** — `/analyze-image-test` never calls **`analyze_image_with_bedrock`**. **`model_id`** from body or **`BEDROCK_TEST_MODEL_ID`** or **`INFERENCE_PROFILE_ARN`**; response **`test_invoke_path`** (**`test_converse`** | **`test_invoke_model`**).

### Removed
- **`BEDROCK_TEST_ALLOW_MODEL_OVERRIDE`** — superseded by always using the test-only invoke path.

### Documentation
- **`lambda/bedrock_analysis/README.md`**, **`NewKnowledgeBase.md`**.

## [2026-04-23] - Bedrock: batch Step Function + dry-run test API

### Added
- **`lambda/bedrock_analysis/app/test_app.py`:** **`POST /analyze-image-test`** — reuses **`analyze_image_with_bedrock`**; no DB or S3 writes.
- **`lambda/bedrock_analysis/template.yaml`:** **`BedrockImageAnalysisTest`**, output **`BedrockAnalysisTestApiUrl`**.
- **`lambda/bedrock_analysis/list_bedrock_batch_groups/`:** Lists incomplete GenAI work (same SAST window as retry), groups by **`product_id`**, chunks to **`BATCH_MAX_SHELF_IMAGES`**, returns **`groups`** pages for Step Functions.
- **`lambda/bedrock_analysis/bedrock_batch_analyze/`:** One Bedrock **Claude** call per group: shared product reference + multiple shelf photos; structured **JSON** results; per-row Supabase updates (same columns as single-item analysis). Partial apply when a shelf is missing from the model output.
- **`lambda/bedrock_analysis/template.yaml`:** **`BedrockBatchWorkflow`**, **`ListBedrockBatchGroups`**, **`BedrockBatchAnalyze`**, parameters **`BedrockBatchMapMaxConcurrency`**, **`BedrockBatchMaxShelfImages`**, **`BedrockBatchListPageSize`**, outputs **`BedrockBatchWorkflowArn`**, **`ListBedrockBatchGroupsArn`**, **`BedrockBatchAnalyzeArn`**.

### Documentation
- **`lambda/bedrock_analysis/README.md`**, **`NewKnowledgeBase.md`**.

## [2026-04-22] - Bedrock: optional fallback inference profile (e.g. Sonnet 4.6 then 4.5 on throttle)

### Added
- **`lambda/bedrock_analysis/app/app.py`:** `invoke_bedrock_primary_then_fallback` — after primary profile exhausts throttle retries, one call to **`BEDROCK_FALLBACK_INFERENCE_PROFILE_ARN`**; **`BedrockUsage`** includes **`model_id`**.
- **`lambda/bedrock_analysis/template.yaml`:** Parameter **`BedrockFallbackInferenceProfileArn`**; IAM **`anthropic.claude-sonnet-4-5`** resource.

### Documentation
- **`lambda/bedrock_analysis/README.md`**, **`BEDROCK_SETUP.md`**, **`NewKnowledgeBase.md`**.

## [2026-04-21] - Bedrock analysis: backoff on InvokeModel throttling; lower retry Map default

### Changed
- **`lambda/bedrock_analysis/app/app.py`:** `bedrock-runtime` client uses adaptive retries; `invoke_bedrock_with_backoff` adds exponential backoff on `ThrottlingException` / related errors (helps Step Functions Map bursts).
- **`lambda/bedrock_analysis/template.yaml`:** `BedrockRetryMapMaxConcurrency` default **2** (was 10) to stay under typical InvokeModel RPM limits.

### Documentation
- **`lambda/bedrock_analysis/README.md`:** Throttling troubleshooting note.

## [2026-04-21] - Bedrock retry: paginate list + discard Map output (fix DataLimitExceeded)

### Fixed
- **Step Functions `States.DataLimitExceeded`:** Map state no longer stores aggregated Lambda outputs (`ResultPath: null`). Large Bedrock response bodies are no longer written into workflow state.
- **Large backlogs:** List Lambda returns pages of **`RETRY_LIST_BATCH_SIZE`** (default 500) with `has_more` / `next_offset`; the state machine loops until all rows are processed.

### Changed
- **`lambda/bedrock_analysis/list_retry_tasks/app.py`:** Offset pagination, `total_count`, `has_more`; requires `run_date` when `offset > 0`.
- **`lambda/bedrock_analysis/template.yaml`:** `ListRetryFirst` / `ListRetryNext`, `PrepareNextBatch`, env `RETRY_LIST_BATCH_SIZE`.

## [2026-04-21] - Bedrock retry workflow: configurable parallel Map concurrency

### Changed
- **`lambda/bedrock_analysis/template.yaml`:** Parameter `BedrockRetryMapMaxConcurrency` (default **5**, max **100**) controls Step Functions Map `MaxConcurrency` so multiple Bedrock analyses run in parallel; tune via Service Quotas for your model/region.
- **`lambda/bedrock_analysis/README.md`:** Document throughput limits and deploy override example.

## [2026-04-21] - Bedrock analysis: Step Functions retry for incomplete GenAI (fixed today SAST)

### Added
- **`lambda/bedrock_analysis/list_retry_tasks/`:** Lambda lists `product_images` / `task` rows where GenAI is incomplete and `task.fixed_at` falls in the current Africa/Johannesburg calendar day; caps at 500 items.
- **`lambda/bedrock_analysis/template.yaml`:** `BedrockRetryWorkflow` (Step Functions: List → Choice → Map `MaxConcurrency` 1 → `BedrockImageAnalysis`), IAM role, outputs; optional `BedrockRetryScheduleRule` when `EnableBedrockRetrySchedule=true` (cron `0 22 * * ? *` UTC).
- **`lambda/bedrock_analysis/app/app.py`:** Direct invoke path when `image_id` / `task_id` appear at event root (no API Gateway `body`); initializes `image_id` for error handling.

### Documentation
- **`lambda/bedrock_analysis/README.md`:** Retry workflow, manual start, schedule parameter.

## [2026-03-12] - Create Combined Weekly Report: use service_role key so RPC returns data

### Fixed
- **lambda/datafy_new/create_combined_weekly_report/app.py:** Lambda was calling the same RPC as the DB client but getting 0 rows. Cause: Lambda used Supabase **anon** key; the report function runs as the caller (SECURITY INVOKER), so RLS applied to the anon role returned no rows. In the DB client you run as postgres and see all data. Fix: use **service_role** key when available (SSM `/supabase/service_role`), fall back to anon with a warning. Added debug log for first RPC batch (params + row count).
- **lambda/datafy_new/template.yaml:** CreateCombinedWeeklyReport policy now allows `ssm:GetParameter` for `/supabase/service_role` so the Lambda can use the service role key.

### Required
- Add SSM parameter `/supabase/service_role` with your Supabase **service_role** key (Dashboard → Project Settings → API → service_role secret). Until this is set, the Lambda falls back to anon and may still get 0 rows.

## [2026-03-04] - Bedrock analysis: log token usage for cost tracking and CloudWatch Insights

### Added
- **lambda/bedrock_analysis/app/app.py:** After each Bedrock invocation we log one line `BedrockUsage: {"input_tokens": N, "output_tokens": M}` so CloudWatch Logs Insights can sum tokens and compute cost. Enables current-workload cost and projections (e.g. +3000 users, +10 calls/user/day).
- **lambda/bedrock_analysis/BEDROCK_SETUP.md:** Section 4.1 – token usage in logs, CloudWatch Logs Insights queries to get total input/output tokens and invocation count, and cost formula. Section 4.2 – how to project cost when adding users (extra invocations × average tokens, then apply per-1K token pricing).

### Changed
- No breaking changes.

## [2026-02-27] - Read Files: always mark import as processed on failure, log and email errors

### Changed
- **lambda/datafy_new/read_files/app.py:** On any import failure we now always set `processed=True` so the upload record is not retried indefinitely. Error is still stored in the `error` field. (1) Missing required columns: now sets processed=True, logs error, and sends admin + client email before returning. (2) Insert-batch failure: explicitly passes processed=True when updating status. (3) RPC failures (batch_update_sjreport_product_id, update_sjreport_customer_ids, update_product_customer_ids): changed from processed=False to processed=True; errors were already logged and emailed. (4) Top-level exception handler: always calls update_upload_status with processed=True (and already logged/emailed); removed redundant lock-timeout branch.

## [2026-02-23] - Bedrock analysis: finer position (center-left, center-right)

### Changed
- **lambda/bedrock_analysis/app/app.py:** POSITIONS now supports five position values: left | center-left | center | center-right | right (previously only left | center | right). Method and double-check updated accordingly.

## [2026-02-23] - Bedrock analysis: size verification, reference image primary, jars

### Changed
- **lambda/bedrock_analysis/app/app.py:** (1) SIZE MUST MATCH: When Size is specified (e.g. 100G), the model must verify the size marking on shelf items; same brand/product in 200G or 400G is NOT the target. Added size_block (injected when product_size is not N/A), WHAT TO LOOK FOR item 4 (size), METHOD/RULE updates, and DOUBLE-CHECK item 3 (size verification). (2) REFERENCE IMAGE IS PRIMARY: Trust reference image over TARGET text when they conflict. (3) Packaging form: added JAR and BOX; similar-name caution (PAKCO vs PARDO). (4) Optional product_id in request body overrides wrong product link. (5) Log Brand/Product for CloudWatch.

## [2026-02-23] - Bedrock analysis: EXIF orientation correction for upright images

### Changed
- **lambda/bedrock_analysis/app/app.py:** Added `ImageOps.exif_transpose()` in `resize_image_to_fit` so shelf and product images are always sent upright to Bedrock. Phones and cameras store orientation in EXIF; PIL does not apply it by default, so rotated images (e.g. 90° CW from portrait capture) were previously sent as-is, causing inconsistent shelf counts and positions. Now all images are corrected to upright before processing.

## [2026-02-23] - Bedrock analysis: S3 artifacts (shelf + product images), permissions

### Changed
- **lambda/bedrock_analysis/app/app.py:** S3 artifacts now save the exact images sent to the model (resized/compressed by `resize_image_to_fit`), not originals. Both `shelf_image.jpg` and `product_image.jpg` (when product has an image). `encode_image` returns `(base64, processed_bytes)`; analysis returns `_shelf_image_bytes` and `_product_image_bytes` for S3.
- **lambda/bedrock_analysis/template.yaml:** S3 artifacts policy expanded: added `s3:GetBucketLocation` and bucket ARN resource. Ensures Lambda role can write to `ArtifactsBucketName` when enabled. **Redeploy required** after this change for permissions to apply.

### Added
- **lambda/bedrock_analysis:** Optional storage of shelf image, prompt, and model response to S3 per run. Controlled by SAM parameter `ArtifactsBucketName` (default `datafy-bedrock-artifacts`). When set, each run writes to `artifacts/{image_id_or_task_id}/{timestamp}/` as `shelf_image.jpg`, `prompt.txt`, and `response.txt`. Template includes conditional S3 PutObject policy and `GENAI_ARTIFACTS_S3_BUCKET` env var. Easy to remove: redeploy with empty `ArtifactsBucketName`.

## [2026-02-20] - Bedrock shelf prompt: fix shelf count and facing count

### Changed
- **lambda/bedrock_analysis/app/app.py:** Prompt tightened to reduce wrong shelf count (e.g. 6 instead of 8) and wrong facing count (e.g. 4 instead of 3). Added SHELF COUNT section: look at full image to bottom, many images have 7–8 rows, state "exactly N shelves". Added FACING COUNT section: number units 1-2-3 and report exact count, re-count to avoid 4 when there are 3. Strengthened product rule: only list a shelf if confident it is the exact product (e.g. Fruit Cocktail), not other Rhodes variants; when in doubt do not count. DOUBLE-CHECK now includes: re-check bottom of image for more shelves, remove locations if not sure it's exact product, re-count facings (3 not 4).

## [2026-02-20] - Bedrock: switch to Claude Sonnet 4.6 (same cost as 4.5)

### Changed
- **lambda/bedrock_analysis:** Model updated from Claude Sonnet 4 (anthropic.claude-sonnet-4-20250514-v1:0) to Claude Sonnet 4.6 (anthropic.claude-sonnet-4-6). Pricing unchanged ($3 input / $15 output per 1M tokens). Updated app.py BEDROCK_MODEL_ID, template.yaml INFERENCE_PROFILE_ARN and foundation-model resource, BEDROCK_SETUP.md.

## [2026-02-20] - Bedrock shelf analysis: exact shelf count (no range), clear task columns on FALSE

### Changed
- **lambda/bedrock_analysis/app/app.py:** (1) Prompt: Report the EXACT number of shelves (e.g. 8), not a range (e.g. "7-8" is wrong). DOUBLE-CHECK asks to confirm a single integer. (2) update_analysis_result: Always set genai_facing_count, genai_position, genai_has_label, and genai_confidence on every run; use None when the model returns N/A or when result is FALSE so previous run's values are cleared and the UI does not show stale facings/position/label.

## [2026-02-20] - Bedrock shelf analysis: exact product only, no missed shelves, POSITIONS array

### Changed
- **lambda/bedrock_analysis/app/app.py:** (1) EXACT PRODUCT ONLY: Target is the exact product (e.g. "Fruit Cocktail"); same brand but different product (e.g. Peach Slices, Peach Halves, Guava) must NOT be counted. (2) Do not miss a shelf: count total visible shelves first, then scan shelf 1, 2, … up to that number. (3) POSITIONS output is a JSON array: [{"shelf", "position", "facings"}, ...]. Parser added: _parse_positions_json, _positions_to_string; response includes "positions" (array) and "position" (string for DB). (4) Six-line output format: FINAL_RESULT, CONFIDENCE, FACINGS_COUNT, POSITIONS, HAS_PI_LABEL.

## [2026-02-13] - Create Tasks bulk: timestamped S3 path so runs do not overwrite each other

### Changed
- **create_tasks_bulk/app.py:** S3 batches now use prefix `batches/{week_start_date}/{YYYYmmdd_HHMMSS}/` so each run has a unique path. Removed the logic that deleted existing objects under `batches/{date}/` before upload; multiple runs for the same date no longer overwrite previous batch files.

## [2026-02-10] - Create Tasks bulk: clarify flow; PrepareBatches pass-through of items

### Changed
- **CREATE_TASKS_CONTEXT.md:** Clarified that CreateTasksBulk is the **trigger** (invoke this Lambda first); the Step Function does not run CreateTasksBulk as a step. For bulk runs, invoke CreateTasksBulk with `week_start_date`; it starts the state machine with `items`. Documented that starting the state machine from the console with only `week_start_date` yields empty items unless batches were created by CreateTasksBulk first.
- **create_tasks_list_batches (PrepareBatches):** When the execution input already contains `items` (e.g. when CreateTasksBulk started the state machine), pass them through instead of always listing S3. Ensures the same run’s batch list is used and avoids reliance on S3 list timing.

- **insert_tasks_batch (InsertOneBatch):** Supabase statement_timeout increased from 5min to 20min for batch upserts.
- **CreateTasksBulk:** Before uploading new batches, deletes any existing objects under `batches/{week_start_date}/` so a new import does not see old files. Added S3 ListBucket and DeleteObject permissions.

## [2026-02-10] - Create Tasks: optional bulk path via Step Function + S3 for 34k+ tasks

### Added
- **CreateTasksBulk Lambda** (`create_tasks_bulk/`): Same fetch+transform as CreateTasks, but writes batches (5,000 rows per file) to S3 and starts the Step Function instead of inserting in-process. Use this when you have 34k+ tasks to avoid the 15 min timeout; trigger it explicitly (e.g. invoke by name or a dedicated API/schedule).
- **Create Tasks workflow (Step Functions):** InsertTasksBatch (per-batch upsert from S3) then CreateTasksFinalize (store update, StartProcessQueue, success email). Used only when CreateTasksBulk is invoked.
- **InsertTasksBatch Lambda:** Reads one JSON file from S3 and upserts into Supabase `task` table (same `on_conflict` as before).
- **CreateTasksFinalize Lambda:** Runs after all batches complete; updates store issue counts, invokes StartProcessQueue, sends success notification.
- **CreateTasksStagingBucket (S3):** Holds batch JSON files; lifecycle deletes objects under `batches/` after 1 day.
- **create_tasks_workflow.asl.json:** Step Function definition (Map over batches → Finalize).

### Unchanged
- **CreateTasks** (`create_tasks/app.py`): Behaviour unchanged. Still does fetch + transform + in-process batch insert (1,000 per batch), update_store_issue_counts, invoke StartProcessQueue, success notification. Use for normal runs; BatchUpdateSjreport and any existing triggers continue to call CreateTasks as before.

## [2025-12-18] - Start Process Queue: clear store_processing_queue without truncate()

### Fixed
- In `lambda/datafy_new/start_process_queue/app.py`, replaced unsupported `supabase.table('store_processing_queue').truncate().execute()` with a safe delete-all pattern using `delete().neq('id', 0)` before calling `populate_store_process_queue`.
- This avoids the `AttributeError: 'SyncRequestBuilder' object has no attribute 'truncate'` while still ensuring the queue table is emptied before repopulation.

## [2025-12-12] - Combined Weekly Report: paginated export with 10k pages

### Changed
- In `lambda/datafy_new/create_combined_weekly_report/app.py`, align Supabase RPC pagination with the database function signature by using `page_offset` starting at `0` and `limit_count = 10000`, looping in batches until no more rows are returned.
- This ensures the combined weekly report always retrieves **all** rows in predictable 10k chunks before building the Excel file.

## [2025-12-11] - Combined Weekly Report: cron-only email with report link

### Added
- In `lambda/datafy_new/create_combined_weekly_report/app.py`, detect cron/EventBridge-style invocations (or an explicit `is_cron`/`send_email` flag) and send a client email with the public report link **only** for those scheduled runs.
- In `lambda/datafy_new/create_combined_weekly_report/email_service.py`, extend the success template to render an optional “details” block so we can safely include the generated report URL in the email body.

## [2025-12-02] - Read Files: ignore duplicate products on RPC

### Fixed
- In `lambda/datafy_new/read_files/app.py` and `app copy.py`, wrapped `insert_distinct_products` RPC to ignore Postgres duplicate-key violations (code `23505`). This prevents the entire "Read Files Processing" run from failing when products already exist.

### Impact
- Re-running imports for the same week no longer fails due to existing product rows. Subsequent RPCs (`batch_update_sjreport_product_id`, `update_sjreport_customer_ids`, `update_product_customer_ids`) still execute.

## [2025-12-02] - Read Files: increase HTTP read timeout for long RPCs

### Changed
- Increased Supabase PostgREST HTTP read timeout to 300s during client initialization in:
  - `lambda/datafy_new/read_files/app.py`
  - `lambda/datafy_new/read_files/app copy.py`
- This avoids `httpx.ReadTimeout: The read operation timed out` on long-running RPCs like `update_sjreport_customer_ids`.

### Notes
- We still send `SET statement_timeout = '0'` to disable server-side query timeout; this change addresses the client-side HTTP timeout.

## [2025-12-02] - Read Files: wrap RPCs and add params to batch update

### Changed
- Wrapped each post-processing RPC in `lambda/datafy_new/read_files/app.py` in its own try/except block to continue processing if one fails:
  - `batch_update_sjreport_product_id` loops until no more work with `{"p_batch_size": 1000, "p_hours": 1}`
  - `update_sjreport_customer_ids` loops until no more work with `{"p_batch_size": 1000, "p_hours_back": 1}`
  - `update_product_customer_ids` loops until no more work with `{"p_batch_size": 1000, "p_hours_back": 1}`
- On failure, we log, email admin, and update the upload status with the error, but we do not abort the entire run.

## [2025-12-02] - Start Process Queue: loop task customer-id updates

### Changed
- In `lambda/datafy_new/start_process_queue/app.py`, call `update_task_customer_ids` in a loop until empty with `{"p_batch_size": 1000, "p_hours_back": 1}`. On failure, notify via email but continue processing queue population.

## [2025-11-28] - Docs: Single-function SAM sync instructions

### Added
- README section showing how to build and sync only one Lambda with:
  - `sam build <LogicalId> -t lambda\datafy_new\template.yaml`
  - `sam sync --resource <LogicalId> --code`
- Notes explaining when to use `--code` vs full `sam deploy`.

## [2025-11-28] - Shared Lambda Layer for Python dependencies

### Added
- Created `CommonPythonLayer` in `lambda/datafy_new/template.yaml` with BuildMethod `python3.9` and content at `lambda/datafy_new/lambda_layer/`.
- Added `lambda/datafy_new/lambda_layer/requirements.txt` including: `requests`, `supabase`, `pandas`, `openpyxl`, `rapidfuzz`.
- Attached the layer to all functions in `lambda/datafy_new/template.yaml`.

### Changed
- Removed the above dependencies from each function’s `requirements.txt` in `lambda/datafy_new/*/requirements.txt` to avoid duplication.

### Impact
- Smaller per-function artifacts and faster `sam sync --code` iterations when only code changes.
- When layer dependencies change, rebuild with `sam build --use-container --clean` and deploy to publish a new layer version.

## [2025-11-28] - Fix Supabase client init and error handling

### Fixed
- Removed unsupported `ClientOptions(http_client=...)` from Supabase initialization in `lambda/datafy_new/read_files/app.py` to resolve `__init__() got an unexpected keyword argument 'http_client'`.
- Added safe defaults and guards in `lambda_handler` to prevent `UnboundLocalError: local variable 'file_name' referenced before assignment` during early failures.

### Notes
- We continue to disable Postgres statement timeout per invocation using:
  - `supabase.postgrest.rpc("raw_sql", {"query": "SET statement_timeout = '0';"})`
- If long-running RPCs are expected, prefer DB-side timeout configuration rather than overriding HTTP client internals.

## [2024-04-16] - Updated Manifest Freshness Check for Weekly Training Schedule

### Changed
- Modified the `check_manifest_freshness` function to use a 7-day (168 hours) default instead of 24 hours
- Updated the maximum age threshold to align with the weekly training schedule
- Enhanced logging to show both hours and days for better readability
- Updated CloudFormation template to set `MAX_AGE_HOURS` environment variable to 168

### Added
- Added age calculation in days in addition to hours
- Added more detailed logging about maximum allowed age
- Enhanced the response payload to include both hours and days information

### Technical Notes
- The function now checks if the manifest file was updated within the last 7 days instead of 24 hours
- This change aligns with the scheduled weekly training job (runs every Sunday at 1:00 AM UTC)
- The freshness threshold can be customized by modifying the `MAX_AGE_HOURS` environment variable
- The function returns a `manifest_fresh` flag that controls whether the training workflow continues

### Implementation Details
- Updated function signatures and default values to use 168 hours instead of 24
- Added `age_days` calculation and included in response
- Improved logging to include days for better human readability
- Added `max_age_days` to the response for clarity

## [2024-04-09] - Added Dataset Management Lambda Function

### Added
- Created a new Lambda function (ManageDatasetsFunction) dedicated to dataset management
- Added support for three operations:
  - Creating datasets from manifest files
  - Deleting existing datasets
  - Listing datasets in a project
- Added REST API endpoint at `/datasets` for dataset operations

### Changed
- Separated dataset management from training operations
- Created utility functions in `dataset_utils.py` for better code organization
- Added waiting functionality to verify dataset creation completion

### Technical Notes
- Dataset creation is a time-consuming operation that should not be part of the training process
- The new function handles:
  - Proper formatting of S3 URIs in manifest references
  - Waiting for dataset creation to complete
  - Automatic deletion of existing datasets before creating new ones
  - Detailed error handling for AWS API operations
- Lambda timeout set to 300 seconds to accommodate dataset operations
- New API allows for independent management of datasets without triggering training

### Implementation Details
- Created new utility functions:
  - `create_dataset_from_manifest`: Creates a new dataset from a manifest file
  - `delete_dataset`: Deletes an existing dataset
  - `wait_for_dataset_creation`: Polls until dataset creation completes
  - `list_datasets`: Lists all datasets in a project
- Added proper error handling and validation for all dataset operations

### Troubleshooting
If dataset operations fail:
1. Check S3 permissions for the manifest file
2. Verify manifest file is properly formatted and has correct content type
3. Ensure project ARN is correctly formatted with project ID
4. Check CloudWatch logs for specific error details

## [2024-04-09] - Simplified Training Approach for AWS Rekognition

### Changed
- Simplified training process to use a single consistent approach
- Now always using console datasets with AutoCreate=True for test dataset creation
- Removed complex conditional logic and multiple fallback mechanisms
- Clearer error messages for common configuration issues

### Technical Notes
- Training function no longer attempts to use manifest files directly for training
- Removed dataset detection and fallback mechanisms
- This approach requires initial setup in the AWS Console:
  1. Create a project in AWS Rekognition Custom Labels console
  2. Upload initial training images and label them
  3. The Lambda function will automatically handle test dataset creation
- For projects with existing test datasets, the AutoCreate parameter is ignored by AWS
- This provides a more predictable and reliable training process
- Project ARN format: `arn:aws:rekognition:eu-west-1:587594388832:project/Datafy/1725357683732`

### Implementation Details
- Modified `start_rekognition_training` function to always use console datasets
- Simplified error handling for clearer user feedback
- Removed manifest-based training options and fallbacks
- Added additional logging for training job initiation

### Troubleshooting
If training fails:
1. Ensure the project is correctly set up in AWS Rekognition Console
2. Check that training images and labels are properly configured in the console
3. Verify the PROJECT_ARN has the correct format with project ID
4. Look for specific error messages in the Lambda logs

## [2024-04-08] - Manifest Format Discrepancy Between AWS Console and API

### Issue
- The Lambda function encounters "Manifest is not valid" errors when using the API directly
- The same manifest works when manually creating a dataset through the AWS Console
- AWS Console appears to be more lenient with manifest validation or performs additional corrections
- Console UI shows image labeling errors but still allows dataset creation

### Solution Options
1. **Manual Dataset Creation**
   - Create the dataset manually through AWS Console using the manifest
   - Fix labeling errors shown in the console UI by:
     - Selecting "Start labelling" for each image with errors
     - Reassigning labels or redrawing bounding boxes as needed
   - Once dataset is created, use it for training instead of direct manifest reference

2. **Manifest Validation Enhancements**
   - Modify validation function to match AWS Console's more lenient validation
   - Add checks for "ERROR_NO_VALID_ANNOTATIONS" and "ERROR_NO_VALID_LABEL_ATTRIBUTES"
   - Address common issues:
     - Bounding boxes might be too small or outside image boundaries
     - Labels might be missing required attributes
     - Classes need to have valid mappings

### Technical Notes
- API-based training via CreateProjectVersion has stricter validation than Console
- Console dataset creation appears to do additional correction or validation
- Common manifest errors seen in Console:
  - Missing annotations (ERROR_NO_VALID_ANNOTATIONS)
  - Invalid label attributes (ERROR_NO_VALID_LABEL_ATTRIBUTES)
  - These can be fixed manually through the Console interface

### Troubleshooting
1. If programmatic training fails with "manifest is not valid":
   - Try creating a dataset manually through the AWS Console
   - Use the S3 URL: `s3://datafy-rekognition-training-data-dev/manifest.jsonl`
   - Fix labeling errors shown in the Console UI
   - Use the created dataset for training instead of direct manifest reference

2. For manual dataset creation:
   - Select "Import images labeled by SageMaker Ground Truth" option
   - Enter the manifest location: `s3://datafy-rekognition-training-data-dev/manifest.jsonl`
   - Review and fix any labeling errors identified by the Console
   - Create dataset and use for training

## [2024-04-08] - Project ARN Format Fix - RESOLVED

### Fixed
- Fixed the format of the Rekognition project ARN in the training function
- Updated project ARN to use the specific project ID `1725357683732`
- Set the ARN to use the exact project name "Datafy" instead of dynamic project name
- Confirmed successful training with ARN format: `arn:aws:rekognition:eu-west-1:587594388832:project/Datafy/1725357683732`
- Resolved the "ResourceNotFoundException" error

### Technical Notes
- AWS Rekognition project ARNs must follow this format:
  `arn:aws:rekognition:region:account-id:project/project-name/project-id`
- The project ID at the end must match the existing project in AWS Rekognition
- The function now automatically sets the project ID to `1725357683732` to match the existing project
- Fixed the "ResourceNotFoundException" error by ensuring the correct project ID is used
- Added logging to track ARN format changes

### Implementation Details
- Updated the PROJECT_ARN environment variable to use "Datafy" instead of the variable project name
- Added the fix_project_arn function to automatically append the correct project ID
- Modified the lambda_handler to use the corrected ARN
- Added comprehensive logging to troubleshoot any future ARN issues

### Troubleshooting
If training fails with "ResourceNotFoundException" about the project not existing:
1. Check that the PROJECT_ARN environment variable has the correct format
2. Verify the ARN includes the correct project ID at the end: `1725357683732`
3. Check logs to see if automatic format correction was applied
4. If automatic correction isn't working, manually update the PROJECT_ARN to:
   `arn:aws:rekognition:eu-west-1:587594388832:project/Datafy/1725357683732`

## [2024-04-08] - URL Encoded File Paths Fix

### Fixed
- Added URL decoding for S3 file paths that contain URL-encoded characters (`+`, `%20`, etc.)
- Fixed processing of XML files with special characters in their paths
- Ensured proper handling of encoded paths in S3 event notifications

### Technical Notes
- S3 event notifications often contain URL-encoded paths (e.g., spaces as `+` or `%20`)
- Added `urllib.parse.unquote_plus()` to decode these paths before processing
- This fix ensures files with spaces or special characters in their names can be properly accessed

## [2024-04-07] - Manifest Validation Enhancements

### Added
- Added `validate_manifest_file` function that:
  - Checks for the correct content type (application/x-amazon-s3-object-manifest)
  - Validates each line of the manifest file for proper JSON formatting
  - Fixes content type issues automatically
  - Removes invalid entries to ensure a clean manifest file
  - Provides detailed logging about validation issues

### Changed
- Modified `start_rekognition_training` function to:
  - Validate training manifest before starting training
  - Validate test manifest if provided
  - Fail fast with clear error messages about manifest issues

### Fixed
- Resolved "manifest is not valid" error by ensuring:
  - Correct content type is applied to manifest files
  - Only valid entries are included in the manifest
  - Proper JSONL formatting with newlines

### Technical Notes
- Manifest file must use content type `application/x-amazon-s3-object-manifest`
- Each entry must include:
  - Valid `source-ref` field starting with `s3://`
  - `bounding-box` section with annotations
  - `bounding-box-metadata` section with class map and other metadata
- Validation logs provide detailed information about any issues
- The system now automatically fixes content type issues where possible

## [2024-04-07] - Training Process Improvements

### Changed
- Updated training process to use external manifest files directly instead of creating datasets
- Modified `start_rekognition_training` function to:
  - Use TrainingData and TestingData fields with manifest files
  - Allow auto-creation of test dataset from training data
  - Add proper waiting mechanism for training completion
  - Improve logging of training parameters and status
  - Always include required OutputConfig with default values

### Fixed
- Resolved "project already has associated datasets" error by:
  - Removing dataset creation/deletion logic
  - Using manifest files directly in training configuration
  - Properly handling test data configuration
- Fixed "Missing required parameter: OutputConfig" error by:
  - Making OutputConfig a required parameter
  - Using training bucket as default output location
  - Creating organized output folders per version

### Technical Notes
- Training Configuration Format:
```json
{
    "ProjectArn": "project_arn",
    "VersionName": "version_name",
    "OutputConfig": {
        "S3Bucket": "output_bucket",
        "S3KeyPrefix": "training_output/version-name"
    },
    "TrainingData": {
        "Assets": [{
            "GroundTruthManifest": {
                "S3Object": {
                    "Bucket": "bucket",
                    "Name": "manifest.jsonl"
                }
            }
        }]
    },
    "TestingData": {
        "AutoCreate": true
    }
}
```
- AWS Rekognition will automatically split training data if no test data is provided
- Training can take 30 minutes to 24 hours to complete
- Training output is organized in folders by version name

### Known Issues and Solutions
1. **Training Configuration**
   - Ensure manifest files are properly formatted
   - Use correct S3 paths for manifest files
   - Allow sufficient time for training completion
   - Verify output bucket permissions

2. **Testing Data**
   - Can be auto-created from training data
   - Or specify separate test manifest
   - Recommended to have dedicated test dataset for better evaluation

## [2024-04-07] - URI Overencoding and Manifest Filename Fixes

### Fixed
- Fixed excessive URL encoding in S3 URIs
  - Added detection and repair for over-encoded paths (multiple %25 sequences)
  - Simplified encoding to only replace spaces with %20
- Ensured manifest file has the exact name AWS Rekognition requires (`manifest.jsonl`)
- Prevented creation of test manifest when a real manifest already exists

### Technical Notes
- AWS Rekognition specifically looks for a file named `manifest.jsonl` - this filename must be exact
- S3 URIs should have minimal encoding (only spaces replaced with %20)
- Excessive encoding (like %25%25%25...) can cause AWS Rekognition to reject the manifest

## [2024-04-03] - S3 URI and Manifest Fixes

### Fixed
- Changed manifest filename from `manifest.json` to `manifest.jsonl` to match AWS Rekognition requirements
- Changed S3 manifest content type from `application/x-jsonlines` to `application/x-amazon-s3-object-manifest-jsonl`
- Fixed S3 URI handling:
  - Properly URL encode spaces and special characters in paths
  - Maintain correct s3:// prefix
  - Handle each path component separately
  - Example: `s3://bucket/path with spaces/file.jpg` becomes `s3://bucket/path%20with%20spaces/file.jpg`
- Normalized bounding box coordinates to be between 0 and 1
- Fixed field ordering in manifest entries

### Added
- Validation check for normalized coordinates (0 to 1)
- More detailed error logging for manifest validation failures
- Better source-ref validation and error messages
- Logging of final source-ref URLs for debugging

### Changed
- Updated manifest entry structure to match AWS requirements
- Modified bounding box validation:
  - Minimum width: 6 pixels
  - Minimum height: 6 pixels
  - Minimum area: 100 pixels²
  - Better logging of skipped boxes

### Technical Notes
- Manifest File Requirements:
  - Must be named `manifest.jsonl` (not .json)
  - Must use content type `application/x-amazon-s3-object-manifest-jsonl`
  - Each line must be a complete, valid JSON object
  - No empty lines allowed between entries
- S3 URI Format Requirements:
  - Must start with exactly "s3://"
  - Spaces and special characters must be URL encoded
  - Forward slashes must be preserved
  - Example: `s3://bucket/path%20name/file.jpg`
- Coordinates must be normalized
- Field order matters for AWS Rekognition compatibility

### Known Issues and Solutions
1. **Invalid Source-ref URLs**
   - Problem: Spaces and special characters in S3 paths
   - Solution: URL encode path components while preserving structure
   - Example fix:
     ```python
     bad_url = "s3://bucket/path with spaces/file.jpg"
     good_url = "s3://bucket/path%20with%20spaces/file.jpg"
     ```

2. **Validation Process**
   - The system validates each entry before adding to manifest
   - Invalid entries are logged and skipped
   - Common validation failures:
     - Invalid source-ref format
     - Non-normalized coordinates
     - Missing required fields
   - Check logs for "ERROR" messages

### Troubleshooting
1. If training fails with "manifest is not valid":
   - Verify manifest filename is exactly `manifest.jsonl`
   - Check source-ref URLs in manifest
   - Verify URL encoding of paths
   - Ensure correct s3:// prefix
   - Delete existing manifest and reprocess if needed
   - Command to delete old manifest:
     ```bash
     aws s3 rm s3://datafy-rekognition-training-data-dev/manifest.json
     aws s3 rm s3://datafy-rekognition-training-data-dev/manifest.jsonl
     ```

2. If source-ref validation fails:
   - Check for proper URL encoding
   - Verify s3:// prefix
   - Look for double slashes
   - Check for unencoded spaces or special characters

## [2024-04-03] - Manifest Handling Fixes

### Fixed
- Changed S3 manifest content type from `application/x-jsonlines` to `application/x-amazon-s3-object-manifest-jsonl` to match AWS Rekognition requirements
- Normalized bounding box coordinates to be between 0 and 1 (divided by image width/height) instead of using absolute pixel values
- Added validation to ensure coordinates are properly normalized
- Fixed field ordering in manifest entries to match AWS requirements exactly:
  - Moved `annotations` before `image_size` in the bounding-box structure
  - Moved `objects` to the end of bounding-box-metadata

### Added
- Validation check for normalized coordinates (0 to 1) in `validate_manifest_entry()`
- More detailed error logging for manifest validation failures

### Changed
- Updated manifest entry structure to match AWS requirements:
```json
{
    "source-ref": "s3://bucket/image.jpg",
    "bounding-box": {
        "annotations": [
            {
                "class_id": 0,
                "left": 0.2,  // normalized coordinates
                "top": 0.2,   // normalized coordinates
                "width": 0.6, // normalized coordinates
                "height": 0.6 // normalized coordinates
            }
        ],
        "image_size": [{
            "width": 500,
            "height": 500,
            "depth": 3
        }]
    },
    "bounding-box-metadata": {
        "class-map": {"0": "class-name"},
        "type": "groundtruth/object-detection",
        "human-annotated": "yes",
        "creation-date": "timestamp",
        "job-name": "job-name",
        "objects": [{"confidence": 1}]
    }
}
```

### Technical Notes
- Coordinates must be normalized by dividing by image dimensions:
  - `left = xmin / image_width`
  - `top = ymin / image_height`
  - `width = (xmax - xmin) / image_width`
  - `height = (ymax - ymin) / image_height`
- The manifest file must use the content type `application/x-amazon-s3-object-manifest-jsonl`
- Each line in the manifest must be a complete, valid JSON object
- Field order matters for AWS Rekognition compatibility

### Known Issues and Solutions
1. **Existing Invalid Manifests**
   - Problem: Existing manifest entries may contain non-normalized coordinates (values > 1)
   - Solution: Delete the existing manifest.json file and reprocess all XML files to create a new manifest
   - Command to delete manifest:
     ```bash
     aws s3 rm s3://datafy-rekognition-training-data-dev/manifest.json
     ```

2. **Validation Process**
   - The system validates each entry before adding to manifest
   - Invalid entries are logged and skipped
   - Common validation failures:
     - Non-normalized coordinates (values not between 0 and 1)
     - Missing required fields
     - Invalid field order
   - Check logs for "ERROR" messages to identify specific validation failures

3. **Manifest Statistics**
   - The system logs manifest statistics after each update:
     - Total entries
     - Unique classes
     - Total annotations
   - Use these statistics to verify manifest integrity

4. **AWS Rekognition Requirements**
   - Minimum Requirements:
     - At least 1 image per class
     - Recommended: 10+ images per class for better model performance
     - Each image should have clear, well-defined annotations
   - S3 URI Format:
     - Must start with "s3://"
     - No double slashes after the prefix
     - Example: "s3://bucket-name/path/to/image.jpg"

### Troubleshooting
1. If training fails with "manifest is not valid":
   - Check logs for validation errors
   - Verify all coordinates are normalized (between 0 and 1)
   - Ensure manifest content type is correct
   - Delete existing manifest and reprocess if needed
   - Check sample manifest entry in logs for format issues
   - Verify S3 URIs are correctly formatted
   - Ensure you have enough training samples per class

2. If coordinates are incorrect:
   - Verify XML file dimensions match actual image dimensions
   - Check normalization calculation in `convert_xml_to_rekognition_format()`
   - Log intermediate values for debugging

3. Debug Steps:
   - Review sample manifest entry in logs
   - Check source-ref format warnings
   - Verify class distribution
   - Ensure all required metadata fields are present
   - Validate JSON format of each entry 

## [2024-04-07] - Manifest Format and Content Type Updates

### Fixed
- Updated manifest content type to match AWS documentation exactly: `application/x-amazon-s3-object-manifest`
- Improved JSONL format handling:
  - Each line is a complete JSON object with no pretty printing
  - Added newline after each entry
  - No trailing whitespace
- Added verification steps after manifest creation/update
- Fixed excessive URL encoding in S3 URIs
  - Added detection and repair for over-encoded paths (multiple %25 sequences)
  - Simplified encoding to only replace spaces with %20
- Ensured manifest file has the exact name AWS Rekognition requires (`manifest.jsonl`)

### Technical Notes
- AWS Rekognition manifest requirements:
  - Content type must be exactly `application/x-amazon-s3-object-manifest`
  - Each line must be a complete, valid JSON object
  - Lines must be separated by newlines
  - No trailing whitespace or pretty printing
  - Filename must be exactly `manifest.jsonl`
- S3 URIs should have minimal encoding (only spaces replaced with %20)
- Manifest format example:
  ```jsonl
  {"source-ref":"s3://bucket/image1.jpg","bounding-box":{"annotations":[...]}}
  {"source-ref":"s3://bucket/image2.jpg","bounding-box":{"annotations":[...]}}
  ```

### Troubleshooting
1. If training fails with "manifest is not valid":
   - Verify content type is exactly `application/x-amazon-s3-object-manifest`
   - Check manifest format:
     - Each line should be a complete JSON object
     - No pretty printing or extra whitespace
     - Lines end with newline character
   - Delete existing manifest and reprocess:
     ```bash
     aws s3 rm s3://datafy-rekognition-training-data-dev/manifest.jsonl
     ``` 

## [2024-04-07] - Enhanced Manifest Validation and Content Type Fix

### Fixed
- Updated manifest content type to `application/x-amazon-s3-object-manifest-jsonl` to match AWS requirements
- Added thorough validation of manifest entries:
  - Validates bounding-box structure and required fields
  - Validates bounding-box-metadata structure and required fields
  - Ensures type is 'groundtruth/object-detection'
  - Verifies number of objects matches annotations
- Improved JSONL formatting:
  - Ensures each line ends with a newline
  - Removes trailing whitespace
  - Adds final newline to file
- Added verification step after updating manifest

### Technical Notes
- Manifest Requirements:
  - Content type must be exactly `application/x-amazon-s3-object-manifest-jsonl`
  - Each line must be a valid JSON object with required fields:
    - source-ref (s3:// URL)
    - bounding-box (with annotations and image_size)
    - bounding-box-metadata (with objects, class-map, type, etc.)
  - Type must be 'groundtruth/object-detection'
  - Number of objects must match number of annotations
  - Each line must end with a newline
  - File must end with a newline

### Troubleshooting
If training fails with "manifest is not valid":
1. Check logs for validation errors
2. Verify content type is exactly `application/x-amazon-s3-object-manifest-jsonl`
3. Ensure all required fields are present
4. Check JSONL formatting (newlines, no trailing whitespace)
5. Verify type and object count match 

## [2024-04-07] - Fixed Nested Paths in Manifest

### Fixed
- Added automatic fixing of nested paths in manifest source-ref URLs
- Improved JSON handling and formatting in manifest entries
- Fixed content type and newline handling

### Technical Notes
- Manifest Path Requirements:
  - Source-ref URLs must not contain nested product folders
  - All paths are flattened to the format: `s3://bucket/product-name/filename`
  - Example fix: 
    - From: `s3://bucket/product1/product2/image.jpg`
    - To: `s3://bucket/product2/image.jpg`

### Troubleshooting
If training fails with "manifest is not valid":
1. Check logs for path fixing messages
2. Verify source-ref URLs are properly formatted
3. Ensure no nested product folders exist
4. Check JSON formatting and newlines 

## [2024-04-07] - S3 Path Validation Fix

### Changed
- Removed automatic path modification that was incorrectly "fixing" nested paths
- Added validation to verify files exist in S3 at their original paths
- Preserved original folder structure from source data

### Fixed
- Stopped modifying source-ref paths in manifest
- Added existence checks for S3 files
- Improved error logging for missing files

### Technical Notes
- Manifest entries must point to existing files in S3
- Original folder structure is preserved
- Invalid entries (pointing to non-existent files) are removed from manifest

### Troubleshooting
If training fails with "manifest is not valid":
1. Check logs for "File not found" warnings
2. Verify all referenced files exist in S3
3. Ensure paths in manifest match actual S3 paths
4. Check JSON formatting and content type

## [2024-04-03] - Manifest Handling Fixes

### Fixed
- Changed S3 manifest content type from `application/x-jsonlines` to `application/x-amazon-s3-object-manifest-jsonl` to match AWS Rekognition requirements
- Normalized bounding box coordinates to be between 0 and 1 (divided by image width/height) instead of using absolute pixel values
- Added validation to ensure coordinates are properly normalized
- Fixed field ordering in manifest entries to match AWS requirements exactly:
  - Moved `annotations` before `image_size` in the bounding-box structure
  - Moved `objects` to the end of bounding-box-metadata

### Added
- Validation check for normalized coordinates (0 to 1) in `validate_manifest_entry()`
- More detailed error logging for manifest validation failures

### Changed
- Updated manifest entry structure to match AWS requirements:
```json
{
    "source-ref": "s3://bucket/image.jpg",
    "bounding-box": {
        "annotations": [
            {
                "class_id": 0,
                "left": 0.2,  // normalized coordinates
                "top": 0.2,   // normalized coordinates
                "width": 0.6, // normalized coordinates
                "height": 0.6 // normalized coordinates
            }
        ],
        "image_size": [{
            "width": 500,
            "height": 500,
            "depth": 3
        }]
    },
    "bounding-box-metadata": {
        "class-map": {"0": "class-name"},
        "type": "groundtruth/object-detection",
        "human-annotated": "yes",
        "creation-date": "timestamp",
        "job-name": "job-name",
        "objects": [{"confidence": 1}]
    }
}
```

### Technical Notes
- Coordinates must be normalized by dividing by image dimensions:
  - `left = xmin / image_width`
  - `top = ymin / image_height`
  - `width = (xmax - xmin) / image_width`
  - `height = (ymax - ymin) / image_height`
- The manifest file must use the content type `application/x-amazon-s3-object-manifest-jsonl`
- Each line in the manifest must be a complete, valid JSON object
- Field order matters for AWS Rekognition compatibility

### Known Issues and Solutions
1. **Existing Invalid Manifests**
   - Problem: Existing manifest entries may contain non-normalized coordinates (values > 1)
   - Solution: Delete the existing manifest.json file and reprocess all XML files to create a new manifest
   - Command to delete manifest:
     ```bash
     aws s3 rm s3://datafy-rekognition-training-data-dev/manifest.json
     ```

2. **Validation Process**
   - The system validates each entry before adding to manifest
   - Invalid entries are logged and skipped
   - Common validation failures:
     - Non-normalized coordinates (values not between 0 and 1)
     - Missing required fields
     - Invalid field order
   - Check logs for "ERROR" messages to identify specific validation failures

3. **Manifest Statistics**
   - The system logs manifest statistics after each update:
     - Total entries
     - Unique classes
     - Total annotations
   - Use these statistics to verify manifest integrity

4. **AWS Rekognition Requirements**
   - Minimum Requirements:
     - At least 1 image per class
     - Recommended: 10+ images per class for better model performance
     - Each image should have clear, well-defined annotations
   - S3 URI Format:
     - Must start with "s3://"
     - No double slashes after the prefix
     - Example: "s3://bucket-name/path/to/image.jpg"

### Troubleshooting
1. If training fails with "manifest is not valid":
   - Check logs for validation errors
   - Verify all coordinates are normalized (between 0 and 1)
   - Ensure manifest content type is correct
   - Delete existing manifest and reprocess if needed
   - Check sample manifest entry in logs for format issues
   - Verify S3 URIs are correctly formatted
   - Ensure you have enough training samples per class

2. If coordinates are incorrect:
   - Verify XML file dimensions match actual image dimensions
   - Check normalization calculation in `convert_xml_to_rekognition_format()`
   - Log intermediate values for debugging

3. Debug Steps:
   - Review sample manifest entry in logs
   - Check source-ref format warnings
   - Verify class distribution
   - Ensure all required metadata fields are present
   - Validate JSON format of each entry 

## [2024-04-07] - Manifest Format and Content Type Updates

### Fixed
- Updated manifest content type to match AWS documentation exactly: `application/x-amazon-s3-object-manifest`
- Improved JSONL format handling:
  - Each line is a complete JSON object with no pretty printing
  - Added newline after each entry
  - No trailing whitespace
- Added verification steps after manifest creation/update
- Fixed excessive URL encoding in S3 URIs
  - Added detection and repair for over-encoded paths (multiple %25 sequences)
  - Simplified encoding to only replace spaces with %20
- Ensured manifest file has the exact name AWS Rekognition requires (`manifest.jsonl`)

### Technical Notes
- AWS Rekognition manifest requirements:
  - Content type must be exactly `application/x-amazon-s3-object-manifest`
  - Each line must be a complete, valid JSON object
  - Lines must be separated by newlines
  - No trailing whitespace or pretty printing
  - Filename must be exactly `manifest.jsonl`
- S3 URIs should have minimal encoding (only spaces replaced with %20)
- Manifest format example:
  ```jsonl
  {"source-ref":"s3://bucket/image1.jpg","bounding-box":{"annotations":[...]}}
  {"source-ref":"s3://bucket/image2.jpg","bounding-box":{"annotations":[...]}}
  ```

### Troubleshooting
1. If training fails with "manifest is not valid":
   - Verify content type is exactly `application/x-amazon-s3-object-manifest`
   - Check manifest format:
     - Each line should be a complete JSON object
     - No pretty printing or extra whitespace
     - Lines end with newline character
   - Delete existing manifest and reprocess:
     ```bash
     aws s3 rm s3://datafy-rekognition-training-data-dev/manifest.jsonl
     ``` 

## [2024-04-07] - Enhanced Manifest Validation and Content Type Fix

### Fixed
- Updated manifest content type to `application/x-amazon-s3-object-manifest-jsonl` to match AWS requirements
- Added thorough validation of manifest entries:
  - Validates bounding-box structure and required fields
  - Validates bounding-box-metadata structure and required fields
  - Ensures type is 'groundtruth/object-detection'
  - Verifies number of objects matches annotations
- Improved JSONL formatting:
  - Ensures each line ends with a newline
  - Removes trailing whitespace
  - Adds final newline to file
- Added verification step after updating manifest

### Technical Notes
- Manifest Requirements:
  - Content type must be exactly `application/x-amazon-s3-object-manifest-jsonl`
  - Each line must be a valid JSON object with required fields:
    - source-ref (s3:// URL)
    - bounding-box (with annotations and image_size)
    - bounding-box-metadata (with objects, class-map, type, etc.)
  - Type must be 'groundtruth/object-detection'
  - Number of objects must match number of annotations
  - Each line must end with a newline
  - File must end with a newline

### Troubleshooting
If training fails with "manifest is not valid":
1. Check logs for validation errors
2. Verify content type is exactly `application/x-amazon-s3-object-manifest-jsonl`
3. Ensure all required fields are present
4. Check JSONL formatting (newlines, no trailing whitespace)
5. Verify type and object count match 

## [2024-04-07] - Fixed Nested Paths in Manifest

### Fixed
- Added automatic fixing of nested paths in manifest source-ref URLs
- Improved JSON handling and formatting in manifest entries
- Fixed content type and newline handling

### Technical Notes
- Manifest Path Requirements:
  - Source-ref URLs must not contain nested product folders
  - All paths are flattened to the format: `s3://bucket/product-name/filename`
  - Example fix: 
    - From: `s3://bucket/product1/product2/image.jpg`
    - To: `s3://bucket/product2/image.jpg`

### Troubleshooting
If training fails with "manifest is not valid":
1. Check logs for path fixing messages
2. Verify source-ref URLs are properly formatted
3. Ensure no nested product folders exist
4. Check JSON formatting and newlines 

## [2024-04-07] - Fixed Path Handling and File Organization

### Changed
- Modified path handling in manifest entries:
  - Now using flattened paths with only parent folder and filename
  - Example: `s3://bucket/parent_folder/image.jpg` instead of nested paths
  - Added automatic file copying to flattened locations
  - Improved file existence validation

### Added
- Automatic file organization:
  - Files are now copied to flattened locations before manifest creation
  - Original files are preserved in their original locations
  - Files are only copied if they don't already exist in the flattened location

### Fixed
- Resolved issue with nested paths in source-ref URLs
- Improved S3 file validation to handle path changes
- Added file copying to ensure files exist at referenced locations

### Technical Notes
- Manifest entries now use simplified paths:
  - Format: `s3://bucket/parent_folder/filename.jpg`
  - Parent folder is preserved for organization
  - Files are physically copied to match manifest paths
- File validation checks both flattened and original paths
- Detailed logging added for path transformations and file copying

### Troubleshooting
If training fails with "manifest is not valid":
1. Check logs for path transformation messages
2. Verify files exist in S3 at the flattened paths
3. Ensure manifest entries use correct path format
4. Check JSON formatting and content type

## [2024-04-10] - Optimized Training Function

### Changed
- Optimized the training function to use only the successful minimal parameters approach
- Removed all fallback mechanisms that were unnecessary
- Simplified the code for better readability and maintenance
- The function now directly uses the approach that worked consistently

### Technical Notes
- The minimal parameters approach uses just:
  - ProjectArn
  - VersionName 
  - OutputConfig with S3Bucket and S3KeyPrefix
- This approach works correctly with console-created datasets
- The function is now more efficient and has less error-prone code

### Implementation Details
- Removed unnecessary attempts and fallbacks
- Improved documentation on how the function works
- Ensured consistent logging of training process
- Maintained error handling for common error cases