# AWS Rekognition Custom Labels Project Guide

This document captures essential information, requirements, and common issues when working with AWS Rekognition Custom Labels in this project.

## Project Configuration

### Project ID and ARN Format

- **Project ID**: `1725357683732` - This specific project ID must be used consistently.
- **Project Name**: `Datafy` - Use this exact name for consistency.

### ARN Formats

- **Project ARN Format**: `arn:aws:rekognition:[region]:[account]:project/Datafy/1725357683732`
- **Dataset ARN Format**: `arn:aws:rekognition:[region]:[account]:project/[project-name]/dataset/[train|test]/[project-id]`
- **Model Version ARN Format**: `arn:aws:rekognition:[region]:[account]:project/[project-name]/version/[version-name]/[project-id]`

All ARNs must follow these formats exactly. The project ID must always be `1725357683732`.

## Working with Datasets

### Dataset Creation

- Dataset creation is done through `create_dataset` function
- Use the `create_dataset_from_manifest` utility to create datasets from manifest files
- The API will return the correct dataset ARN which should be stored and reused

### Checking Dataset Status

1. Do NOT use `list_datasets` - this method doesn't exist in our version of the AWS SDK
2. Instead, use the pattern from `get_project_details`:
   - Extract project name from ARN
   - Call `describe_projects` with the project name
   - Extract dataset information from the response

### Deleting Datasets

- Use the `delete_dataset` function with proper dataset ARN
- Check for transitional states (CREATING, UPDATING, DELETING)
- Use proper error handling for ResourceNotFoundException

## Common API Issues and Solutions

### ARN Format Validation Errors

If receiving validation errors like:
```
ValidationException - 1 validation error detected: Value 'arn:aws:rekognition...' failed to satisfy constraint
```

Check:
1. The ARN format is correct (see ARN Formats section)
2. Dataset type is lowercase (`train` or `test`, not `TRAIN` or `TEST`)
3. Project ID is the correct one (`1725357683732`)

### API Method Not Found

If receiving errors like:
```
'Rekognition' object has no attribute 'list_datasets'
```

This indicates the method doesn't exist in our AWS SDK version. Use alternative methods:
- Instead of `list_datasets`, use `describe_projects` to get dataset information
- Instead of direct dataset construction, use utility functions that have been tested

### Project ARN Handling

- Always use `fix_project_arn` to ensure proper ARN format
- Do not assume the ARN from environment variables is correctly formatted
- Handle missing or incorrect ARNs with proper fallbacks

## Training Models

- Use `start_training` function with properly formatted project ARN
- Never construct dataset or model ARNs manually - always use proper API calls to get them
- Use minimal parameter approach when starting training

### Model Status Fields

The project uses two separate status fields in the `model_versions` table:

- `status`: Tracks the training status of models (TRAINING, TRAINING_COMPLETED, etc.)
- `model_status`: Tracks the deployment status of models (STARTING, RUNNING, etc.)

This separation allows the training and deployment states to be tracked independently.

#### Training Status Values (status field)
- `TRAINING` or `TRAINING_IN_PROGRESS`: Model is currently being trained
- `TRAINING_COMPLETED`: Model has completed training successfully
- `FAILED`: Training failed

#### Deployment Status Values (model_status field)
- `STARTING`: Model is being started for inference
- `RUNNING`: Model is running and available for inference
- `STOPPING`: Model is being stopped
- `STOPPED`: Model has been stopped
- `ERROR`: Error occurred during model startup or operation

### Database Schema Update

When deploying for the first time, make sure to run the schema update script:
`lambda/Rekognition/supabase_schema_update.sql`

This script adds the necessary `model_status` column to the `model_versions` table.

## Analyzing Images with Trained Models

The `analyze_image` function provides image analysis capabilities using trained models. This function:

1. Retrieves unprocessed images from the `product_images` table in Supabase
2. Finds an active model to use for analysis
3. Processes each image using the Rekognition Custom Labels API
4. Saves results and processed images back to Supabase

### Model Selection Process

The function uses a multi-tiered approach to find the appropriate model:

1. First tries to find models with `model_status='RUNNING'` (new approach)
2. Falls back to models with `status='RUNNING'` (legacy approach)
3. If no running model is found, tries models with `model_status='TRAINING_COMPLETED'`
4. Finally tries models with `status='TRAINING_COMPLETED'`

This approach ensures backward compatibility during the transition to the dual-status system.

### Required Tables

The analyze_image function requires these tables in Supabase:

- `model_versions`: Contains model information and status
- `product_images`: Stores images to be processed
- `product_ml`: Stores analysis results

If implementing the dual-status approach, ensure all tables have been updated with the schema update script.

## Environment Variables

Critical environment variables:
- `PROJECT_ARN`: ARN of the Rekognition project
- `PROJECT_NAME`: Name of the Rekognition project (defaults to "Datafy")
- `AWS_REGION`: Region for AWS resources (default fallback: `eu-west-1`)
- `AWS_ACCOUNT_ID`: AWS account ID (default fallback: `587594388832`)

Do not try to modify reserved Lambda environment variables like `AWS_REGION` directly.

## Important AWS SDK Notes

1. The AWS SDK version in use doesn't support all Rekognition Custom Labels APIs
2. Some methods might be called differently than the official documentation suggests
3. Always use tested patterns from working functions

## Best Practices

1. Log ARNs and parameters extensively for debugging
2. Use consistent error handling patterns
3. Add fallbacks where possible
4. Extract components with regex for consistency
5. Handle transitional states properly
6. Use the SSM parameter store for configuration where appropriate

## Troubleshooting Steps

If encountering issues:
1. Check ARN formats first
2. Verify dataset exists through AWS Console
3. Check logs for specific error messages
4. Try listing projects/datasets directly to see what AWS recognizes
5. Implement fallbacks for different name formats

## Lambda Function Dependencies

Each Lambda function should include these imports:
```python
import boto3
import logging
import os
import json
import re
from botocore.exceptions import ClientError
```

## Manifest File Requirements

- Must be properly formatted JSONL files
- Must include proper object locations and labels
- Train and test manifests should be separated

## Testing Recommendations

- Test dataset creation and deletion in isolation
- Verify dataset status through AWS Console
- Use minimal approach for training to avoid unnecessary complexity