# Datafy AWS Rekognition Custom Labels Project

This project implements AWS Lambda functions for managing AWS Rekognition Custom Labels datasets, models, and inference operations for the Datafy project.

## Project Structure

```
lambda/Rekognition/
├── functions/                  # Lambda function code
│   ├── check_dataset_status/   # Checks if datasets have been created and are ready
│   ├── check_manifest_freshness/ # Checks if manifest file is recent enough
│   ├── check_model_availability/ # Verifies if a model is in running state
│   ├── check_pending_images/   # Checks if there are images to process
│   ├── check_training_status/  # Checks status of training models
│   ├── create_dataset/         # Creates datasets from manifest files
│   ├── create_manifest/        # Creates manifest files from raw data
│   ├── create_model/           # Creates a model from a dataset
│   ├── delete_datasets/        # Deletes datasets when needed
│   ├── manage_datasets/        # Manages dataset operations
│   ├── split_manifest/         # Splits manifest files for train/test
│   ├── start_model/            # Starts a trained model for inference
│   ├── start_training/         # Starts model training
│   ├── stop_model/             # Stops a running model to save costs
│   └── analyze_image/          # Processes images using a running model
├── statemachine/               # Step Functions state machine definitions
├── template.yaml               # CloudFormation/SAM template
└── REKOGNITION_PROJECT_GUIDE.md # Detailed project guide
```

## Lambda Functions

### Dataset Management

- **create_dataset**: Creates train and test datasets from manifest files in S3
- **check_dataset_status**: Verifies datasets have been created and are in READY state
- **delete_datasets**: Handles dataset deletion with proper error handling
- **manage_datasets**: API endpoint for dataset operations
- **check_manifest_freshness**: Checks if manifest files are recent enough (less than 24 hours old) before training

### Model Management

- **create_model**: Creates a new model from a dataset
- **start_training**: Initiates model training
- **check_training_status**: Monitors training progress
- **start_model**: Starts a trained model for inference use
- **stop_model**: Stops a running model to save costs
- **check_pending_images**: Checks if there are unprocessed images before starting a model
- **check_model_availability**: Verifies if a started model is in RUNNING state and ready to use
- **analyze_image**: Processes images using a running Rekognition model

### Data Preparation

- **create_manifest**: Creates manifest files from source data
- **split_manifest**: Splits manifests into train/test sets

## State Machine Workflows

### Image Processing Workflow

The Rekognition state machine processes unprocessed images using a trained model. This workflow:
1. Checks for pending images
2. Starts the model if needed
3. Analyzes images with the running model
4. Stops the model to save costs

**Scheduled Execution**: This workflow runs automatically every day at 2:00 AM UTC to process any pending images.

### Model Training Workflow

The training state machine orchestrates the model training process. This workflow:
1. Checks if the manifest file is fresh enough (< 24 hours old)
2. Creates or refreshes datasets from the manifest
3. Initiates model training
4. Monitors training progress until completion

**Scheduled Execution**: This workflow runs automatically every Sunday at 1:00 AM UTC to retrain the model weekly with the latest data.

## Key Requirements

1. All project ARNs must use the specific project ID: `1725357683732`
2. The project name must be `Datafy`
3. Dataset ARNs have a specific format that must be followed
4. API methods require specific approaches due to AWS SDK limitations

## Training Process Guide

### Important Requirements for Training

1. **Project ID Consistency**: All operations related to training must use the project ID `1725357683732`. This is enforced by the `fix_project_arn` function in the codebase.

2. **ARN Formats**: The following ARN formats must be used:
   - Project ARN: `arn:aws:rekognition:[region]:[account]:project/Datafy/1725357683732`
   - Dataset ARN: `arn:aws:rekognition:[region]:[account]:project/Datafy/dataset/[train|test]/1725357683732`
   - Model ARN: `arn:aws:rekognition:[region]:[account]:project/Datafy/version/[version-name]/1725357683732`

### Training Workflow

1. **Pre-training Validation**:
   - `check_manifest_freshness` verifies manifest file is recent (< 24 hours old)
   - Prevents training with outdated data
   - Returns `manifest_fresh` flag to control workflow progression

2. **Data Preparation**:
   - Upload labeled data to S3
   - `create_manifest` function generates manifest files
   - `split_manifest` function divides data into training and test sets

3. **Dataset Creation**:
   - `create_dataset` function uses manifest files to create datasets
   - Ensures ARNs use the correct project ID format
   - `check_dataset_status` verifies datasets are in READY state

4. **Model Training**:
   - `start_training` initiates the training process
   - Training job runs asynchronously in AWS Rekognition
   - `check_training_status` monitors training progress

5. **Model Evaluation and Deployment**:
   - Upon training completion, model is evaluated
   - Model metrics are stored (if Supabase integration is configured)
   - Model becomes available for inference

### Troubleshooting Training Issues

1. **ARN Format Errors**: If training fails, verify that all ARNs contain the correct project ID (`1725357683732`)
2. **Manifest File Issues**: Ensure manifest files are properly formatted and accessible
3. **Manifest Freshness**: If training is skipped, check the last modified date of manifest files in S3
4. **Training Resource Limits**: Check AWS service limits if training jobs are queued for long periods
5. **Dataset Status**: Confirm datasets are in READY state before starting training

## Common Issues and Solutions

See the [REKOGNITION_PROJECT_GUIDE.md](./REKOGNITION_PROJECT_GUIDE.md) for detailed information on:
- ARN formats and validation
- API method limitations
- Environment variables
- Best practices
- Troubleshooting steps

## Deployment

This project uses AWS SAM/CloudFormation for deployment. The template defines:
- Lambda functions with appropriate IAM permissions
- S3 buckets for source and training data
- Step Functions state machines for orchestration
- EventBridge scheduled rules for automated execution
- SSM parameters for configuration

## Scheduled Executions

The system includes two EventBridge scheduled rules:

1. **Daily Image Processing**:
   - Schedule: Every day at 2:00 AM UTC
   - Target: Rekognition state machine
   - Purpose: Automatically process any pending images once per day

2. **Weekly Model Training**:
   - Schedule: Every Sunday at 1:00 AM UTC
   - Target: Rekognition training state machine
   - Purpose: Retrain the model weekly with the latest data
   - Note: Training will only proceed if the manifest file is fresh (less than 24 hours old)

These schedules can be modified by updating the CloudFormation template.

## Development Guidelines

1. Follow consistent error handling patterns
2. Use utility functions for common operations
3. Maintain detailed logging
4. Test thoroughly before deployment
5. Review the REKOGNITION_PROJECT_GUIDE.md for important details