# Manage Datasets Lambda Function

This Lambda function automatically creates both TRAIN and TEST datasets for AWS Rekognition Custom Labels.

## Overview

The function performs the following operations automatically:
1. Gets the training bucket and project ARN from environment variables or SSM parameters
2. Validates the manifest file located at `s3://{training_bucket}/manifest.jsonl`
3. Checks for existing TRAIN dataset, deletes it if found
4. Creates a new TRAIN dataset with the manifest file
5. Checks for existing TEST dataset, deletes it if found
6. Creates a new TEST dataset with the same manifest file
7. Waits for both dataset creations to complete

## Configuration

The function uses the following environment variables:
- `TRAINING_BUCKET`: The S3 bucket where the manifest file is stored (set directly in the CloudFormation template)
- `PROJECT_ARN`: The ARN of the Rekognition project

If these variables are not set in the environment, the function will attempt to retrieve them from SSM Parameter Store:
- `/datafy-rekognition-stack/training-bucket` for the training bucket
- `/datafy-rekognition-stack/project-name` for constructing the project ARN

## Invocation

The function is designed to be invoked directly rather than through an API endpoint. It can be triggered by:
- Scheduled events (CloudWatch Events/EventBridge)
- Direct Lambda invocation from the AWS SDK
- Step Functions workflow

No input parameters are required, as the function gets all configuration from environment variables:

```json
{}
```

## Response Format

The Lambda response has the following format:

```json
{
  "statusCode": 200,
  "body": {
    "message": "Dataset creation completed",
    "trainDatasetArn": "arn:aws:rekognition:region:account:dataset/train-dataset-id",
    "trainPreviousDatasetArn": "arn:aws:rekognition:region:account:dataset/previous-train-id",
    "testDatasetArn": "arn:aws:rekognition:region:account:dataset/test-dataset-id",
    "testPreviousDatasetArn": "arn:aws:rekognition:region:account:dataset/previous-test-id",
    "testSuccess": true,
    "manifestFile": "s3://bucket-name/manifest.jsonl"
  }
}
```

## Project ARN Format

The function expects project ARNs to have a specific format that includes the project ID:

```
arn:aws:rekognition:eu-west-1:587594388832:project/Datafy/1725357683732
```

The function will automatically correct ARNs that don't include the project ID.

## Manifest Requirements

The manifest file must:
1. Be named exactly `manifest.jsonl` and located at the root of the training bucket
2. Be in JSONL format with each line containing a valid JSON object
3. Have the correct content type: `application/x-amazon-s3-object-manifest-jsonl`
4. Include the required fields for each annotation
5. Be accessible from the Lambda function's execution role

## Error Handling

The function includes detailed error handling for various scenarios:
- Missing configuration variables
- Invalid project ARN format
- Missing or inaccessible manifest file
- Invalid manifest content
- AWS service errors
- Dataset creation failures

All errors are logged and returned with appropriate status codes and messages.

## Development

### Prerequisites

- Python 3.10+
- AWS CLI configured with appropriate permissions
- SAM CLI for local testing

### Local Testing

To test locally:

```bash
sam local invoke ManageDatasetsFunction --event '{}'
```

### Deployment

The function is deployed as part of the CloudFormation stack defined in `template.yaml`. The template sets the `TRAINING_BUCKET` environment variable directly, eliminating the need to retrieve it from SSM Parameter Store.

```bash
sam build
sam deploy
```

### Integration with Other Systems

Since this Lambda function is designed to be called directly (not via API Gateway), it can be integrated into various workflows:

1. Triggered after the manifest file is created:
   ```
   CreateManifest Lambda → ManageDatasets Lambda
   ```

2. Scheduled to run periodically to refresh datasets:
   ```
   CloudWatch Events (cron) → ManageDatasets Lambda
   ```

3. As part of a Step Functions workflow:
   ```
   StepFunctions → ManageDatasets Lambda → StartTraining Lambda
   ```