# Create Dataset

This Lambda function creates datasets in AWS Rekognition Custom Labels from manifest files and updates dataset information in Supabase. It's designed to be triggered when manifest files are ready for training or testing a model.

## Functionality

The function performs the following operations:

1. Retrieves project ARN from environment variables or constructs it
2. Ensures the project ARN has the correct project ID (`1725357683732`)
3. Validates the manifest files from S3
4. Creates training and/or testing datasets in AWS Rekognition
5. Updates dataset records in Supabase with ARNs and status

## Environment Variables

- `PROJECT_ARN`: ARN of the Rekognition Custom Labels project (will be fixed if format is incorrect)
- `MANIFEST_BUCKET`: S3 bucket containing the manifest files
- `SUPABASE_URL_SSM`: SSM parameter name for Supabase URL (defaults to `/supabase/url`)
- `SUPABASE_KEY_SSM`: SSM parameter name for Supabase key (defaults to `/supabase/anon`)

## Required Project ID

All Rekognition operations require the correct project ID: `1725357683732`. The function includes logic to fix ARNs with incorrect project IDs.

## AWS Resources

The function requires:
- Permission to access AWS Rekognition
- Permission to access S3 for manifest files
- Permission to access SSM Parameter Store
- The following SSM parameters:
  - `/supabase/url`: Supabase URL
  - `/supabase/anon`: Supabase anonymous key
  - `/datafy-rekognition-stack/project-name`: (Optional) Project name parameter

## Manifest File Validation

The function validates the manifest files by:
- Checking that the file exists in S3
- Verifying the file has the correct format
- Ensuring the file contains the required attributes
- Confirming image sources are accessible

## Database Integration

The function integrates with Supabase to:
- Create or update dataset records
- Store dataset ARNs for future reference
- Track dataset creation status

## Error Handling

The function includes robust error handling for various scenarios:
- Missing or invalid project ARN
- Missing or invalid manifest files
- S3 access errors
- Rekognition API errors
- Supabase connection failures

## Troubleshooting

Common issues:
- If manifest files are not found, check the S3 bucket and file paths
- If dataset creation fails, review the manifest file format
- Check logs for detailed error messages about manifest validation
- Ensure the project ID in the ARN is `1725357683732`