# Create Manifest Function Changelog

## [2024-04-09] - Simplified Manifest Handling for Safe Updates

### Changed
- Simplified the approach for managing manifest entries:
  - Read the existing manifest file (if it exists)
  - Add new entry or replace existing entry with the same source-ref
  - Write back to S3 without deleting the file first
  - This prevents race conditions and data loss during updates

### Fixed
- Solved potential data loss issues when processing files
- Fixed URL formatting inconsistencies in the `source-ref` field
- Ensured consistent use of `s3://` (double slash) in all URLs
- Removed the practice of deleting the manifest before recreating it

### Technical Notes
- The function checks if an entry with the same source-ref already exists
- If a duplicate is found, it's replaced with the new entry
- The manifest is written directly back to S3 without deletion
- This preserves entries even if errors occur during processing

## [2024-04-08] - Improved Manifest File Handling for Single Entries

### Changed
- Modified the approach for adding entries to the manifest file:
  - Now preserving the existing manifest file instead of rebuilding it each time
  - Added validation for new entries before adding them to the manifest
  - Improved detection and handling of duplicate entries
  - Added proper JSONL formatting with correct line endings

### Fixed
- Resolved issue where valid entries might be lost during manifest regeneration
- Fixed content type verification and correction after upload
- Improved error handling to prevent invalid entries from being added

### Technical Notes
- The function now processes single XML files incrementally without rebuilding the entire manifest
- Each entry is validated individually before being added
- Duplicates are detected and replaced instead of filtered out
- This preserves all valid entries in the manifest, even if some entries have errors

## [2024-04-08] - Updated Validation in rekognition_utils.py for Pixel Values

### Fixed
- Updated validation code in `rekognition_utils.py` to expect pixel values rather than normalized values
- Modified validation error messages to clearly indicate valid pixel ranges based on image dimensions
- Aligned validation logic with the changes made in `manifest_utils.py`
- Fixed rebuilding logic to properly retain valid entries

### Technical Notes
- Previous validation in `rekognition_utils.py` was still checking for normalized values (0-1)
- Updated to check pixel coordinates against actual image dimensions from the manifest
- This ensures consistent validation throughout the create_manifest function

## [2024-04-08] - Fixed Bounding Box Coordinate Format

### Issue
- The Lambda function was using normalized values (0-1) for bounding box coordinates 
- AWS Custom Labels requires pixel values for coordinates as per documentation
- This mismatch caused "Manifest is not valid" errors in the API
- The same manifest could be loaded in the AWS Console but would show labeling errors

### Solution
- Modified `convert_xml_to_rekognition_format` to use pixel values for coordinates (left, top, width, height)
- Updated validation logic to check pixel values against image dimensions
- Ensured coordinates and dimensions are integers as required by AWS
- The manifest now follows the format shown in AWS documentation

### Technical Notes
- Bounding box coordinates should be in pixels not normalized values
- According to AWS docs:
  - `top` – Distance from the top of the image to the top of the bounding box, in pixels
  - `left` – Distance from the left of the image to the left of the bounding box, in pixels
  - `width` – Width of the bounding box, in pixels
  - `height` – Height of the bounding box, in pixels
- This change ensures compatibility with the AWS CreateProjectVersion API

### Troubleshooting
- If manifest validation fails, check that coordinates are in pixels not normalized values
- Verify that bounding boxes are completely within image boundaries
- Ensure that the manifest format matches the example in AWS documentation

## [2024-04-08] - URL Encoded File Paths Fix

### Fixed
- Added URL decoding for S3 file paths that contain URL-encoded characters (`+`, `%20`, etc.)
- Fixed processing of XML files with special characters in their paths
- Ensured proper handling of encoded paths in S3 event notifications

### Technical Notes
- S3 event notifications often contain URL-encoded paths (e.g., spaces as `+` or `%20`)
- Added `urllib.parse.unquote_plus()` to decode these paths before processing
- This fix ensures files with spaces or special characters in their names can be properly accessed

## [2024-04-07] - Fixed Path Handling and File Organization

### Changed
- Modified path handling in manifest entries:
  - Now using flattened paths with only parent folder and filename
  - Example: `s3://bucket/parent_folder/image.jpg` instead of nested paths
  - Improved file existence validation

### Fixed
- Resolved issue with nested paths in source-ref URLs
- Improved S3 file validation to handle path changes

### Technical Notes
- Manifest entries now use simplified paths:
  - Format: `s3://bucket/parent_folder/filename.jpg`
  - Parent folder is preserved for organization
- File validation checks both flattened and original paths
- Detailed logging added for path transformations

## [2024-04-07] - S3 Path Validation Fix

### Changed
- Removed automatic path modification that was incorrectly "fixing" nested paths
- Added validation to verify files exist in S3 at their original paths
- Preserved original folder structure from source data

### Fixed
- Stopped modifying source-ref paths in manifest
- Added existence checks for S3 files
- Improved error logging for missing files

## [2024-04-07] - Manifest Format and Content Type Updates

### Fixed
- Updated manifest content type to match AWS documentation exactly: `application/x-amazon-s3-object-manifest-jsonl`
- Improved JSONL format handling:
  - Each line is a complete JSON object with no pretty printing
  - Added newline after each entry
  - No trailing whitespace
- Added verification steps after manifest creation/update
- Fixed excessive URL encoding in S3 URIs
- Ensured manifest file has the exact name AWS Rekognition requires (`manifest.jsonl`)

## [2024-04-03] - S3 URI and Manifest Fixes

### Fixed
- Changed manifest filename from `manifest.json` to `manifest.jsonl` to match AWS Rekognition requirements
- Changed S3 manifest content type from `application/x-jsonlines` to `application/x-amazon-s3-object-manifest-jsonl`
- Fixed S3 URI handling:
  - Properly URL encode spaces and special characters in paths
  - Maintain correct s3:// prefix
  - Handle each path component separately
- Normalized bounding box coordinates to be between 0 and 1
- Fixed field ordering in manifest entries

### Added
- Validation check for normalized coordinates (0 to 1)
- More detailed error logging for manifest validation failures
- Better source-ref validation and error messages
- Logging of final source-ref URLs for debugging

### Technical Notes
- Manifest File Requirements:
  - Must be named `manifest.jsonl` (not .json)
  - Must use content type `application/x-amazon-s3-object-manifest-jsonl`
  - Each line must be a complete, valid JSON object
  - No empty lines allowed between entries
- S3 URI Format Requirements:
  - Must start with exactly "s3://"
  - Spaces and special characters must be URL encoded
  - Forward slashes must be preserved
- Coordinates must be normalized
- Field order matters for AWS Rekognition compatibility

### Troubleshooting
If training fails with "manifest is not valid":
1. Verify manifest filename is exactly `manifest.jsonl`
2. Check source-ref URLs in manifest
3. Verify URL encoding of paths
4. Ensure correct s3:// prefix
5. Delete existing manifest and reprocess if needed