# Direct Integration Module

## Overview
The Direct Integration module is a background service that automates the processing of Excel file uploads for the Auto Clicker application. It provides a seamless way to integrate external data by monitoring unprocessed uploads in MySQL and processing files stored in AWS S3.

## Key Features
- Background polling for unprocessed uploads
- S3 file download and processing
- Excel data extraction
- Support for both new and existing playlists
- Local development mode support
- Automatic UI updates based on upload configurations

## How It Works

### 1. Upload Processing Flow
1. The watcher polls MySQL every 30 seconds (configurable) for unprocessed direct uploads
2. If no uploads are found, it keeps polling
3. When an upload is found:
   - It is immediately marked as "user interaction" (status = 4)
   - The UI is prepared based on `playlist_mode`
   - For `playlist_mode = "new"`: S3 is skipped entirely — no file is expected
   - For `playlist_mode = "existing"`: if `s3_bucket` and `s3_key` are present and not in local mode, the file is downloaded and parsed
4. When the user clicks the "I'm done" button in the UI, the row is marked as "processing" (status = 2)
5. Playback runs. On successful completion:
   - Any local direct integration uploads for the current playlist are marked `processed = 1`
   - The playlist itself is marked `processed = 1`
   - The Direct Integration watcher restarts polling to fetch the next available record
6. On error, the row is marked as `processed = 3` and the watcher waits 30 seconds before continuing

### 2. Playlist Modes
- **New Playlist Mode** (`playlist_mode="new"`)
  - Creates a new playlist with the specified name
  - Auto-fills the recording name and hides the input field
  - Skips S3 entirely (no file is expected for new playlists)
  - More lenient behavior: S3-related errors are ignored for this mode
  
- **Existing Playlist Mode** (`playlist_mode="existing"`)
  - Links to an existing playlist by ID and loads its actions
  - If S3 details are present (and not in local mode), the file is downloaded and parsed
  - Stricter error handling

### 3. Environment Modes
- **Production Mode**
  - Full S3 integration
  - File downloads and processing
  - AWS credentials required
  
- **Local Development Mode** (`ENV=local`)
  - S3 operations disabled entirely
  - UI-only setup
  - No file downloads required

## Configuration

### Environment Variables
- `ENV`: Set to `local` for local development mode (disables S3)
- `AWS_REGION`: AWS region for S3 operations (default: "ap-southeast-2")

### AWS Requirements
- S3 bucket access
- Proper IAM permissions for:
  - `s3:GetObject`
  - `s3:ListBucket`

## Error Handling

### Retry Logic
1. **New Playlists**
   - S3 is skipped — no download attempts
   - S3-related errors are ignored
   - UI remains available for manual recording

2. **Existing Playlists**
   - Stricter error handling
   - Retries only for transient errors:
     - Network timeouts
     - Connection issues
     - Temporary AWS failures

### Post-Playback Behavior
- On successful playback of a playlist:
  - Direct Integration uploads associated to the current playlist (local entries) are marked `processed = 1`
  - The playlist record is marked `processed = 1`
  - The Direct Integration watcher restarts and resumes polling for the next unprocessed record

### Status Codes (state)
- 0: Unprocessed
- 1: complete
- 2: Processing
- 3: Error
- 4: User interaction
- 5: canceled

## Integration with UI
- Automatically updates application selection
- Configures playlist dropdown
- Updates recording name
- Displays data preview in live clicks pane
- Maintains user/company context

## Best Practices
1. Always check upload status before processing
2. Handle S3 credentials properly
3. Clean up temporary files after processing
4. Log all significant operations
5. Preserve user and company context
6. Implement proper error recovery

## Example Usage

```python
# Initialize the watcher
watcher = DirectIntegrationWatcher(app, poll_interval=30)  # 30 second polling

# Start the background thread
watcher.start()

# Stop the watcher (if needed)
watcher.stop()
```

## Troubleshooting

### Common Issues
1. **S3 Access Denied**
   - Check AWS credentials
   - Verify IAM permissions
   - Ensure bucket policy allows access

2. **Processing Failures**
   - Check Excel file format
   - Verify file exists in S3
   - Confirm database connectivity

3. **UI Not Updating**
   - Verify application_id exists
   - Check playlist_id for existing playlists
   - Ensure UI components are properly initialized

### Logging
The module uses Python's logging system to track operations and errors. Monitor logs for:
- S3 download status
- Processing steps
- Error messages
- Retry attempts