# OpenAI Analysis Process Documentation

## Overview

The OpenAI Analysis system in AuditWhizz AutoClicker is designed to automatically detect and interact with supporting document links found in invoice screenshots. This system uses OpenAI's GPT-4 Vision model to analyze screenshots and identify clickable elements that could be supporting documents, receipts, or downloadable content.

## 🎯 Purpose

The primary goal is to automate the process of:
1. **Detecting supporting document links** in invoice screenshots
2. **Automatically clicking** on identified links
3. **Capturing post-click screenshots** for documentation
4. **Uploading evidence** to S3 for audit trails
5. **Navigating back** to continue processing multiple links

## 🔧 Technical Architecture

### Core Components

#### 1. **OpenAIAnalyzer Class** (`autoclicker/openai_analyzer.py`)
- **Main analyzer** that coordinates the entire process
- **OpenAI API integration** using GPT-4 Vision model
- **Automated clicking** using PyAutoGUI
- **S3 integration** for file uploads
- **Comprehensive logging** to `logs/openai.log`

#### 2. **ScreenshotManager** (`autoclicker/screenshot.py`)
- **Screenshot capture** functionality
- **Integration bridge** between UI and analyzer
- **Error handling** and user feedback

#### 3. **Constants Configuration** (`autoclicker/constants.py`)
- **API key management** from environment variables
- **Configurable wait times** (default: 10 seconds)
- **S3 bucket configuration** for production use

## 🚀 Workflow Process

### Step 1: User Initiation
```
User clicks "Analyze Docs" button → App triggers analysis → ScreenshotManager captures current screen
```

### Step 2: Screenshot Capture
```python
def capture_screenshot(self):
    """Capture a screenshot of the current screen."""
    # Create screenshots directory
    # Generate timestamped filename
    # Use PyAutoGUI to capture screen
    # Save to local storage
    # Return file path for analysis
```

### Step 3: OpenAI Analysis
```python
def _analyze_screenshot_for_links(self, screenshot_path: str):
    """Analyze screenshot using OpenAI GPT-4 Vision."""
    # Encode image to base64
    # Prepare prompt for supporting document detection
    # Send to OpenAI API with gpt-4o model
    # Parse JSON response for coordinates
    # Validate coordinate data
    # Return list of clickable elements
```

### Step 4: Automated Clicking
```python
def perform_automated_clicking(self, clickable_elements, screenshot_path):
    """Perform automated clicking on all identified links."""
    for element in clickable_elements:
        # Extract coordinates (x, y)
        # Click on element using PyAutoGUI
        # Wait configured time (default: 10s)
        # Capture post-click screenshot
        # Upload to S3 (if configured)
        # Navigate back (backspace)
        # Continue to next element
```

### Step 5: Documentation & Storage
```python
def _capture_post_click_screenshot(self, index, description):
    """Capture screenshot after clicking an element."""
    # Generate timestamped filename
    # Save to screenshots/post_click_debug/
    # Return file path for S3 upload

def _upload_to_s3(self, file_path, key_prefix):
    """Upload file to S3 bucket."""
    # Generate S3 key with timestamp
    # Upload to configured bucket
    # Return accessible S3 URL
```

## 📋 OpenAI API Integration

### Model Configuration
- **Primary Model**: `gpt-4o` (GPT-4 Vision)
- **Fallback Models**: `gpt-4o-mini` (if needed)
- **API Endpoint**: `https://api.openai.com/v1/chat/completions`

### Prompt Engineering
The system uses a carefully crafted prompt to ensure consistent results:

```json
{
  "role": "user",
  "content": [
    {
      "type": "text",
      "text": "Analyze this screenshot and identify ONLY supporting document links that might be found on invoices. Look for: Download links for PDFs, receipts, or supporting documents, 'View' or 'Download' buttons for attachments, Links to related documents or evidence, File attachment links, Receipt or proof of payment links. Return ONLY a JSON array with this exact format: [{\"type\": \"supporting_document_link\", \"description\": \"brief description of what the link provides\", \"coordinates\": [x, y], \"confidence\": 0.95}]"
    },
    {
      "type": "image_url",
      "image_url": {"url": "data:image/png;base64,{base64_encoded_image}"}
    }
  ]
}
```

### Response Format
Expected JSON response structure:
```json
[
  {
    "type": "supporting_document_link",
    "description": "Download PDF receipt",
    "coordinates": [450, 320],
    "confidence": 0.95
  },
  {
    "type": "supporting_document_link", 
    "description": "View supporting documents",
    "coordinates": [780, 450],
    "confidence": 0.92
  }
]
```

## ⚙️ Configuration

### Environment Variables
```bash
# Required
OPENAI_API_KEY=your_openai_api_key_here

# Optional (for S3)
AWS_ACCESS_KEY_ID=your_aws_key
AWS_SECRET_ACCESS_KEY=your_aws_secret
AWS_REGION=ap-southeast-2
```

### Constants Configuration
```python
# Wait time after each click (configurable)
POST_CLICK_WAIT_TIME = 10  # seconds

# S3 bucket configuration
OPENAI_ANALYZER_S3_BUCKET = "auditwhizz-supporting-documents"
OPENAI_ANALYZER_S3_PREFIX = "supporting_documents/"
```

## 🔍 Error Handling & Logging

### Logging System
- **Log File**: `logs/openai.log`
- **Log Levels**: INFO, WARNING, ERROR
- **Structured Format**: `[Timestamp] [Level] [Module] Message`

### Common Error Scenarios
1. **API Key Issues**: Missing or invalid OpenAI API key
2. **Model Availability**: Model not accessible in region/account
3. **Coordinate Validation**: Invalid coordinates returned by OpenAI
4. **Click Failures**: PyAutoGUI unable to click at specified coordinates
5. **S3 Upload Issues**: Network or permission problems

### Error Recovery
- **Individual Element Failures**: Continue processing other elements
- **API Failures**: Retry with fallback models
- **Click Failures**: Log error and continue to next element
- **S3 Failures**: Store locally if S3 unavailable

## 📊 Performance & Optimization

### Timing Considerations
- **Screenshot Capture**: ~100-500ms (depending on screen size)
- **OpenAI Analysis**: ~2-10 seconds (depending on image complexity)
- **Post-Click Wait**: Configurable (default: 10 seconds)
- **Navigation Back**: ~2 seconds (fixed)
- **Total per Element**: ~15-20 seconds

### Optimization Strategies
- **Batch Processing**: Process multiple elements sequentially
- **Parallel S3 Uploads**: Upload screenshots while processing next element
- **Coordinate Caching**: Store successful coordinates for future reference
- **Error Rate Monitoring**: Track success/failure rates for optimization

## 🧪 Testing & Validation

### Testing Scenarios
1. **Valid Screenshots**: Test with various invoice layouts
2. **Edge Cases**: Empty screenshots, very complex layouts
3. **Error Conditions**: Invalid API keys, network failures
4. **Performance**: Large numbers of clickable elements

### Validation Checks
- **Coordinate Bounds**: Ensure coordinates are within screen dimensions
- **JSON Parsing**: Validate OpenAI response format
- **Click Success**: Verify PyAutoGUI clicks are successful
- **File Integrity**: Check screenshot quality and S3 uploads

## 🔒 Security Considerations

### API Key Management
- **Environment Variables**: Store keys securely, not in code
- **Access Control**: Limit API key permissions to necessary scopes
- **Rotation**: Regular key rotation for production use
- **Monitoring**: Track API usage and costs

### Data Privacy
- **Screenshot Storage**: Secure local and S3 storage
- **Log Sanitization**: Remove sensitive data from logs
- **Access Logs**: Monitor who accesses analysis results
- **Compliance**: Ensure GDPR/CCPA compliance for data handling

## 🚀 Future Enhancements

### Planned Features
1. **Multi-Model Support**: Automatic fallback between different AI models
2. **Learning System**: Improve accuracy based on user feedback
3. **Batch Processing**: Analyze multiple screenshots simultaneously
4. **Advanced Navigation**: Support for complex multi-step workflows
5. **Integration APIs**: Connect with external document management systems

### Performance Improvements
1. **Async Processing**: Non-blocking analysis and clicking
2. **Smart Caching**: Cache common patterns and results
3. **Predictive Analysis**: Pre-analyze common invoice layouts
4. **Distributed Processing**: Scale across multiple machines

## 📚 Troubleshooting Guide

### Common Issues & Solutions

#### Issue: 401 Unauthorized Error
**Cause**: Invalid or missing OpenAI API key
**Solution**: 
- Verify API key in `.env` file
- Check API key format (should start with `sk-` or `sk-proj-`)
- Ensure sufficient credits in OpenAI account

#### Issue: 404 Model Not Found Error
**Cause**: Model name incorrect or not available
**Solution**:
- Use `gpt-4o` (confirmed working model)
- Check model availability in your region
- Verify API key has access to requested model

#### Issue: Click Coordinates Out of Bounds
**Cause**: OpenAI returned invalid coordinates
**Solution**:
- Check coordinate validation in logs
- Verify screen resolution settings
- Review OpenAI response format

#### Issue: S3 Upload Failures
**Cause**: AWS credentials or permissions issues
**Solution**:
- Verify AWS credentials
- Check S3 bucket permissions
- Ensure bucket exists and is accessible

## 📞 Support & Maintenance

### Monitoring
- **Log Analysis**: Regular review of `logs/openai.log`
- **Performance Metrics**: Track processing times and success rates
- **Error Rates**: Monitor failure patterns and frequencies
- **Cost Tracking**: Monitor OpenAI API usage and costs

### Maintenance Tasks
- **Log Rotation**: Archive old log files
- **Model Updates**: Stay current with OpenAI model changes
- **Security Updates**: Regular review of access controls
- **Performance Tuning**: Optimize based on usage patterns

---

*This document is maintained as part of the AuditWhizz AutoClicker project. For updates or questions, refer to the project repository or contact the development team.*