# BigPond Frontend

## Custom Flow Data Management
**Latest Update (2025-01-22):**  
Added explicit version field (2.1) to localStorage data structure to support backward compatibility

- Uses versioned local storage structure (v2.1)
- Data keys:
  - `customFlowData`: Combined settings and files storage
- Data persistence handled through `useLocalStorage` hook
- Migration logic maintains backward compatibility

A React-based frontend application for financial document processing and reconciliation, built with AWS Amplify.

## Overview

BigPond is a sophisticated financial document processing platform that helps users manage, analyze, and reconcile financial documents. The application leverages AWS services and OpenAI's GPT-4 for intelligent document processing and matching.

## Key Features

- **Custom Flow Processing**: Create and manage custom document processing workflows using a modular component architecture
- **Intelligent Matching**: AI-powered matching of financial documents with confidence scoring
- **Document Management**: Upload, process, and manage various financial documents
- **Data Export**: Export results in multiple formats (Excel, CSV, PDF)
- **Real-time Processing**: Live processing status and results display
- **Secure Authentication**: AWS Cognito-based user authentication

## Technical Stack

- React.js
- AWS Amplify
- OpenAI GPT-4o
- React Bootstrap
- Material-UI
- DataTables.net
- XLSX-JS-Style (for Excel exports with styling support)
- jsPDF for PDF exports

## Data Storage Structure

### customFlowData Schema

```js
{
  settings: {
    currentStep: number,
    flowName: string,
    lastModified: ISO Date,
    version: "v2.1"
  },
  files: {
    [stepNumber: number]: Array<{
      id: string,
      filename: string,
      processedData: object,
      matches: Array<{
        targetId: string,
        confidence: number,
        rulesApplied: string[]
      }>
    }>
  },
  temporaryTables: {
    [tableId: string]: {
      name: string,
      columns: string[],
      rows: Array<object>,
      createdAt: ISO Date,
      expiresAt: ISO Date
    }
  }
}
```

### custom_match_page Schema

```js
{
  steps: Array<{
    id: number,
    number: number,
    description: string,
    type: string,
    fields: Array<{
      id: number,
      number: number,
      name: string,
      type: string
    }>,
    files: Array<{
      filename: string,
      content: {
        records: Array<object>
      },
      status: string,
      timestamp: ISO Date
    }>,
    isMultiRow: boolean,
    isExcelDocument: boolean
  }>,
  match: Array<{
    stepIndex: number,
    filename: string,
    results: {
      records: Array<{
        current: object,
        previous: object,
        confidence: number,
        reasoning: string
      }>
    },
    timestamp: ISO Date
  }>
}
```

### localStorage Queue System

To prevent race conditions when multiple components access the same localStorage keys, BigPond implements a queue-based system for critical keys:

```js
// Queue-based localStorage operations
const queueEnabledKeys = [
  'custom_match_page',
  'result_table_db',
  'match_table_db',
  'match_results'
];

// Each key has its own operation queue
const updateQueueMap = new Map();
const processingMap = new Map();

// Safe operations
safeGetItem(key, defaultValue);  // Read with error handling
safeSetItem(key, value);         // Queue-based write
safeUpdateItem(key, updateFn);   // Atomic update
```

#### Key Characteristics:

- Request queuing for high-contention localStorage keys
- Sequential processing of updates to the same key
- Transformation functions for atomic read-modify-write cycles
- Proper error handling and fallback values
- Event-based cross-tab synchronization
- Debug utilities for queue monitoring

### Key Characteristics:
- Stored under `localStorageKey: "customFlowData"` and `"custom_match_page"`
- Versioned for backward compatibility
- Automatic 30-day expiration for temporary tables
- Encoded with base64 for special characters
- Validated on load with JSON Schema
- Maximum size: 5MB per flow instance
- Data persistence: 2 days for flow state
- Debounced write operations (300ms)
- Merge strategy for concurrent updates
- Metadata-driven expiration cleanup

## Changelog

### 2025-01-30
- Fixed localStorage race conditions:
  - Implemented key-based queue system for synchronized localStorage access
  - Added safeLocalStorage utility for critical localStorage operations
  - Fixed ResultsTable component to prevent data corruption during concurrent updates
  - Enhanced error handling for JSON parsing and storage operations
  - Added proper synchronization for cross-component updates

### 2025-01-28
- Fixed state management in Custom Match Page:
  - Resolved infinite update loop in StepCard component
  - Improved localStorage synchronization
  - Optimized file processing state updates
  - Enhanced parent-child component communication
  - Added proper dependency tracking in effects
  - Improved error handling and progress tracking
  - Centralized state management through parent component
  - Added memoization for performance optimization
  - Removed redundant localStorage updates
  - Enhanced data integrity and consistency

### 2025-01-27
- Added new Custom Match Page:
  - Implemented step-based document processing workflow
  - Added Field Builder for customizable data extraction
  - Integrated OpenAI for document analysis and matching
  - Added dynamic results table with DataTables.net
  - Implemented match results with confidence scoring
  - Added export functionality (Excel, CSV, PDF)
  - Added navigation for both admin and user roles
  - Implemented local storage for state persistence
  - Added file processing service with queue management
  - Added support for PDF and Excel documents

### 2025-01-22
- Added versioned local storage support (v2.1) to customFlowPage
- Implemented data migration logic for legacy storage formats
- Added validation checks for localStorage data structure
- Documented localStorage v2.1 data structure in README
    - Documented localStorage implementation details including versioning and error handling
    - Added error boundary documentation for storage operations
    - Documented initialization sequence requirements
    - Added concurrency handling documentation for cross-tab updates

### 2025-01-22
- Enhanced custom flow documentation:
  - Added component structure documentation
  - Detailed localStorage integration patterns
  - Documented temporary table lifecycle management
  - Added data storage schema documentation to README
  - Implemented storage management documentation:
    * Data persistence policies
    * Write optimization strategies
    * Concurrent update handling
    * Automated cleanup mechanisms
  - Updated localStorage implementation:
    - Unified data structure for settings/files
    - Simplified storage operations
    - Added error boundaries for data integrity
    - Fixed infinite loop in useCustomFlow hook
    - Optimized localStorage operations with change detection
    - Added proper step initialization sequence
    - Fixed array validation in steps loading
    - Added type checking for stored steps
    - Improved localStorage initialization
    - Moved data loading to useState initializer
    - Removed redundant data loading effects
  - Code quality improvements:
    - Fixed indentation and formatting in useCustomFlow hook
    - Added missing closing braces in initialization logic
    - Standardized 4-space indentation throughout custom flow components
    - Fixed callback function syntax consistency
    - Fixed indentation consistency in useLocalStorage hook
    - Added array validation for processed files storage
    - Aligned return statement formatting with project style guidelines
    - Added AI-powered document matching integration with processUnprocessedFiles Lambda

### 2024-01-10
- Added scheduled Lambda function for automated file processing:
  - Processes unprocessed PDF/Excel files with OpenAI every 5 minutes
  - Automatically analyzes documents and saves results
  - Integrates with existing file processing tables
  - Runs asynchronously for improved performance

### 2024-01-09
- Enhanced matching functionality:
  - Added dynamic data tables with search and pagination
  - Implemented export functionality (Excel, CSV, PDF)
  - Added confidence scoring system (0-10) for matches
  - Integrated tooltips with match details and explanations
  - Added local storage persistence for matching results
  - Added cursor-following tooltips for detailed information

## Getting Started

1. Clone the repository
2. Install dependencies: `npm install`
3. Configure AWS Amplify
4. Set up environment variables
5. Run the development server: `npm start`

## Environment Variables

Required environment variables:
- `REACT_APP_OPENAI_API_KEY`: OpenAI API key for matching functionality
- AWS Amplify configuration variables

## Contributing

Please refer to CONTRIBUTING.md for guidelines on contributing to this project.

## License

This project is proprietary and confidential. All rights reserved.