# Changelog

## [Unreleased]

### Added
- **Bulk User Import Feature**: Added ability to import multiple users via CSV file upload
  - CSV format: `company_name,fullname,email`
  - Automatic defaults: `role_id=3` (User), `active=1` (Active)
  - Company name lookup automatically converts to company ID
  - Real-time validation and preview before import
  - Progress tracking and detailed error reporting
  - CSV template download with sample data
  - Integrated into existing user management page with "BULK IMPORT" button

### Technical Changes
- Added `BulkImportUserModal` component (`src/components/tableBasic/users/bulkImportUser.jsx`)
- Enhanced `UserTable` component with bulk import functionality
- Added `/bulkaddusers` API endpoint for batch user processing
- Updated user management page to include bulk import modal
- Added company name validation against existing companies in database

### Added
- ContactForm component for user inquiries
  - Created new modal-based contact form with Name, Email, and Message fields
  - Integrated with existing AWS Amplify API to send contact messages via `/contactmail` endpoint
  - Added form validation with user-friendly error messages
  - Implemented loading states and success notifications
  - Updated Welcome component to trigger contact form modal from "CONTACT US" button
  - Follows existing project patterns for styling and API integration
- Comprehensive OpenAI usage logging system
  - Created new `workpaper_logs` MySQL table to track all OpenAI API usage
  - Added logging for both document processing and matching operations
  - Captures user name, audit firm name, operation type, document details, token usage, and processing performance
  - Includes input tokens, output tokens, total tokens, model used, and processing time
  - Automatically logs both successful operations and errors
  - Enhanced OpenAI service to capture token usage from API responses
  - Added batch processing support for token aggregation across multiple API calls

### Changed
- Refactored data type detection utilities for better code organization
  - Moved `isDateValue`, `isDecimalValue`, `isNumericValue`, and `detectFieldTypeFromData` functions from StepCard.jsx to utils/formatters.js
  - Updated import statements to use centralized utility functions
  - Removed useCallback wrappers for utility functions to improve performance
  - Updated function dependency arrays to remove imported functions
  - Improved code reusability and maintainability by consolidating data type detection logic
  - Enhanced modularity by keeping utility functions separate from component logic

### Fixed
- Fixed React DOM validation error in ResultsTable component
  - Resolved "div cannot appear as child of tbody" error by moving tooltip rendering outside table structure
  - Modified RowTooltip component to only return tr elements inside tbody
  - Added tooltip state management to main ResultsTable component
  - Tooltip now renders outside table using fixed positioning to avoid DOM nesting violations
- Enhanced Excel field type detection with intelligent field name analysis
  - Modified parseExcelHeaders function to combine field name patterns with data analysis
  - Added detectFieldTypeFromData function that considers both field names and data patterns
  - Implemented smart field type detection for dates, currency, and identifiers based on field names
  - Added date field detection for fields named "date", "time", "created", "updated", etc.
  - Added currency field detection based on field names (debit, credit, balance, amount, total, etc.)
  - Added identifier field detection for text classification (invoice number, reference, account code, etc.)
  - Prioritized field name patterns over data patterns for well-known field types
  - Enhanced date pattern recognition to support text-based formats like "30 Nov 2022", "Nov 30, 2022"
  - Date fields (date, time, created, updated, etc.) are always classified as DATE regardless of data patterns
  - Currency fields (debit, credit, balance, etc.) are always classified as CURRENCY regardless of data patterns
  - Identifier fields (numbers, codes, references) are always classified as TEXT regardless of numeric content
  - Improved date detection to prevent numeric values from being misclassified as dates
  - Uses three-tier priority system: field name patterns → strong data patterns → fallback analysis
  - Samples up to 10 non-empty values per column for accurate type detection
- Fixed hyperlink styling in Excel exports
  - Migrated from `xlsx` to `xlsx-js-style` library for proper cell styling support
  - Implemented proper blue color and underline styling for hyperlinks in Excel files
  - Added trimming of filenames and folder paths to prevent broken links
  - Removed post-processing styling code in favor of direct cell styling during creation
- Fixed race conditions in localStorage access operations
  - Implemented a key-based queue system for synchronized access to localStorage
  - Added multiple key support for safe reading and writing to localStorage
  - Added support for four critical localStorage keys: 'custom_match_page', 'result_table_db', 'match_table_db', 'match_results'
  - Created safe utility functions for localStorage operations to prevent race conditions
- Fixed ResultsTable data display issues
  - Corrected double parsing bug in loadDataFromLocalStorage function
  - Fixed data handling when using safeGetItem utility
  - Added debug logging to track data loading process
- Fixed match_table_db data not showing correctly in results table
  - Enhanced record matching algorithm in ResultsDataTable.jsx
  - Added multi-strategy matching to find the correct corresponding records
  - Implemented value-only matching for fields when names don't match exactly
  - Added partial text matching as a fallback method
- Fixed incorrect row mapping for bank account transactions
  - Added specialized account number extraction and normalization
  - Implemented bank account number pattern recognition
  - Enhanced matching priority system for financial record identification
  - Added field synchronization to ensure consistent column display
- Fixed dynamic field handling in matching algorithm
  - Removed all hardcoded field assumptions and lists
  - Implemented pure data-driven field detection and matching
  - Added dynamic field uniqueness analysis for better matching
  - Prioritized matching based on statistical uniqueness of values
- Fixed complete field synchronization between matching records
  - Implemented bidirectional field copying for matched records
  - Added field name similarity detection for differently named fields
  - Applied exact value matching to identify related fields
  - Ensured complete field mirroring between matched steps
- **buildLargeDocumentPrompt Function**: Fixed large document processing prompt to properly extract all specified fields from images and match them with previous step records. The function now:
  - Uses the `fields` parameter to specify what fields to extract (was previously ignored)
  - Includes document type and options context
  - Uses dynamic field matching instead of hardcoded field names - compares against available fields in previous step records
  - Restructured to match the buildMatchingPrompt approach with clear two-phase process: EXTRACT → COMPARE → MATCH
  - Implements extremely strict matching criteria requiring ALL THREE criteria to be met: exact amount match, exact/close date match, and clear company name match
  - Prevents false matches by requiring triple verification (amount + date + company name)
  - Explicitly excludes random product names like "Samsung Galaxy", "Motorola V50", "Siemens S35J"
  - Adds "when in doubt, exclude" logic to prioritize quality over quantity
  - Adds professional auditor context for handling prepayments, partial payments, and installments
  - Maintains consistent formatting requirements across all extracted records

### Added
- Added **ZipWorkPapers** Lambda function for creating workpapers archive:
  - Creates zip files containing all user workpapers from S3 bucket
  - Maintains original folder structure (Step{number}-{description}/)
  - Automatically includes Results.xlsx file in the ZIP archive
  - Accessible via `/workpapers/{userId}` API endpoint
  - Supports pagination for large numbers of files
  - Includes comprehensive error handling and CORS support
  - Automatically cleans up previous zip files before creating new ones
  - Returns signed download URL for immediate file download
  - Added "Download ZIP" button in ResultsTable for easy access
- Added **Excel Export to S3** functionality:
  - Automatically generates Excel report when creating ZIP download
  - Uses same formatting and styling as manual Excel export
  - Saves Results.xlsx file to S3 alongside workpapers
  - Included in ZIP archive for comprehensive reporting
  - Added hyperlink columns linking to source files in Step folders
  - File links automatically point to Step{number}-{description}/{filename}
  - Enhanced filename detection with fallback to generic filenames
  - Ensures every row has a hyperlink (folder link if no file found)
- Added new utility file `safeLocalStorage.js` with functions:
  - `safeGetItem`: Safely read from localStorage with proper error handling
  - `safeSetItem`: Queue-based writing to localStorage to prevent race conditions
  - `safeUpdateItem`: Apply a transform function to update localStorage data
  - `safeRemoveItem`: Safely remove items from localStorage
  - `getPendingOperationsStats`: Debug function to monitor queue status
- Added `safeLocalStorage.js` to customFlowPage
  - Implemented the same robust storage management for customFlowPage
  - Ensured consistent localStorage handling across different pages
  - Fixed race conditions in customFlowPage localStorage operations
- Added account number processing utilities:
  - `normalizeAccountNumber`: Consistent formatting for account numbers
  - `extractAccountNumber`: Extract account numbers from various text formats
  - Detects common account number patterns (e.g., XXX-XXX-XXXX, XXXXXX-XX-XXXX)
  - Handles different separators and formats automatically
- Added dynamic field analysis system:
  - Statistical analysis of field value uniqueness
  - Automatic detection of potential key fields
  - Dynamic priority scoring based on value distribution
  - Cross-record value combination matching
- Added intelligent field ordering system:
  - Grouped related fields together for logical display
  - Applied semantic field type detection
  - Ordered columns consistently across steps
  - Used field name patterns to determine display order
- Added batch processing for large PDF documents in openAIFileProcessing function:
  - Implemented automatic batch processing when `isMultiRow = true` and `pdfPages > 3`
  - Processes documents in batches of 3 pages to handle large multi-page PDFs efficiently
  - Combines results from all batches into a single unified response
  - Added comprehensive progress logging for batch processing operations
  - Enhanced buildDocumentPrompt to include batch-specific context for AI processing
  - Added batch information tracking (total batches, pages processed, errors per batch)
  - Implemented parallel processing of batches for improved performance
  - Added error handling for individual batch failures while continuing with other batches

### Changed
- **Refactored ZipWorkPapers Lambda function** to reduce cognitive complexity:
  - Extracted helper functions to improve code maintainability
  - Created `createCorsResponse()` to handle CORS response creation
  - Created `listAllS3Objects()` to handle S3 object listing with pagination
  - Created `filterWorkpaperObjects()` to handle workpaper filtering logic
  - Created `createArchiveWithFiles()` to handle archive creation and file processing
  - Created `generateSignedDownloadUrl()` to handle signed URL generation
  - Created `clearExistingZipFile()` to handle existing zip file cleanup
  - Reduced cognitive complexity from 18 to under 15 allowed threshold
  - Maintained all original functionality and logic unchanged
- **Refactored getFilenameForStep function** in exportExcelToS3.js to reduce cognitive complexity:
  - Extracted helper functions to improve code maintainability and readability
  - Created `getStepResults()` to handle result table data retrieval
  - Created `getSingleFileFilename()` to handle single file case logic
  - Created `findMatchingRecord()` to handle record matching in file content
  - Created `getMultipleFilesFilename()` to handle multiple files with matching logic
  - Created `getFilenameFromKnownFields()` to handle filename extraction from known field names
  - Created `looksLikeFilename()` to validate filename-like values
  - Created `getFilenameFromFilenameFields()` to handle filename extraction from filename-like fields
  - Created `getFilenameFromCustomMatchData()` to handle final fallback to custom match data
  - Reduced cognitive complexity from 28 to under 15 allowed threshold
  - Maintained all original functionality and logic unchanged
- Updated `ResultsTable.jsx` to use safe localStorage access operations:
  - Fixed loadDataFromLocalStorage to handle race conditions
  - Improved error handling throughout localStorage operations
  - Added proper cleanup when components unmount
  - Enhanced cross-tab synchronization for localStorage changes
- Enhanced ResultsTable matching algorithm:
  - Implemented dynamic field detection and indexing
  - Improved match detection with value-based field matching
  - Added pure data-driven field matching without assumptions
  - Enhanced logging for match detection troubleshooting 
- Improved match_table_db data processing:
  - Added 4 matching strategies with progressive fallbacks
  - Fixed JSON parsing error when data was already parsed by safeGetItem
  - Enhanced debugging by logging the matching method used for each record
  - Optimized performance by stopping search after finding the first match
- Enhanced field synchronization between steps:
  - Added bidirectional field mapping between related steps
  - Ensured matched rows display all fields from both steps
  - Implemented consistent column alignment for matched data
  - Added detailed logging of field mappings between steps
- Moved to completely data-driven field handling:
  - Replaced hardcoded field lists with dynamic value frequency analysis
  - Implemented adaptable matching that works with any field structure
  - Added statistical scoring of field uniqueness for smarter matching
  - Prioritized matching based on actual data patterns in the dataset
- Improved table display organization:
  - Implemented semantic column ordering for related fields
  - Grouped fields by type (account, payee, amount, date, etc.)
  - Applied consistent column ordering across all steps
  - Enhanced visual alignment of related fields between steps