# Technical documentation — dms_archive (WCA Archive)

Evidence date: 2026-10-17.

Source: `git@bitbucket.org:overdrive/dms_archive.git`, `main` at `2bcf278` (2026-10-16, "update back to 50").

This describes the tree that was inspected. It is not an architecture decision.

## Stack

| Item | Evidence |
|---|---|
| Language | PHP. `composer.json` requires `^8.4`. `docs/README.md` says 8.3. Pipeline image is `php:8.3-cli`. `docs/overview.md` says `^8.0`. |
| Framework | CodeIgniter 4, `codeigniter4/framework` `^4.0`. Front controller `public/index.php`. CLI `spark`. |
| Database | MySQL or MariaDB via MySQLi. Template in `env`: host `localhost`, database `extract`, user `root`. Prefix support is used (`$db->DBPrefix`, `prefixTable` in project rules). |
| Web root | `public/` |
| Tests | PHPUnit 9 (`phpunit.xml.dist`). `composer test` runs phpunit. |
| CI | Bitbucket Pipelines, SonarQube pipe `sonarsource/sonarqube-scan:3.0.2`, project key `overdrive_dms_archive_26db4c01-33e0-4d19-9740-57bf7be41072`. |

`docs/README.md` also lists PHP extensions intl, curl, json, mbstring, mysqlnd, xml, and for OCR the binaries `pdftotext`, `pdftoppm` or ImageMagick, and `tesseract`.

## Application structure

```text
app/Commands          CLI: mailbox poll, campaign send, system monitor, deploy notify
app/Config            Routes, filters, BulkEmail worker settings, CI config
app/Controllers       HTTP controllers, plus Admin\Documents and Developers\*
app/Database          Migrations from 2022-11 through 2026-09, seeds
app/Filters           Authentication, Authorization, TimeAccess, throttles
app/Helpers           Permissions, vault, documents, RAG, mail, chat memory
app/Libraries         OCR, PDF text, mailbox, bulk email, vector store, JWT
app/Models            CI models for each table group
app/Neuron            DocumentRAG.php (Neuron AI agent)
app/Views             documents, bulkEmail, mailbox, users, system settings, legacy EDI views
public/               Web root and bulk-email JS/CSS
docs/                 In-repo markdown (not Silicon Ark)
scope/                Task notes, SAWIS PDF, HTML scope site
tests/                unit, feature, database, session
writable/             Logs, cache, uploads (runtime)
```

`setAutoRoute(true)` is set in `Routes.php`. Explicit routes and conventional `controller/method` URLs both work.

Code style required by `docs/PROJECT_RULES.md`: tabs, PHPDoc on classes and public methods, controllers extend `BaseController`. Sonar rule `php:S105` (tabs) is suppressed for all PHP, which matches the tab rule and hides it from the scan.

## Request pipeline

`app/Config/Filters.php`:

1. `isLoggedIn` (`Authentication`) on every request except login, registration, password reset, 2FA, tracking pixels, `cronJobs`, `deploy` and `deploy/*`, `slack`, `email/test`, organisations, store-api, and a few error paths.
2. `isGranted` (`Authorization`) on the same set plus it also allows `home`, `Welcome/*`, `chatGPT/*`, and `rag/*` without a menu check.
3. `timeAccess` on `documents*` (global filter) and on the `mailbox` route group.
4. CSRF on every POST (`$methods`).
5. `throttle` (`EmailTrackingThrottle`) on `email-tracking` and `email-tracking/*`.
6. `deployThrottle` on `deploy/trigger` and `deploy/complete`.
7. Debug toolbar after the request, except ChatGPT and the two document upload URLs (toolbar HTML would break JSON parsing).

## Data model (from migrations)

Logical names. A non-empty `database.default.DBPrefix` is applied by CodeIgniter.

Identity and shell (2022–2024):

- `users`, `user_role`, `user_access`, `user_menu`, `user_submenu`, menu categories
- `organisations`
- sessions
- `settings`, `language`
- `cronjob_actions`, `cronjob_log`
- `sent_queue` (migration filename spells Qeue)
- `public_holidays`
- email template tables
- two-factor table, `login_history`, blocked users
- `companies`, `company_users` (2025-01-20)
- `password_reset`
- `media` / uploaded file tables from 2023 migrations

Documents (2025-01 onward):

- `documents` (year column added 2026-05-27)
- `document_types`, `document_categories` (vault columns `scope`, `owner_user_id` added 2026-07-29)
- `document_tags`, document search cache
- `user_documents`, `user_access_times`
- `user_document_type_tabs` (year column, then a cleanup of rows without a year)
- `audit_logs`
- `vault_folder_shares`
- `vector_embeddings`
- `chatgpt_logs`, `chatgpt_settings`
- `chat_memories`, `document_contexts`

Bulk email (2026-04, 2026-05, 2026-09):

- `c_email_templates`, `c_email_groups`, `c_email_recipients`, `c_recipients_email_groups`
- `c_email_campaigns`, `c_email_campaign_groups`, `c_email_campaign_recipients`
- `c_email_logs`
- `c_custom_email_tracking`, `c_custom_email_tracking_events`
- campaign queue table from `2026-09-16-100000_CreateBulkEmailCampaignQueueTable.php`

Mailbox (2026-05-19): ticket, message, and attachment tables from `CreateMailboxTicketTables`.

`docs/bulk_email_spec.md` warns that one path in `EmailTracking::customPixel()` uses literal table names without `prefixTable`. If production uses a prefix, that path will not match the migrated tables.

## Auth

Session cookie. Template keys: `app.sessionDriver`, `app.sessionCookieName`, `app.sessionSavePath`, `app.sessionMatchIP`, `app.sessionTimeToUpdate`, `app.sessionRegenerateDestroy`.

Login sets `isLoggedIn`, `userID`, and `role` (role id). 2FA emails a code through `sendMail` when setting `2FA` is on.

`JwtLibrary` exists under `app/Libraries`. Which HTTP routes require a JWT was not found in the filter list. **Unknown / Requires Confirmation** whether JWT is used or leftover.

Password reset uses `password_reset`. The reset bugfix commit is `e6e2031`.

## Documents, OCR, and RAG

Upload path: `Documents::uploadStep1` then `Documents::upload`.

Extraction: `PdfTextExtractor` and `OcrLibrary` shell out to Poppler and Tesseract. `smalot/pdfparser` is also a dependency.

Indexing: `indexDocumentInRAG` chunks text at 1000 characters, embeds, and stores through `app/Libraries/VectorStore/MySQLVectorStore.php`. `app/Neuron/DocumentRAG.php` is the Neuron AI entry. OpenAI client is `orhanerday/open-ai`. Config template: `OPENAI_API_KEY`, `OPENAI_MODEL=gpt-4`.

Vault exclusion is in `document_helper.php` and `DocumentModel` (test `DocumentModelVaultRlsTest`).

Files land under `writable`. Exact subdirectory was not mapped file by file.

## Bulk email worker

Configuration class `Config\BulkEmail` (file `app/Config/BulkEmail.php`):

| Property | Value at `2bcf278` |
|---|---|
| `batchSize` | 50 |
| `cronIntervalMinutes` | 5 (ETA only) |
| `maxAttempts` | 3 |
| `processingStaleMinutes` | 15 |
| `bulkSkipBcc` | true |
| `bulkSmtpKeepAlive` | true |

Command: `php spark send:bulk-email-campaigns`.

Libraries: `BulkEmailCampaignService`, `BulkEmailCampaignQueueService`, `BulkEmailCampaignAttachmentService`, `BulkEmailMailTransport`, `BulkEmailAiTemplateAssistant`.

Direct send helper: `sendMailDirect()` in `mail_helper.php`.

UI assets: `public/assets/js/bulk-email/`, `public/assets/css/bulk-email.css`.

## Mailbox

Command: `php spark mailbox:poll`.

Libraries: `MailboxImapService`, `MailboxImapAttachmentExtractor`, `MailboxMimeParser`, `MailboxThreadingService`, `MailboxTicketImportService`, `MailboxOutboundService`, `MailboxHtmlSanitizer`, `MailboxHtmlPreviewCleaner`.

Env keys are documented in `docs/README.md` and `docs/mailbox_env.md` (`mailbox.imapHost`, `mailbox.imapPort`, `mailbox.imapSslVerify`, `mailbox.imapUser`, `mailbox.imapPassword`, `mailbox.imapFolder`, and the matching `mailbox.smtp*` and `mailbox.fromEmail` keys). `composer.json` suggests `ext-imap`.

HTMLPurifier (`ezyang/htmlpurifier`) supports the sanitiser.

## Cron and queues

Two queues:

1. `sent_queue` for ordinary and scheduled application mail, drained by `CronJobs` when `GET cronJobs` runs inside the weekday window.
2. The bulk-email campaign queue, drained by the spark command above. `CronJobs` can also be configured with a `cronjob_actions` row. `docs/EMAIL_QUEUE_AND_CRON_SETUP.md` describes inserting that row. It does not mention the 2026-09 campaign queue. Operators need both if both systems are in use. **Requires Confirmation** which crontab entries production actually has.

`CronJobs::SECURITY_KEY` is the literal `add your security key here` and is enforced only when `CI_ENVIRONMENT` is `production`.

## Deploy and infrastructure

Pipeline on branch `main` only (`branches: '{main}'`):

1. `php:8.3-cli` image, install intl, gd, zip, `composer install --no-dev`.
2. SonarQube scan.
3. FTPS via `lftp` of files changed since `HEAD~1`, or a full mirror when there is no parent commit. Excludes `tests/`, `.git/`, `docs/`, `scope/`, `vendor/`, `.env`, composer files, phpunit, README, CHANGELOG, pipeline file.
4. `curl` `https://wineauthority.co.za/deploy/trigger?token=$DEPLOY_WEBHOOK_TOKEN` with a 120 second timeout. HTTP status must be 200.

`DeployWebhook::trigger` checks the token, requires `exec()`, starts the host deploy script in the background, and uses `DeployNotificationService` to email start and finish. CLI `php spark deploy:notify-log` emails a deploy log.

Repository variables named in the pipeline comments: `SONAR_HOST_URL`, `SONAR_TOKEN`, `FTP_USERNAME`, `FTP_PASSWORD`, `FTP_SERVER`, `FTP_PORT` (default 21), `DEPLOY_WEBHOOK_TOKEN`.

The pipeline sets `ftp:ssl-force yes` and `ssl:verify-certificate no`.

No Docker Compose file for the app was in the tree. `docs/FTP_DEPLOYMENT_SETUP.md` and `docs/DEPLOYMENT_CHECKLIST.md` are the in-repo runbooks.

Local template base URL is `http://localhost/dms/`. `docs/README.md` also shows an Apache vhost example for `adminOverdrive.local` pointing at `c:/wamp64/www/adminOverdrive/public`, which is the sibling Admin Overdrive layout, not this repository's folder name.

## Security controls present in code

- Session authentication and menu authorisation filters.
- CSRF on POST.
- Time-window filter on documents and mailbox.
- Vault row filters in `DocumentModel`.
- HTML purification on mailbox content.
- Throttles on email tracking and the deploy webhook.
- Folder-mutation throttle (30 requests per 60 seconds).
- Upload extension and size checks.
- `ezyang/htmlpurifier` and CodeIgniter escaping. A full XSS review was not done. `docs/CHAT_SECURITY_AUDIT.md` is a prior note in the source repo.

## Security controls named but not established

- `docs/PROJECT_RULES.md` is the convention file. It was not re-checked paragraph by paragraph against every controller.
- Rate limiting is not global. It is specific to tracking, deploy, and folder mutations.
- Row-level security is application-level queries, not MySQL RLS policies. Confirmed for documents by `DocumentModelVaultRlsTest`. Not confirmed for every other table.
- Captcha on auth routes was not found.
- SonarQube exclusions remove `app/**` controllers, models, filters, helpers, libraries, and tests from analysis, and turn coverage off (`sonar.coverage.enabled=false`). The pipeline step can pass without reviewing that code.

The committed `env` file contains live-looking secrets (database, SMTP, OpenAI, encryption key). They are intentionally omitted here. **Requires Confirmation** that those values have been rotated if the file was ever pushed.

## Background work

| Command | Role |
|---|---|
| `php spark send:bulk-email-campaigns` | Queue due campaigns and send one batch |
| `php spark mailbox:poll` | IMAP ingest |
| `php spark monitor:system` | CPU / usage thresholds |
| `php spark deploy:notify-log` | Email the deploy log |
| `GET cronJobs` | Run active `cronjob_actions`, including `sent_queue` |

There is no queue broker (no Redis, SQS, or database-queue package in `composer.json`). Both queues are MySQL tables polled by cron.

## Development and test process

From `docs/README.md`:

1. Copy `env` to `.env`.
2. `composer install`.
3. `php spark db:create`, `php spark migrate`, `php spark db:seed users`, `php spark key:generate`.
4. `php spark serve` or an Apache vhost whose document root is `public/`.

Seeds that exist: `Users`, `UserRole`, `UserAccess`, `UserMenu`, `UserMenuCategory`, `Organisations`, `DocumentTypes`, `TwoFactorAuthSetting`, `DocumentsMenuSeeder`, `BulkEmailMenuSeeder`, `MailboxMenuSeeder`, `CategoriesMenuSeeder`, `UploadedFile`. `Holidays.sql` is a SQL file in the seeds directory, not a seeder class.

Tests worth knowing:

- Vault and documents: `VaultPermissionsTest`, `DocumentModelVaultRlsTest`, `DocumentFoldersTest`, folder view tests, download/edit/delete helper tests.
- Bulk email: `BulkEmailAuthTest`, `BulkEmailCampaignQueueServiceTest`, `BulkEmailLogSnapshotTest`, migration test.
- Mailbox: `MailboxAuthTest`, MIME parser tests, fixtures under `tests/_support/Fixtures/mailbox/`.
- Deploy: `DeployWebhookTest`.
- Users: `UserProfileAccessTest`.

`docs/testsprite_run.md` records earlier TestSprite runs. Those results were not re-executed for this document.

`scripts/copy-documents-module.sh` and `scripts/copy-mailbox-module.sh` copy those modules to another CI4 app. `docs/bulk_email_spec.md` is the matching porting note for email.

## Integrations (technical)

| Integration | Mechanism |
|---|---|
| OpenAI | `chatgpt_helper.php`, Neuron `DocumentRAG`, bulk-email assistant |
| SMTP | `mail_helper.php` and `BulkEmailMailTransport` |
| IMAP | `MailboxImapService` |
| MySQL vectors | `MySQLVectorStore` |
| FTP deploy | `lftp` in the pipeline |
| Webhook deploy | `DeployWebhook` |
| AWS SDK | Composer require only. Call sites not confirmed. |
| mPDF, PhpSpreadsheet, League CSV | Composer requires. CSV import for recipients is implemented. mPDF and spreadsheet call sites were not listed. |
| Slack | `Slack` controller, route excluded from auth |
| Africa's Talking, Salesforce | Helper files. Bodies were not treated as live integrations. |

## Configuration keys (names only)

From the committed `env` template and docs:

- `CI_ENVIRONMENT`, `CI_DEVELOPER`
- `OPENAI_API_KEY`, `OPENAI_MODEL`
- `email.SMTPHost`, `email.SMTPUser`, `email.SMTPPass`, `email.SMTPPort`, `email.SMTPCrypto`, `email.fromEmail`, `email.fromName`
- `app.baseURL` and `app.session*`
- `database.default.hostname`, `database`, `username`, `password`, `DBDriver`
- `encryption.key`
- `mailbox.imap*` and `mailbox.smtp*` (documented, not all present in the template that was read)
- `deployment.webhookToken` (read by `DeployWebhook`, referenced by the pipeline)
- `TESSERACT_PATH`, `OCR_EXTRA_PATH`

## Discrepancies

1. PHP version: composer `^8.4`, README and pipeline 8.3, overview `^8.0`.
2. `docs/overview.md` is not a trustworthy map of this product.
3. `docs/bulk_email_spec.md` and `docs/EMAIL_QUEUE_AND_CRON_SETUP.md` lag the campaign queue and current route line numbers.
4. `docs/README.md` package list omits Neuron, pdfparser, and HTMLPurifier, which `composer.json` requires.
5. Role seeder versus Developer checks in helpers (`business-rules.md` BR-001).
6. Routes for retail/EDI controllers whose classes are gone.
7. Onboarding snapshot (2026-09-28, commit `45627d3`, with `app_old`) does not match `main` at `2bcf278`.
8. Apache example in the README points at `adminOverdrive`, not `dms_archive`.
9. Sonar exclusions contradict a reading of the pipeline as "the code is scanned".