# Project plan — test_harness

Evidence date: 2026-10-01.

Source repository: `git@bitbucket.org:overdrive/test_harness.git`.

Inspected revision: `main` at `9f2f006` (2025-03-17, Dirk, "adding deployment scripts"). The scan-host path `~/projects/sonar_scan/repos/test_harness` was not on this machine. The tree below was read from a clone of that git revision. No secret values are copied. Placeholder names in `README.md` (`your_access_key`, `your_bucket_name`) are documentation, not live credentials.

There was no `silicon_ark/test_harness/` directory before this set. The source repository has a `README.md` and `lambda-sqs-handler/README.md`. Those are incorporated below. Where they disagree with the code, the disagreement is recorded and is not silently resolved.

This plan describes what the repository contains. It does not decide what the product should become.

## Product

**Documented** name in `README.md` and `package.json`: a load-testing tool for AWS Lambda, package name `lambda-load-test`, description "Load testing solution for AWS Lambda using k6".

**Implemented** behaviour is broader and, in the later scripts, more specific: k6 posts JSON "fix" payloads to an API Gateway URL whose path is `/Stage/device/fix/<id>`. Comments and filenames say Tagmarshal (`scripts/TagmarshalTestHarnessSSMDocument V1.yaml`, README manual steps, a comment in `src/k6-script.js` that spells it "tagmarshall"). A second component, `lambda-sqs-handler/`, is a Node.js Lambda that validates a `{ fixes: [...] }` body and sends each fix to SQS.

**Inferred** purpose: generate GPS-style device fixes along a golf-hole coordinate list and measure the HTTP API that receives them. The coordinate list is data, not a named course. **Unknown / Requires Confirmation:** the official product name, the course the coordinates belong to, and whether this harness is still operated.

## Branch reality

| Line | Tip | Date | What it is |
|---|---|---|---|
| `main` (default; `origin/HEAD`) | `9f2f006` | 2025-03-17 | Only branch. Six commits from 2024-11-06 to 2025-03-17 |

No other local or remote branches were present at inspection.

Authors on `main`: Dirk (2), jacobrits (2), Ettiene (1), Jaco Brits (1).

History:

- 2024-11-06 — Jaco Brits, `9982a47`, initial commit (`.gitignore` only).
- 2025-02-13 — jacobrits, `d05f843`, Express server, first k6 script, Dockerfile, CodeDeploy `appspec.yml`, Bitbucket pipeline, and the first README.
- 2025-02-13 — jacobrits, `3448466`, `lambda-sqs-handler/` (typo "lamda" in the commit subject).
- 2025-03-05 — Ettiene, `e16fe88`, course coordinate list inside `src/k6-script.js`, `src/k6-script-updated.js`, and committed `results.csv` / `results.json`.
- 2025-03-17 — Dirk, `e89320a`, memory-oriented k6 script, `src/fixData.js`, multi-process shell wrappers, README manual install steps. `results.json` removed.
- 2025-03-17 — Dirk, `9f2f006`, SSM Automation document and IAM role template.

**Discrepancy:** the task-creation scan listed `sonar-project.properties` and `sonarqube_scan.bat` at the repository root. Neither file is in the tree at `9f2f006`.

## Objectives (inferred from the implementation and the READMEs, not from a written plan)

- Run k6 against a device-fix HTTP endpoint, with a configurable URL and a built-in default.
- Offer a small HTTP API (`GET /health`, `POST /trigger-test`) that starts `src/k6-script.js`.
- Offer a separate multi-process k6 runner that writes metrics and uploads CSV files to S3.
- Offer a Lambda that accepts a batch of fixes and forwards them to SQS.
- Deploy the Node server to EC2 with CodeDeploy from Bitbucket Pipelines on `main`.
- Provide an SSM Automation document that installs packages on a tagged EC2 instance and unpacks a zip from S3.

There is no milestone document, ticket list, or roadmap in the repository.

## Scope

### In the tree at `9f2f006`

1. Express server `src/server.js` and root `package.json` (`express`, `cors`, `dotenv`).
2. Three k6 entry points: `src/k6-script.js`, `src/k6-script-updated.js`, `src/K6-Updated-MemEfficientS3Archive.js`, plus shared data `src/fixData.js`.
3. Shell launchers `src/start_k6_processes.sh` and `src/run_k6_wrapper.sh`.
4. Lambda handler `lambda-sqs-handler/` (`@aws-sdk/client-sqs`). Its dependencies are not in the root `package.json`.
5. Container file `Dockerfile` (Node 18 slim, installs k6, `npm start`).
6. CodeDeploy `appspec.yml` and `scripts/before_install.sh`, `after_install.sh`, `start_application.sh`, `validate_service.sh`.
7. `bitbucket-pipelines.yml`: `npm install` and `npm test` on every pipeline; on `main`, zip, upload to S3, create a CodeDeploy deployment.
8. CloudFormation-shaped YAML for an SSM document and an IAM role, under `scripts/`.
9. `README.md`, `lambda-sqs-handler/README.md`, `.gitignore`, `results.csv`.

### Out of this repository

- The API behind the default `/Stage/device/fix/...` URLs. It is not implemented here.
- The SQS queue, the Lambda deployment package upload, the EC2 instance, the CodeDeploy application, and the S3 buckets. They are named in files and are not created by an executable stack in this repo (the YAML files are templates, not a wired pipeline step).
- Any database, queue consumer, dashboard, or login UI.

## System shape

Two install layouts exist in the same repository. They are not connected to each other in code.

```text
Caller
  |  POST /trigger-test { lambdaUrl }
  v
Express on port 3000 (PM2 name lambda-load-test)
  |  exec: k6 run src/k6-script.js -e LAMBDA_URL=...
  v
HTTP POST JSON fix  -->  API Gateway URL (default or request body)

Operator shell
  |  ./start_k6_processes.sh
  v
N x run_k6_wrapper.sh
  |  k6 run src/K6-Updated-MemEfficientS3Archive.js
  v
same style of HTTP POST, plus local CSV / S3 archive

Separate component
  HTTP POST { fixes: [...] } --> lambda-sqs-handler --> SQS
```

CodeDeploy copies the bundle to `/opt/lambda-load-test` and starts `src/server.js` as user `ec2-user`. The SSM document and the shell wrappers assume `/home/ssm-user/test_harness` and a `src/newversion/` directory that is not in git.

## Major components

| Component | Role | Status |
|---|---|---|
| `src/server.js` | Health check and remote trigger of `src/k6-script.js` | **Implemented** |
| `src/k6-script.js` | k6 script the server and `npm test` actually run | **Implemented**. Ramp-up stages in the README are commented out in this file |
| `src/k6-script-updated.js` | Constant-arrival-rate k6 script with a CSV summary | **Implemented**. Not called by the server or by `package.json` |
| `src/K6-Updated-MemEfficientS3Archive.js` | Constant-arrival-rate script with custom Trend/Rate metrics; reuses one payload object | **Implemented**. Invoked only by the shell wrapper |
| `src/fixData.js` | 53 course points exported for the memory-oriented script | **Implemented** |
| `src/start_k6_processes.sh`, `src/run_k6_wrapper.sh` | Multi-process runner, JSON-to-CSV, S3 upload | **Implemented** as files. Paths and a duration variable do not match the tree. See technical documentation |
| `lambda-sqs-handler/index.js` | Validate fixes and send to SQS | **Implemented** as source. No deploy automation in this repo |
| CodeDeploy scripts + pipeline | Build, zip, S3, CodeDeploy, PM2 | **Implemented** as pipeline and hooks. Live AWS account state is **Unknown** |
| SSM YAML + IAM role YAML | Install packages and unpack `test_harness.zip` onto a tagged instance | **Implemented** as templates. Whether they were applied is **Unknown** |
| `Dockerfile` | Image that runs the Express server with k6 installed | **Implemented** as a file. Not referenced by the pipeline or `appspec.yml` |

## Milestones visible in git (history, not a plan)

| When | Commit | What landed |
|---|---|---|
| 2024-11-06 | `9982a47` | Repository created |
| 2025-02-13 | `d05f843` | Load-test server, first k6 script, Docker, CodeDeploy, pipeline |
| 2025-02-13 | `3448466` | SQS Lambda handler |
| 2025-03-05 | `e16fe88` | Course coordinates, second k6 script, sample `results.csv` |
| 2025-03-17 | `e89320a` | Multi-process runner and memory-oriented script |
| 2025-03-17 | `9f2f006` | SSM document and execution role |

Nothing in git after 2025-03-17 was on `origin/main` at inspection.

## Dependencies known from the repo

- Node.js. README says 18+. Dockerfile is `node:18-slim`. `scripts/before_install.sh` installs Node 18 from NodeSource on a yum-based host.
- k6. Used by scripts and by `npm test`. Not an npm dependency of the root package.
- npm packages: `express`, `cors`, `dotenv` (root); `@aws-sdk/client-sqs` (lambda package). `package-lock.json` is present for the root package only.
- AWS services named in files: EC2, S3, CodeDeploy, Lambda, API Gateway, SQS, IAM, SSM, CloudWatch (mentioned only in the lambda README).
- Host tools named in shell/SSM steps: `jq`, `aws` CLI, `rsync`, `unzip`, `pm2`, `curl`.
- k6 remote library in `src/k6-script.js` only: `https://jslib.k6.io/k6-utils/1.2.0/index.js` (`randomIntBetween`).

## Current project status

**Implemented** and present on `main`: the files in the scope list above.

**Unknown / Requires Confirmation:**

- Whether the CodeDeploy application `lambda-load-test` / group `lambda-load-test-group` exists and receives pushes to `main`.
- Whether the Express server is running anywhere.
- Whether `lambda-sqs-handler` was ever uploaded to Lambda, and what `QUEUE_URL` is.
- Whether the SSM document `TestHarnessConfiguration` was deployed, and whether any host still uses `/home/ssm-user/test_harness`.
- Which URL is the intended target. Two different default API Gateway hosts are hard-coded, and `POST /trigger-test` can override the one used by `src/k6-script.js`.
- Whether `sonar-project.properties` and `sonarqube_scan.bat` exist only on the scan host as untracked files.
- Product owner and whether the harness is still in use after March 2025.

## Known inconsistencies (do not treat either side as corrected)

1. README ramp-up (0 to 1000 users over staged minutes) versus `src/k6-script.js` options `vus: 600` and `duration: '7.2s'`. The staged scenario is commented out.
2. README and `package.json` describe a Lambda load test. The k6 scripts call a device-fix URL. The Lambda in this repo expects a different JSON shape (`fixes` array, required `section`) and is not what those scripts call.
3. CodeDeploy layout (`/opt/lambda-load-test`, `ec2-user`, yum) versus SSM/shell layout (`/home/ssm-user/test_harness`, `src/newversion/`, apt). `src/newversion/` is not in git.
4. Pipeline artifact key `lambda-load-test/deployment.zip` in `$AWS_S3_BUCKET` versus SSM download `s3://k6testharnessrepo/test_harness.zip`.
5. `scripts/before_install.sh` uses `yum` and then writes a Debian apt source list for k6.
6. CloudFormation description on the IAM template says "configure MongoDB Config Servers". The role name and output text say Test Harness.
7. `npm test` is `k6 run src/k6-script.js`. The pipeline does not install k6 before that step.