# PRD — test_harness

Evidence date: 2026-10-01.

Source: `git@bitbucket.org:overdrive/test_harness.git`, `main` at `9f2f006` (2025-03-17).

There is no prior PRD in the source repository or in Silicon Ark. Requirements below are recovered from `README.md`, `lambda-sqs-handler/README.md`, `src/`, `lambda-sqs-handler/index.js`, `bitbucket-pipelines.yml`, `appspec.yml`, the CodeDeploy hooks, the SSM YAML, `Dockerfile`, `package.json`, and the six commits on that revision.

`README.md` is a real description of the February 2025 server. Later commits changed the k6 script and added a second runner. Where the README and the code disagree, both are recorded. The code is the implemented behaviour.

## Product purpose

**Documented.** `README.md` title: "Lambda Load Testing Tool". A load test for AWS Lambda using k6, with a REST API to start a test, Docker, and Bitbucket deploy to EC2.

**Implemented.** k6 clients that POST a single JSON fix to an HTTP URL, an Express process that can start one of those clients, shell scripts that start several k6 processes and archive metrics to S3, and a separate Lambda that writes a different payload shape to SQS.

**Inferred.** The fix fields (`lat`, `lng`, `hole`, `speed`, `battery`, `accuracy`, `date`, `source`) and the hole names in the coordinate list are a stand-in for a Tagmarshal device-fix ingest. That product is not in this repository.

**Unknown / Requires Confirmation:** product owner, intended caller, and whether either runner is still used.

## Users and interactions

| Actor | Evidence | Status |
|---|---|---|
| HTTP caller of `GET /health` and `POST /trigger-test` | `src/server.js` | **Implemented**. No authentication. README says endpoints "should be protected in production" and does not implement that protection |
| Operator on a host who runs k6 or `src/start_k6_processes.sh` | README local-dev section and the manual steps at the bottom of `README.md` | **Documented**. The shell path in the README does not match the committed file paths |
| Bitbucket Pipelines on push | `bitbucket-pipelines.yml` | **Implemented** as YAML. Whether the pipeline is enabled is **Unknown** |
| CodeDeploy on an EC2 host | `appspec.yml` and `scripts/*.sh` | **Implemented** as hooks. The host is not in this repo |
| SSM Automation against an instance tagged `Name=tagmarshal-test-harness-test-instance` | `scripts/TagmarshalTestHarnessSSMDocument V1.yaml` | **Implemented** as a template |
| Caller of the SQS Lambda | `lambda-sqs-handler/index.js` | **Implemented** as a handler. The README says HTTP POST. The handler reads `event.body`, which matches an API Gateway proxy event. No API Gateway resource is defined in this repo |
| Person reading CloudWatch | `lambda-sqs-handler/README.md` | **Documented** only. No metric filter or alarm is in the repo |

No user roles, accounts, or sessions exist in this repository.

## Systems the product talks to

1. **k6 target HTTP API.** Default in `src/k6-script.js`: `https://tfl2idmvci.execute-api.us-east-2.amazonaws.com/Stage/device/fix/863717050222574`. Default in `src/k6-script-updated.js` and `src/K6-Updated-MemEfficientS3Archive.js`: `https://lt19abpplb.execute-api.us-east-2.amazonaws.com/Stage/device/fix/863717050222574`. Override for the first script is the k6 environment variable `LAMBDA_URL`. Region in the host name is `us-east-2`, stage path `Stage`. The service behind the URL is not in this repo.
2. **Express** on `PORT` or 3000. CORS middleware is `cors()` with no origin option. JSON body parser is enabled.
3. **AWS CodeDeploy** application name `lambda-load-test`, deployment group `lambda-load-test-group`, S3 key `lambda-load-test/deployment.zip`. Bucket is `$AWS_S3_BUCKET`. Region, keys, and bucket are Bitbucket variables named in the README: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION`, `AWS_S3_BUCKET`, `LAMBDA_URL`.
4. **S3 bucket name `k6testharnessrepo`**, written in the shell wrappers, the SSM document, and the README manual steps. Object names: `test_harness.zip`, `metrics_<process>_<timestamp>.csv`, `aggregated_metrics.csv`.
5. **SQS** from the Lambda, queue URL `process.env.QUEUE_URL`. SDK client is created with default region resolution. IAM example in the lambda README allows `sqs:SendMessage` and `sqs:SendMessageBatch` on one queue ARN. The ARN in that README is a placeholder.
6. **EC2** install paths `/opt/lambda-load-test` (CodeDeploy) and `/home/ssm-user/test_harness` (SSM / shell).
7. **SSM** document name `TestHarnessConfiguration`, assume role `SSM-Automation-TestHarness-ConfigRole`.
8. **Docker** image based on `node:18-slim`, exposes 3000, command `npm start`. Not referenced by the pipeline.

No database, cache, mailer, or monitoring vendor is configured in code. CloudWatch is mentioned only in the lambda README.

## Major workflows

### WF-1 — Health check

**Implemented** in `src/server.js`, `GET /health`.

1. Respond `200` and JSON `{ "status": "healthy" }`.
2. No dependency check. The process being up is the whole check.
3. `scripts/validate_service.sh` waits 10 seconds, curls `http://localhost:3000/health`, and succeeds only if the body contains `"status":"healthy"`.

### WF-2 — Trigger the original k6 script over HTTP

**Implemented** in `src/server.js`, `POST /trigger-test`. **Documented** in `README.md` with the same path and a JSON body field `lambdaUrl`.

1. Read `req.body.lambdaUrl`.
2. If it is missing, respond `400` and `{ "error": "Lambda URL is required" }`. Empty string is treated as missing.
3. If it is present, run `k6 run src/k6-script.js -e LAMBDA_URL=<value>` via `child_process.exec`. The value is concatenated into the command string. It is not quoted and it is not checked as a URL.
4. The HTTP response is sent when k6 exits, not when it starts. There is no request timeout in this file.
5. On exec error, `500` and `{ "error": "Test execution failed", "details": <error.message> }`.
6. On success, `200` and `{ "message": "Load test completed successfully", "output": <stdout>, "errors": <stderr> }`. k6 often writes progress to stderr, so a successful run can still populate `errors`.

The working directory of `exec` is the Node process cwd. CodeDeploy starts the server from `/opt/lambda-load-test` (`scripts/start_application.sh`), so `src/k6-script.js` resolves there. A start from another directory would not find the script. **Inferred** from the relative path. Not executed in this inspection.

This route always runs `src/k6-script.js`. It does not run `src/k6-script-updated.js` or the memory-oriented script.

### WF-3 — k6 script that the server runs

**Implemented** in `src/k6-script.js`.

1. Options object sets `vus: 600` and `duration: '7.2s'`. A ramping scenario (100, then 500, then 1000, hold, ramp to 0) is commented out. The README still describes that commented scenario.
2. Each iteration builds one fix object and `JSON.stringify`s it. The body is the object itself, not `{ "fixes": [ ... ] }`.
3. `POST` that string to `LAMBDA_URL` or the `tfl2idmvci` default, header `Content-Type: application/json`.
4. Checks: HTTP status is 200, and `timings.duration < 500`. A failed check does not stop the test.
5. A module-level counter sleeps `0.25` seconds when it reaches 600, then resets. Because k6 VUs do not share JS state, that counter is per VU, not global. **Inferred** from how k6 isolates VUs. The comment above the payload generator says "13600 devices sends fix every 15 Seconds". That rate is not what the options object configures.
6. Coordinates come from a 53-row list (holes 1–18, plus a row named `Halfway` with `hole: null` and `number: 28`). Accuracy, speed, battery, hole, goalTime, and number are randomised on top of the chosen row. The random lat/lng function is commented out. `section` is commented out and is not sent.
7. The coordinate index starts at 0 and increments before use, so row 0 is skipped. When the index equals the array length, the next read is past the end of the array. **Implemented** in `generateRandomCoordinates`. Not executed here.

`npm test` and the pipeline step "Build and Test" run this file with no `LAMBDA_URL`, so they use the hard-coded default URL if k6 is available.

### WF-4 — Updated single-process k6 script

**Implemented** in `src/k6-script-updated.js`. Nothing in `server.js`, `package.json`, or the shell wrappers calls this file.

1. Environment variables, with defaults: `BATCH_SIZE` 227, `WAIT_TIME_MS` 250, `DURATION_DAYS` 0, `DURATION_HOURS` 0, `DURATION_MINUTES` 1.
2. Duration in seconds is days×86400 + hours×3600 + minutes×60. Default is 60 seconds.
3. Scenario `constant_load`: executor `constant-arrival-rate`, `rate` = batch size, `timeUnit` = wait time in seconds, `preAllocatedVUs` = batch size, `maxVUs` = batch size × 2. Response bodies are discarded.
4. Each iteration POSTs one JSON fix. Hole, goalTime, and number come from the current course row. Accuracy is 0–10, speed 0–30, battery 0–100, source is `"Test"`, date is `new Date().toISOString()`. Hole `null` on the Halfway row is sent as JSON `null`.
5. Default URL is the `lt19abpplb` host. Same two checks as WF-3.
6. `handleSummary` writes `results.csv` with columns `Metric,Value` and rows for HTTP request count, `http_req_failed.values.fails`, average duration, p(95), check passes, and check fails.

The committed `results.csv` uses that header and those six metrics. It was added in `e16fe88` (2025-03-05). It is historical output stored in git, not a measurement from this inspection. The generator labels `http_req_failed.values.fails` as "Failed Requests". In k6, a Rate metric's `fails` count is the number of zero-valued samples. **Inferred:** that label does not match the field. The file's request count and failed-request count are the same number (169702), while check failures are 392. This inspection did not re-run k6.

### WF-5 — Multi-process runner and S3 archive

**Documented** at the bottom of `README.md`: install k6, jq, and awscli; download `s3://k6testharnessrepo/test_harness.zip`; rsync to `/home/ssm-user/test_harness`; chmod scripts under `src/newversion/`; run `./start_k6_processes.sh 3 121 400 "1,,"`.

**Implemented** in `src/start_k6_processes.sh` and `src/run_k6_wrapper.sh`, with these differences from a working layout:

1. `start_k6_processes.sh` parses arguments: process count, batch size, wait milliseconds, and a duration string `days,hours,minutes` (empty fields become 0). The README example `"1,,"` is 1 day, 0 hours, 0 minutes.
2. It starts that many `nohup` copies of `/home/ssm-user/test_harness/src/newversion/run_k6_wrapper.sh`, one second apart, and writes pid files under `src/newversion/pid/`. The committed wrapper is `src/run_k6_wrapper.sh`. The directory `src/newversion/` is not in the repository.
3. It waits until those pid files are not running processes, then builds `aggregated_metrics.csv` from local `metrics_*.csv` and from every `metrics_*.csv` object it lists in `s3://k6testharnessrepo/`, sorts the combined file, and uploads `aggregated_metrics.csv` to that bucket.
4. The wrapper is supposed to run `src/K6-Updated-MemEfficientS3Archive.js` in 60-second slices, convert k6 JSON output to CSV with `jq`, and upload a CSV to the same bucket when the file reaches 200 MB or when the run ends.
5. The wrapper computes `totalSeconds` and then sets `remaining_seconds=$total_seconds`. Those names differ. **Inferred** from the script text: with default bash (no `set -u`), the `while` test does not see a positive integer, so the k6 loop body does not run. This was not executed here.

The k6 file this wrapper names (`src/K6-Updated-MemEfficientS3Archive.js`):

1. Defaults: batch 121, wait 400 ms, duration 7 days (the other components are 0). Imports `fixData` from `./fixData.js` (53 rows, same hole/number sequence as the inline lists).
2. Same constant-arrival-rate shape as WF-4. Default URL is the `lt19abpplb` host.
3. Reuses one payload object. Accuracy, speed, and battery are randomised. Hole, goalTime, number, lat, and lng come from the course row selected by `(__ITER * length + __VU - 1) % length`.
4. `date` on that object is refreshed every 1000 calls of `updatePayload`, not every request. The counter is module state per VU.
5. `http.post` is given the payload object, not `JSON.stringify(payload)`. The other two scripts stringify. **Unknown** what k6 sends for a plain object body. Not executed here.
6. Records custom metrics `response_time` (Trend) and `error_rate` (Rate, 1 when status is not 200). The two checks run only when `__ITER % 10 === 0`.
7. On a thrown POST, the script substitutes `{ status: 0, timings: { duration: 0 } }` and still records metrics.

The wrapper's `jq` filters expect JSON lines with `.metrics[].type`, `.name`, `.sample.timestamp`, and `.values`. **Unknown / Requires Confirmation** whether `k6 run --out json=...` emits that shape. The filters do not match the `handleSummary` object used by WF-4.

### WF-6 — Lambda receives fixes and sends them to SQS

**Documented** in `lambda-sqs-handler/README.md`. **Implemented** in `lambda-sqs-handler/index.js`.

1. `JSON.parse(event.body)`. If `body.fixes` is missing or is not an array, return status 400 and `{ "error": "Invalid request body. Expected \"fixes\" array." }`.
2. A fix is valid only when it is an object and has numeric `accuracy`, `speed`, `battery`, `number`, `lat`, `lng`, and string `source`, `hole`, `section`, `date`. Extra fields are ignored. `goalTime` is not required. `hole: null` fails the string check.
3. If any fix fails, return 400 and `{ "error": "Invalid fix objects found in payload.", "invalidCount": <n> }`. No fixes are sent.
4. If the array length is greater than 1, send `SendMessageBatch` in chunks of 10, in parallel. Each entry body is `{ fix, metadata: { timestamp, source: "lambda-handler" } }`. Entry `Id` is the index as a string.
5. If the length is 1, send one `SendMessage` with the same body shape.
6. If the length is 0, send nothing and still return 200 `{ "message": "Successfully sent to queue", "count": 0 }`.
7. On success after a non-empty send, the same 200 body with `count` set to the array length. Batch `Failed` entries from SQS are not inspected.
8. On any throw (invalid JSON, missing `event.body`, SQS error, missing queue URL), return 500 `{ "error": "Internal server error", "message": <error.message> }` and `console.error` the error.
9. The README's sample fix matches this validator, including `section`. The k6 payloads do not: they omit `section`, add `goalTime`, and are not wrapped in `fixes`.

Deployment steps in the lambda README are manual: `npm install`, zip `index.js`, `node_modules/`, and `package.json`, upload to Lambda. Runtime text says Node.js 18.x, handler `index.handler`, memory 128 MB, timeout 30 seconds, env `QUEUE_URL`. None of that is in a template in this repo. The Bitbucket pipeline does not deploy this function. The CodeDeploy bundle includes the folder only because the pipeline zips the whole working tree.

### WF-7 — Pipeline deploy of the Node server

**Documented** in `README.md` (build, test, zip, S3, CodeDeploy, then the lifecycle hooks). **Implemented** in `bitbucket-pipelines.yml` and `appspec.yml`.

1. Image `atlassian/default-image:3`.
2. Every pipeline, and `main` before deploy: `npm install`, then `npm test` (`k6 run src/k6-script.js`). Artifacts: `node_modules/**`, `src/**`, `scripts/**`, `appspec.yml`, `package*.json`.
3. On `main` only, a second step installs AWS CLI v2, sets credentials from the three AWS variables, zips the workspace excluding `.git`, uploads to `s3://$AWS_S3_BUCKET/lambda-load-test/deployment.zip`, and calls `aws deploy create-deployment` for application `lambda-load-test`, group `lambda-load-test-group`, `file-exists-behavior OVERWRITE`.
4. The deploy step does not `wait` for the deployment. It does not pass `LAMBDA_URL` into CodeDeploy. `scripts/after_install.sh` writes `LAMBDA_URL=${LAMBDA_URL}` into `/opt/lambda-load-test/.env`. That shell variable is not set by the pipeline YAML. **Inferred:** unless the CodeDeploy host environment already defines `LAMBDA_URL`, the file stores an empty value. `src/server.js` does not read `LAMBDA_URL` from the environment for the k6 command. The trigger uses the JSON body. The k6 script reads `LAMBDA_URL` only when the process is started with `-e`.

CodeDeploy hooks, all `runas: root`, timeout 300 seconds:

1. **BeforeInstall** (`scripts/before_install.sh`): `yum update`, Node 18 from NodeSource, a Debian apt source line for k6, `yum install k6`, `mkdir -p /opt/lambda-load-test`, `pm2 stop/delete lambda-load-test` (failure ignored), `npm install -g pm2`.
2. **AfterInstall**: `npm install` in `/opt/lambda-load-test`, `chmod +x scripts/*.sh`, write `.env` with `PORT=3000`, `NODE_ENV=production`, and `LAMBDA_URL`, then `chown` the tree to `ec2-user`.
3. **ApplicationStart**: `pm2 start src/server.js --name lambda-load-test`, `pm2 save`, `pm2 startup systemd -u ec2-user --hp /home/ec2-user`.
4. **ValidateService**: WF-1 curl. Non-zero exit fails the deployment.

`appspec.yml` `files` copies `/` to `/opt/lambda-load-test`. OS is `linux`.

### WF-8 — SSM install onto the Tagmarshal test instance

**Implemented** as `scripts/TagmarshalTestHarnessSSMDocument V1.yaml`. **Documented** in overlapping form by the manual shell block in `README.md` (added in `e89320a`). Not invoked by the pipeline.

Steps, each a `AWS-RunShellScript` against tag `Name` = `tagmarshal-test-harness-test-instance`:

1. `apt-get update`, `npm install -g k6`, install `jq`, `awscli`, `rsync`.
2. `aws s3 cp s3://k6testharnessrepo/test_harness.zip /tmp/test_harness.zip`.
3. Unzip to `/tmp/test_harness`.
4. `rsync` to `/home/ssm-user/test_harness/`.
5. `chown` to `ssm-user`, chmod directories, chmod the two scripts under `src/newversion/`.
6. Delete `/tmp/test_harness` and the zip.

The document does not start k6. Assume role is `arn:aws:iam::<account>:role/SSM-Automation-TestHarness-ConfigRole`, defined in `scripts/TagmarshalTestHarnessSSMDocumentExecRole V1.yaml`.

## Functional requirements

### FR-001 — Health response

The server answers `GET /health` with HTTP 200 and JSON `status` equal to `healthy`.

- **Who:** any HTTP caller who can reach the port.
- **Trigger:** the request.
- **Validation:** none.
- **Failure:** if the process is down, the client gets a connection error. The route itself has no error branch.
- **Provenance:** **Implemented** (`src/server.js`). **Documented** (`README.md`).
- **Confidence:** High.

### FR-002 — Reject a trigger without a Lambda URL

`POST /trigger-test` without `lambdaUrl` returns HTTP 400 and `{ "error": "Lambda URL is required" }`. k6 is not started.

- **Provenance:** **Implemented** and **Documented**.
- **Confidence:** High.

### FR-003 — Run `src/k6-script.js` when a URL is supplied

`POST /trigger-test` with `lambdaUrl` starts `k6 run src/k6-script.js -e LAMBDA_URL=<body value>` and returns 200 with stdout and stderr after the process exits, or 500 if `exec` reports an error.

- **Inputs:** JSON `lambdaUrl`.
- **Output:** the JSON messages in WF-2.
- **Provenance:** **Implemented**. The README shows the request shape and does not specify the 200 body.
- **Confidence:** High for the command string. Runtime success depends on k6 being on `PATH`. **Unknown** on deployed hosts.

### FR-004 — Load profile of the script the server runs

While `src/k6-script.js` is the script behind the API and `npm test`, the active options are 600 VUs for 7.2 seconds. The README stages (2 min to 100 users, 3 min to 500, 5 min to 1000, 3 min hold, 2 min to 0) are comments in that file.

- **Provenance:** **Implemented** options versus **Documented** stages. Conflict stands.
- **Confidence:** High.

### FR-005 — Fix payload from the course list

Each k6 iteration sends one JSON object with `accuracy`, `speed`, `source` `"Test"`, `battery`, `lat`, `lng`, `date`, and the hole-related fields that script includes. Coordinates are taken from the 53-row list (inline in the first two scripts, imported in the third). The list includes holes 1–18 and a `Halfway` row (`hole` null, `number` 28). There is no row with `number` 29.

- **Provenance:** **Implemented**. "Random payload generation" in the README is only partly true: accuracy, speed, and battery are random; coordinates are from the list. `src/k6-script.js` also randomises `hole`, `goalTime`, and `number` instead of copying the row. The other two scripts copy `hole`, `goalTime`, and `number` from the row.
- **Confidence:** High.

### FR-006 — Pass criteria inside k6

A response is checked for HTTP 200 and duration under 500 ms. Failed checks are counted. They do not abort the run. The memory-oriented script runs those checks on every 10th iteration only.

- **Provenance:** **Implemented**. Not in the README.
- **Confidence:** High.

### FR-007 — Constant-arrival-rate runner

`src/k6-script-updated.js` and `src/K6-Updated-MemEfficientS3Archive.js` schedule a constant arrival rate from `BATCH_SIZE`, `WAIT_TIME_MS`, and a duration built from day/hour/minute environment variables. Defaults differ (227 / 250 ms / 1 minute versus 121 / 400 ms / 7 days).

- **Provenance:** **Implemented**. The README example `3 121 400 "1,,"` matches the shell argument order and the memory-oriented defaults for batch and wait, with a 1-day duration instead of 7 days.
- **Confidence:** High for the options objects. **Requires confirmation** which script an operator is expected to run.

### FR-008 — Archive metrics to S3

The wrapper and the starter are written to turn k6 JSON into CSV and upload to bucket `k6testharnessrepo`, including an aggregated file after all processes exit.

- **Provenance:** **Implemented** as shell. Path prefix `src/newversion/` and the `total_seconds` / `totalSeconds` mismatch are part of the same files. **Unknown** whether a host outside git has a corrected copy.
- **Confidence:** High that the committed scripts contain those gaps. Low that a deployment currently succeeds.

### FR-009 — SQS Lambda accepts only a valid `fixes` array

The handler returns 400 for a missing or non-array `fixes`, or if any element fails the field checks in WF-6. It returns 200 after the send calls for a valid array, including an empty array. It returns 500 when parsing or SQS throws, and the body includes `error.message`.

- **Provenance:** **Implemented**. The README documents 200 / 400 / 500 and the sample payload. It does not mention the empty-array success or the metadata wrapper.
- **Confidence:** High.

### FR-010 — Pipeline publishes the server on `main`

A pipeline for branch `main` installs dependencies, runs `npm test`, uploads `deployment.zip` to the configured bucket, and creates a CodeDeploy deployment named `lambda-load-test` / `lambda-load-test-group`. Other branches run only the build-and-test step.

- **Provenance:** **Implemented** in YAML. **Documented** in the README. Success of `npm test` requires k6 in the pipeline image, which the YAML does not install.
- **Confidence:** High for the YAML. **Unknown** for the last real pipeline result.

### FR-011 — Deployment validation

CodeDeploy validation curls local `/health` and fails the deployment when the body does not contain `"status":"healthy"`.

- **Provenance:** **Implemented**.
- **Confidence:** High.

## Constraints

- **No auth on the Express routes.** **Implemented.** The README states that production should protect them. That protection is not in the code.
- **Open CORS.** **Implemented** (`app.use(cors())`).
- **`lambdaUrl` is placed in a shell command.** **Implemented.** Any caller who can reach `POST /trigger-test` chooses the string that follows `-e LAMBDA_URL=`.
- **Default URLs and the device path segment `863717050222574` are in source.** **Implemented.** They are not read from a config file.
- **Two payload contracts.** k6 sends one fix object. The Lambda requires `{ fixes: [ { ..., section } ] }`. They do not satisfy each other.
- **k6 is an external binary.** It is not installed by `npm install` at the repo root.
- **No automated test suite** other than `npm test` launching k6. `lambda-sqs-handler` `npm test` is `echo` and exit 1.
- **Secrets.** `.gitignore` ignores `.env`. No `.env` is committed. AWS key values are not in the tree. The pipeline expects them as Bitbucket variables.

## Behaviour that is not a requirement

- The commented ramp, the commented random lat/lng, and the commented `section` field are not active behaviour.
- `results.json` existed in `e16fe88` and was removed in `e89320a`. It is not current behaviour.
- The Dockerfile is an unused definition relative to the pipeline and CodeDeploy hooks. Whether anyone builds it is **Unknown**.