# Technical documentation — test_harness

Evidence date: 2026-10-01.

Source: `git@bitbucket.org:overdrive/test_harness.git`, `main` at `9f2f006` (2025-03-17). Full history on that branch is six commits. Clone used for this note was shallow (`--depth 50`) and still contained every commit.

No Sonar properties, scan batch file, test framework, database migration, or IaC toolchain beyond two CloudFormation-shaped YAML files.

## Repository layout

```text
.gitignore
README.md
package.json
package-lock.json
Dockerfile
appspec.yml
bitbucket-pipelines.yml
results.csv
lambda-sqs-handler/
  index.js
  package.json
  README.md
scripts/
  before_install.sh
  after_install.sh
  start_application.sh
  validate_service.sh
  TagmarshalTestHarnessSSMDocument V1.yaml
  TagmarshalTestHarnessSSMDocumentExecRole V1.yaml
src/
  server.js
  k6-script.js
  k6-script-updated.js
  K6-Updated-MemEfficientS3Archive.js
  fixData.js
  run_k6_wrapper.sh
  start_k6_processes.sh
```

`src/newversion/` is referenced by the shell scripts, the SSM document, and the README. It is not in the tree. `results.json` was added and later deleted. `sonar-project.properties` and `sonarqube_scan.bat` were named by the onboarding file scan and are not in this revision.

## Languages and runtime

| Piece | Choice | Evidence |
|---|---|---|
| HTTP trigger | Node.js, CommonJS | `src/server.js` uses `require`. README: Node.js 18+ |
| Container | `node:18-slim` | `Dockerfile` |
| CodeDeploy bootstrap | NodeSource setup 18.x via yum | `scripts/before_install.sh` |
| k6 scripts | k6 JavaScript (ES module `import`) | `src/k6-script*.js`, `src/K6-Updated-MemEfficientS3Archive.js` |
| Shared course data | ES module `export` | `src/fixData.js`. The Express server does not import it |
| Lambda | Node.js, CommonJS, handler `index.handler` | `lambda-sqs-handler/index.js`. README says runtime Node.js 18.x. Not declared in a SAM/Serverless file |
| Shell | bash | shebang on the four operational scripts under `src/` and `scripts/` |
| Pipeline image | `atlassian/default-image:3` | `bitbucket-pipelines.yml` |

`package.json` does not set `engines`.

## Application structure

### Root package

`package.json` name `lambda-load-test`, version `1.0.0`, main `src/server.js`.

Scripts:

- `start` → `node src/server.js`
- `test` → `k6 run src/k6-script.js`

Dependencies (ranges): `express` ^4.18.2, `cors` ^2.8.5, `dotenv` ^16.0.3. `package-lock.json` is committed (868 lines at the commit that added it). The lambda package is not a workspace member and has no lockfile in the repo.

`src/server.js`:

- Loads `.env` through `dotenv`.
- Port `process.env.PORT` or 3000.
- `express.json()` and `cors()` with no arguments.
- Routes only `/health` and `/trigger-test` (see the PRD). No other middleware.

The trigger uses `child_process.exec`. The command string interpolates `lambdaUrl`. `exec` buffers stdout and stderr. There is no `timeout` option and no `shell: false`.

### k6 programs

All three POST to a URL and check status 200 and duration under 500 ms. None set `thresholds`.

`src/k6-script.js` imports `randomIntBetween` from `https://jslib.k6.io/k6-utils/1.2.0/index.js`. The other scripts use `Math.floor(Math.random() * n)` and do not call that URL. k6 must be allowed to fetch the jslib URL when this file runs, unless that import is cached. **Unknown** whether pipeline or EC2 hosts can reach `jslib.k6.io`.

`options` in `src/k6-script.js` is a live object with `vus: 600` and `duration: '7.2s'`. The `duration` line sits among commented scenario lines and is not itself commented. A `console.log('payload', payload)` runs on every iteration.

`src/k6-script-updated.js` writes `results.csv` from `handleSummary`. Metric names read from the k6 summary object: `http_reqs`, `http_req_failed`, `http_req_duration`, `checks`.

`src/K6-Updated-MemEfficientS3Archive.js` imports `./fixData.js`. k6 can load that local module. Node's root app never does. Custom metrics are `response_time` (Trend, `isTime` true) and `error_rate` (Rate). The POST body argument is the payload object, not a string.

Index bugs visible in source, not executed here:

- `src/k6-script.js` `generateRandomCoordinates` increments a module `index` before the read, skips index 0, and when `index === fixData.length` reads past the array (`if (index > fixData.length)` is strict).
- `src/run_k6_wrapper.sh` sets `totalSeconds` and later reads `$total_seconds` into `remaining_seconds`.

### Lambda package

`lambda-sqs-handler/package.json` name `lambda-sqs-handler`, version `1.0.0`. Dependency `@aws-sdk/client-sqs` ^3.0.0. Its `test` script prints an error and exits 1.

`index.js` constructs `SQSClient` with no region argument (SDK default chain). `QUEUE_URL` is read from the environment at load time. Batch size constant is 10, which matches the SQS `SendMessageBatch` entry limit. `SendMessageBatch` calls for different chunks run under `Promise.all`.

## Data

No database, ORM, or cache.

The only structured data in git is the 53-row course list and `results.csv`.

`results.csv` header:

```text
Metric,Value
HTTP Requests,169702
Failed Requests,169702
Avg Response Time,267.16621001638237
95th Percentile Response Time,277.44946999999996
Successful Status Checks,339012
Failed Checks,392
```

Check counts add up to two checks per request (339012 + 392 = 339404 = 169702 × 2). The "Failed Requests" cell is `http_req_failed.values.fails` from the summary writer. **Inferred** from k6 Rate semantics: `fails` on a Rate is the count of zero samples, so this cell is not a reliable failure count. The file dates from commit `e16fe88`. It was not reproduced.

`.gitignore` ignores `node_modules/`, `.env` and local env variants, IDE files, logs, `dist/`, `build/`, `.docker/`, `k6-results/`, and `.qodo`. `results.csv` is not ignored. It is tracked.

## External integrations

| Integration | Direction | Trigger | Auth in this repo | Failure behaviour in this repo |
|---|---|---|---|---|
| Device-fix HTTP API | Outbound POST from k6 | Each k6 iteration, or `POST /trigger-test` | None in the client. URL may be overridden | Checks record non-200 and slow responses. The process continues. Memory-oriented script catches a thrown POST and records status 0 |
| SQS | Outbound from Lambda | Valid `fixes` length 1 or more | `QUEUE_URL` env and the Lambda execution role (role is not in the repo; README shows a sample policy) | Throw becomes HTTP 500 with `error.message`. Partial batch failures are not handled |
| S3 `k6testharnessrepo` | Outbound upload and list from shell; inbound download from SSM | Wrapper archive, starter aggregation, SSM download of `test_harness.zip` | AWS CLI on the instance. No keys in the scripts | Wrapper logs and, on upload failure, moves the CSV back. Starter appends a failure line. Not executed here |
| S3 `$AWS_S3_BUCKET` | Outbound zip from Bitbucket | Push to `main` after the test step | Bitbucket variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION` | Pipeline step fails if the CLI command fails. No retry block |
| CodeDeploy | Outbound `create-deployment` from Bitbucket; inbound hooks on EC2 | Same `main` step | Same AWS variables. Service role is not in the repo | The pipeline does not wait on the deployment id. Hook failure fails that hook (`set -e` is not set; failing commands still return non-zero unless `|| true`) |
| jslib.k6.io | Outbound import | k6 loads `src/k6-script.js` | None | k6 fails the run if the import cannot be resolved. **Inferred** |
| npm / NodeSource / k6 apt repo | Install time | Docker build, BeforeInstall, SSM | None | Install failure stops that step |

API Gateway, the device-fix service, and any queue consumer are outside this repository.

## Deployment

### Path A — Bitbucket to CodeDeploy to PM2

1. Push. Default branch pipeline is build and test only. `main` continues.
2. Zip excludes paths matching `*.git*` and is uploaded to `lambda-load-test/deployment.zip`.
3. `aws deploy create-deployment` with `--file-exists-behavior OVERWRITE`.
4. `appspec.yml` version 0.0, `os: linux`, destination `/opt/lambda-load-test`.
5. Hooks in order: BeforeInstall, AfterInstall, ApplicationStart, ValidateService. Each timeout is 300 seconds, `runas: root`.
6. PM2 process name `lambda-load-test`. Startup line uses systemd and `/home/ec2-user`.

`scripts/before_install.sh` mixes an RPM/`yum` flow with an apt source file under `/etc/apt/sources.list.d/k6.list` and keyring path `/usr/share/keyrings/`. That combination matches neither a pure Amazon Linux host nor a pure Debian host without extra packages. **Unknown** which AMI the CodeDeploy group uses.

`LAMBDA_URL` in the generated `.env` is the shell's `LAMBDA_URL` at AfterInstall. The pipeline YAML never exports it. The server does not pass `.env`'s `LAMBDA_URL` into k6. The trigger passes the HTTP body's `lambdaUrl` via `-e`.

### Path B — SSM document and manual shell

Templates only. Document type Automation, schema 0.3, name `TestHarnessConfiguration`. Six `aws:runCommand` steps, Ubuntu-style `apt-get`, global npm install of k6, download `s3://k6testharnessrepo/test_harness.zip`. The document does not start a load test.

The IAM template `AWSTemplateFormatVersion` 2010-09-09 creates role `SSM-Automation-TestHarness-ConfigRole` trusted by `ssm.amazonaws.com`. Inline policy allows `ssm:StartAutomationExecution`, `GetAutomationExecution`, `DescribeAutomationExecutions`, `DescribeAutomationStepExecutions`, `ListCommands`, `SendCommand`, `GetCommandInvocation`, `ListCommandInvocations`, `ExecuteAutomation`, `StartSession`, `DescribeInstanceInformation`, and `ec2:DescribeInstances` on `Resource: "*"`. `iam:PassRole` is allowed on `*` when `iam:PassedToService` is `ssm.amazonaws.com`. The template `Description` says the role is for MongoDB config servers. The output description says Test Harness.

Nothing in `bitbucket-pipelines.yml` deploys these templates.

### Path C — Docker

`Dockerfile` installs k6 from `dl.k6.io` using key `C5AD17C747E3415A3642D57D77C6C491D6AC1D69`, copies the repo, `npm install`, exposes 3000, `CMD ["npm", "start"]`. No `docker-compose` file. No pipeline `docker build`.

### Path D — Lambda zip

Manual, per `lambda-sqs-handler/README.md`. Not automated. Environment variable the code reads: `QUEUE_URL`. README also states memory 128 MB and timeout 30 seconds as setup instructions, not as code.

## Environment configuration

| Name | Where it is read | Notes |
|---|---|---|
| `PORT` | `src/server.js` | Default 3000. AfterInstall writes `PORT=3000` |
| `NODE_ENV` | Written to `.env` only | AfterInstall sets `production`. No branch in `server.js` reads it |
| `LAMBDA_URL` | k6 `__ENV.LAMBDA_URL` | Server passes it with `-e` from the JSON body. k6 falls back to a hard-coded URL. AfterInstall writes the shell value into `.env`, which this path does not use |
| `BATCH_SIZE`, `WAIT_TIME_MS`, `DURATION_DAYS`, `DURATION_HOURS`, `DURATION_MINUTES` | The two later k6 scripts | Defaults differ per file. Wrapper also sets `PROCESS_ID`. The memory-oriented script never reads `PROCESS_ID` |
| `QUEUE_URL` | `lambda-sqs-handler/index.js` | Required for a successful send |
| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION`, `AWS_S3_BUCKET` | Pipeline shell via Bitbucket variable substitution | README lists them. Not read by application code |

No `.env.example` is committed.

## Security observations (from the tree, not a penetration test)

- `POST /trigger-test` concatenates caller input into a shell command (`src/server.js`).
- Express has no authentication (BR-001) and open `cors()` usage.
- Default API URLs, the path id `863717050222574`, and bucket `k6testharnessrepo` are committed.
- Lambda 500 responses include `error.message`.
- The SSM role template grants SSM send-command and pass-role on `*`, limited for pass-role by the service condition above.
- AWS access keys are not in the git tree. `.env` is gitignored.
- `npm test` / the pipeline will hit the default device-fix URL if k6 runs.

## Development and testing

Local steps **documented** in `README.md`: `npm install`, `npm start`, `k6 run src/k6-script.js -e LAMBDA_URL=...`.

There is no unit-test directory, no CI lint step, and no Sonar config in this revision. The lambda package's test script always exits 1.

`npm test` is an external load run, not a local assertion suite. The pipeline does not install k6 before calling it. **Inferred:** that step fails on a stock `atlassian/default-image:3` because the YAML never installs k6. Image contents were not inspected.

The memory-oriented script and the wrappers were not run for this document. Findings about the unset `total_seconds` variable and the missing `src/newversion` directory are from reading the files.

## Monitoring

- k6's own summary and checks.
- `handleSummary` CSV for `src/k6-script-updated.js` only.
- Shell logs under `/home/ssm-user/test_harness/src/newversion/log/` and metrics under `.../metrics/`, then S3. Those directories are created by the wrapper (`mkdir -p`) if that script is the one that runs.
- Lambda README says to use CloudWatch invocation metrics, error logs, and SQS metrics. No alarm, dashboard, or log subscription is defined here.
- ValidateService is a one-shot curl during deploy, not a continuous check.

## Related product names in the tree

The string Tagmarshal appears in the SSM filename, the instance tag, and the README. `src/k6-script.js` contains the comment "Replace with tagmarshall AWS Lambda endpoint" (different spelling). No import map or client SDK for another Overdrive repository is present.