# Technical documentation — lambda_grinder

Evidence date: 2026-10-10.

Source: `git@bitbucket.org:overdrive/lambda_grinder.git`, `main` at `ffa0b1e039070852e728d32cf3e3a945885a4a84` (2025-03-22, "update files for deployment").

Secrets are named, not copied. The Monday.com bearer token in `controllers/course.js` and the SMTP password in `controllers/modules/trackmap.js` are omitted on purpose.

## Stack

| Piece | Evidence |
|---|---|
| Language | JavaScript |
| HTTP | Express `^4.17.3` (`package.json`, `app.js`) |
| Realtime | Socket.IO `^4.7.2` |
| Data | Mongoose. Root dependency `^6.2.7`. Lambda layer `layers/layer-node-modules/nodejs/package.json` depends on `mongoose` `^8.12.1` |
| Soft delete | `mongoose-delete` on root models. The Lambda `Device` schema in `layers/database/nodejs/models/device.js` exports the schema only and does not attach that plugin |
| Jobs | `node-cron` in `schedule.js`. `grinder.js` uses `setInterval` |
| Cloud | AWS SAM (`template.yaml`), SDK v2 (`aws-sdk` `^2.1692.0`), X-Ray |
| Geo | `@turf/turf` in the Lambda grind and in the node-modules layer |
| Tests | Jest `^29.6.2` and Supertest. One placeholder test |
| Lambda runtime | `nodejs18.x` in `template.yaml`. README also says Node.js 18.x. Root `package.json` has no `engines` field |

## Repository layout

```text
app.js                 Express and Socket.IO
grinder.js             5-second fix-to-round loop
schedule.js            cron
config/database.js     mongoose.connect helper used by app.js
routes/                Express routers
controllers/           Express handlers
models/                Mongoose models for the Node processes
modules/               Domain helpers for the Node processes
middleware/            auth.js, server.js, twoway.js, csvprocessor.js
handlers/teetimes.js   Socket.IO
teetimes/providers/    one file per tee-sheet provider
FixInsert/index.js     Lambda: HTTP to SQS
grind/index.js         Lambda handler actually named by the template
grind/*.js             other drafts, not the handler
layers/database/       Lambda layer: schemas, modules, bd/connection.js
layers/layer-node-modules/   Lambda layer: npm dependencies
layers/LayerAWSSDKONLY/      Lambda layer: aws-sdk and X-Ray
layers/layer1/         unused copy, commented out of the template
template.yaml          SAM application
template-new.yaml      second template, different layer ContentUri values
samconfig.toml         deploy parameters
.aws-sam/build/        committed sam build output
tests/app.test.js      placeholder
testdb/                commented function
```

`layers/structure.txt` documents only the `layers/database/nodejs` tree. It matches the models and modules that are in that directory.

Three copies of the domain code exist: repository root (`models/`, `modules/`), `layers/database/nodejs/`, and `layers/layer1/`. They were not proven byte-identical. The Lambda connection loads schemas from `/opt/nodejs/models` and helpers from `/opt/nodejs/modules`. Root files call `mongoose.model(...)` and export the model.

## Processes

| Start | Mongo connection | Role |
|---|---|---|
| `node app.js` | Hardcoded NLB URL passed into `config/database.js` | HTTP API |
| `node grinder.js` | `mongoose.connect` to the same URL | Round loop |
| `node schedule.js` | `mongoose.connect` to the same URL | Cron |
| SAM `FixInsert` | None | SQS producer |
| SAM `GrindFunction` | `mongodb://${mongoqrnlb}:27017/tagmarshal` | SQS consumer |

Host in all of those strings:

`tagmarshal-poc-data-config-nlb-3d7a4f2e6b8260e7.elb.us-east-2.amazonaws.com`

Database name: `tagmarshal`. No user, password, replica set, or TLS option is set in these calls.

`config/database.js` exits the process with code 0 on connection failure.

`GrindFunction` connection options in `layers/database/nodejs/bd/connection.js`: `serverSelectionTimeoutMS` 5000, `socketTimeoutMS` 15000, `maxPoolSize` 50, `minPoolSize` 5. Models registered on that connection: `Device`, `Fix`, `Section`, `Round`, `Config`.

## SAM application

Template description: "AWS SAM template to deploy grinder.js functionalities as multiple Lambda functions."

Globals: timeout 600 seconds, runtime `nodejs18.x`, handler `index.handler`, X-Ray active, environment `TIMEZONE` from the parameter (default `UTC`).

| Resource | Type | Notes |
|---|---|---|
| `NodeModulesLayer` | Layer | `ContentUri: ./layers/layer-node-modules/nodejs/`, name `layer-node-modules` |
| `DatabaseLayer` | Layer | `ContentUri: ./layers/database/`, name `database`. Code requires `/opt/nodejs/...`, so the zip must contain a `nodejs` folder. This `ContentUri` includes that folder |
| `LayerAWSSDKONLY` | Layer | `ContentUri: ./layers/LayerAWSSDKONLY/`, name `LayerAWSSDKONLY` |
| `ApiGateway` | `AWS::Serverless::Api` | Stage `prod`, regional, X-Ray, method settings on `/*` `*` |
| `FixQueue` | SQS | Name `FixProcessingQueue`, visibility 120 seconds, retention 14 days, redrive to DLQ after 5 |
| `DeadLetterQueue` | SQS | Name `high-volume-dlq`, retention 14 days |
| `ApiErrorAlarm` | CloudWatch | Namespace `AWS/ApiGateway`, metric `5XXError`, sum, period 60, threshold 10, `GreaterThanThreshold`. No `AlarmActions` |
| `FixInsert` | Function | `CodeUri: FixInsert/`, handler `index.handler`, 512 MB, timeout 10, `AmazonSQSFullAccess`, layer `LayerAWSSDKONLY` |
| `GrindFunction` | Function | `CodeUri: grind/`, handler `index.handler`, 512 MB, timeout 120, reserved concurrency 300, policies `AmazonDynamoDBFullAccess`, `AmazonSQSFullAccess`, `AmazonSSMFullAccess`, layers `NodeModulesLayer` and `DatabaseLayer` |

`FixInsert` environment: `REGION=us-east-2`, `QUEUE_URL` = the queue.

`GrindFunction` environment: `REGION=us-east-2`, `QUEUE_URL`, `mongoqrnlb` = the NLB hostname above. A private IP `10.0.12.75` is commented beside it.

`FixInsert` VPC: security group `sg-003babb1267989dfa`, subnets `subnet-0e60752ccfc80074f` and `subnet-069f4c1a796c82f05`.

`GrindFunction` VPC: same security group, those two subnets plus `subnet-0e40cc02c21511972`.

SQS event on `GrindFunction`: batch size 10, maximum batching window 5 seconds.

Throttling on the API: rate limit 10000, burst 20000. Comments in the same file say 5000 requests per second, a 5-minute visibility timeout, and 900 reserved executions. The numbers in the fields are the ones above.

Commented and not deployed: `Layer1`, `Layer2`, `TestDb`, outputs for `HandleFixFunction` and `GetRawFixesFunction`.

`template-new.yaml` uses `ContentUri: ./layers/layer-node-modules/` and `./layers/database/` (the database URI matches the active template's database layer; the node-modules URI does not include the extra `nodejs/` segment that `template.yaml` uses). `samconfig.toml` does not point at `template-new.yaml`.

### Deploy parameters

`samconfig.toml` version 0.1.

| Profile | Stack name | Region | AWS profile |
|---|---|---|---|
| `default` | `EttieneSamLayerDeployTest14` | `us-east-2` | `tagm` |
| `prod` | `EttieneSamLayerDeployTest8` | `us-east-2` | `tagm` |
| `Prod` | `EttieneSamLayerDeployTest14` | `us-east-2` | `tagm` |
| `Production` | `EttieneSamLayerDeployTest14` | `us-east-2` | `tagm` |

Shared settings: `resolve_s3 = true`, `capabilities = CAPABILITY_IAM`, `parameter_overrides = TIMEZONE="UTC"`, `confirm_changeset = true`. `default`, `Prod` set `disable_rollback = true`. `Production` also sets `disable_rollback = true`.

No pipeline file (no `bitbucket-pipelines.yml`) was in the git file list. Deploy is a manual SAM deploy from this config.

Output `ApiEndpoint` ends in `/prod/messages`. The only event path is `/device/fix/{device_id}`.

## Auth

| Gate | File | Rule |
|---|---|---|
| JWT | `middleware/auth.js` | `jsonwebtoken` verify with `JWT_SECRET_KEY`. Header name is `TOKEN_HEADER_KEY` (example `UserToken`). Fallback: `Bearer` header, then `authorization` with the prefix `Bearer` stripped. Sets `req.userId`, `req.playerId`, `req.caddieId`, `req.name`. Updates `User.lastSeen` for a `userId`. Failure: 401 `{valid:false}` |
| Server | `middleware/server.js` | `x-forwarded-for` or socket address must equal `SERVER_IP`, or `ENV` must be `local` |
| Device | `middleware/twoway.js` | Header `DeviceIMEI` must match `Device.deviceId` |
| API Gateway | `template.yaml` | No authorizer on `FixInsert` |

Token routes are behind the server gate, not only the JWT. `GET /token` adds the JWT middleware inside the router.

## Environment

`.env.example` keys: `ENV`, `LARAVEL_AUTH_SERVER`, `PORT`, `JWT_SECRET_KEY`, `TOKEN_HEADER_KEY`, `JWT_EXPIRES_IN`, `SERVER_IP`, `ALLOWED_ORIGINS`, `FIX_LIMIT`.

`.env` is tracked (it is not in `.gitignore`). It has those keys plus `DATABASE_URL`. Values are omitted.

Read by code and not listed in `.env.example`: `LOCALHOST`, `CLOSEST` (grinder fallback `120`), `MESSAGEBIRD_ACCESS_KEY`, `MAILGUN_API_KEY`, `MAILGUN_DOMAIN`, `MAILGUN_FROM_EMAIL`, `VIRTUALTAG_FIREBASE`, `TMAPP_FIREBASE`. `DATABASE_URL` is not read by `app.js`, `grinder.js`, or `schedule.js`.

`LARAVEL_AUTH_SERVER` was not found in a JavaScript read. User calls use `LOCALHOST`.

Lambda environment is the template variables above, not the `.env` file. `TIMEZONE` is injected for every function and `grind/index.js` does not read `process.env.TIMEZONE`.

`dotenv` in the Node processes loads `.env` from the working directory and from the parent directory.

## Data

MongoDB database `tagmarshal`. Important collections follow the model names.

| Model | Role |
|---|---|
| `Config` | Single course configuration and feature flags |
| `Device` | Tag. `deviceId` is the IMEI used by the APIs. Flags `marshal`, `super`, `drinks`. `assignment`, `round`, `lastFix` |
| `Fix` | Point, time, battery, section fields |
| `LastFix` | Cursor for `grinder.js` (`fixId`) |
| `Section` | GeoJSON Polygon, `2dsphere` index, properties `trigger` comment `start\|stop`, `goalTime`, `hole`, `number` |
| `Round` | Play. Locations array, pace, complete flag |
| `Teetime`, `Player`, `Caddie` | Sheet and people |
| `Geofence`, `GeofenceBreach` | Boundaries and hits |
| `User` | `externalId`, `role`, `permissions`, `notifications` |
| `PinSchedule`, `PinTemplate`, `Hole` | Pin of the day |

`modules/main.js` `getIdByTimestamp` builds an ObjectId bound from a date. `grinder.js` and the midnight job use it as a time filter. The helper body was not re-derived in this pass.

Root models use `mongoose-delete` (`deletedAt`, `deletedBy`). Lambda layer schemas that were opened (`device`) do not. A soft-deleted device can therefore be invisible to the Express model and still visible to the Lambda model, or the reverse, depending on the plugin. **Requires confirmation** if both paths share one database.

## Integrations

| System | Where | Direction |
|---|---|---|
| MongoDB on the POC NLB | Entry points and `connection.js` | Read and write |
| `https://{course}.tagmarshal.com` | `controllers/course.js`, `modules/migration.js` | Read during import |
| `LOCALHOST` `/api/roles`, `/api/users/...`, `/api/permissions/...`, `/api/courses/permissions/...`, `/api/caddies/...` | `controllers/modules/user.js`, `modules/user.js`, `controllers/caddietag.js` | Read and write users |
| Tee-sheet HTTP or SOAP | `teetimes/providers/*.js` | Pull tee times. Jonas files are the SOAP-named providers; the SOAP client is the `soap` dependency |
| Golf Genius | `teetimes/providers/golfgenius.js`, `modules/golfGenius.js` | Pull when an API key is set |
| ETX / European Tour | `modules/europeanTour.js` | Midnight job when provider is `etx` |
| Firebase Admin | `modules/notifications.js` | Push. Credential paths from env |
| Mailgun | `modules/caddie.js` | Mail |
| MessageBird | `modules/messagebird.js` | Messaging |
| Gmail SMTP | `controllers/modules/trackmap.js` | Round share. User `customerexperience@tagmarshal.com`. Password omitted |
| SQS | `FixInsert`, `grind/index.js` | Produce and consume |
| Monday.com | `controllers/course.js` | Unreachable, after `return`. Token omitted |
| GG Golf PHP bridge | `teetimes/providers/gggolf.js` | POST to `golflechampetre.tagmarshal.com` |

`package.json` also depends on `firebase-admin`, `axios`, `axios-retry`, `bcrypt`, `csv-parser`, `json2csv`, `nodemailer`, `socket.io`, `validatorjs`, `xml2js`. Not every dependency was traced to a call site.

## Background work

| Loop | Interval | File |
|---|---|---|
| `grind` | 5000 ms, skipped when `busy` unless bypassed | `grinder.js` |
| Legacy fixes | cron `*/5 * * * * *` (every 5 seconds) | `schedule.js` |
| Tee sheets and Golf Genius | cron `0 */5 * * * *` | `schedule.js` |
| Midnight close | cron `0 0 0 * * *` | `schedule.js` |
| Default tee times | cron `10 0 * * *` | `schedule.js` |
| Pins | cron `5 0 * * *` | `schedule.js` |
| Grind Lambda | SQS batch, not a timer | `grind/index.js` |

There is no separate worker framework. `async` is used inside `grinder.js`. The Lambda uses `async-retry`.

## Security notes from the tree

- `.gitignore` comments out `node_modules/` (`# node_modules/`). Layer dependency trees are committed, including the AWS SDK layer that made a full checkout fail on Windows path length.
- `.env` is committed. It holds `JWT_SECRET_KEY` and `DATABASE_URL`.
- `FixInsert` has no authorizer. Anyone who can call the API Gateway URL can enqueue a body.
- Several Express routes under `/device` and all of `/course` have no JWT and no server gate. See business rule BR-034.
- `GrindFunction` is granted DynamoDB and SSM full access and does not use either service in `grind/index.js`.
- `controllers/course.js` hardcodes a Monday.com JWT in dead code.
- `controllers/modules/trackmap.js` hardcodes an SMTP password.
- VPC security group and subnet ids are committed in the template.
- CORS reflects only origins in `ALLOWED_ORIGINS`.

These are observations from the files. Rotation and exposure are **Requires confirmation** with the owner. Values are not repeated here.

## Testing and quality

`package.json` script `test` runs `jest`.

`tests/app.test.js` imports `../app`, which connects to MongoDB and listens, then asserts nothing (`// TODO: write test`) and closes the server and Mongoose.

No other test files were in the git list. No Sonar properties file is on this commit. The onboarding snapshot named `sonar-project.properties` and `sonarqube_scan.bat` on a scan host; they are not in `ffa0b1e0`.

## Local development

**Documented** only partly. The README lists environment variable `TIMEZONE`, SAM resources, and Node 18. It does not list the three `node` commands.

**Inferred** from the files, if someone runs the Node processes:

1. Node 18, because that is what the Lambda runtime and the README name. The root app does not enforce it.
2. A `.env` with at least `PORT`, `ALLOWED_ORIGINS`, `JWT_SECRET_KEY`, `TOKEN_HEADER_KEY`.
3. Network access to the hardcoded Mongo host, unless those three connection lines are changed. They do not follow `DATABASE_URL`.
4. `npm test` will boot `app.js` and try that same host.

SAM build and deploy are implied by `samconfig.toml` and the committed `.aws-sam/build` directory. The CLI commands themselves are not in the README.

## Git history that explains the tree

| Commit | Date | Subject |
|---|---|---|
| `40f05300` | 2024-11-06 | Initial commit |
| `cd41d752` | 2024-11-21 | Grinder lambda breakout |
| `097ed7e2` through `ff379891` | 2025-03-05 to 2025-03-08 | "updated", "sending to Dirk", "adding new layers" |
| `737d492b` | 2025-03-19 | before tear down |
| `ffa0b1e0` | 2025-03-22 | update files for deployment |

History on this clone was 15 commits. The messages do not say that `grinder.js` was removed. It is still the polling implementation.