add readmes for ui and schema

.gitignore

@@ -100,7 +100,6 @@ web_modules/
 .env.development.local
 .env.test.local
 .env.production.local
-.env.local
 
 # parcel-bundler cache (https://parceljs.org/)
 

README.md

@@ -1,45 +1,64 @@
 # doorman
 
-To install dependencies:
+To install dependencies for all packages
 
 ```bash
 bun install
 ```
 
-To run:
+## Packages
+
+Check out the individual READMEs in the monorepo to see how to run each component of Doorman
+
+- doorman-client (Twilio Function that triggers when receiving buzzer call)
+- doorman-api (Twilio Function API that manages Door state and configs)
+- doorman-ui (frontend React app, served by doorman-api)
+- doorman-schema (future home for all the zod schemas and basic validations)
+
+## CI/CD workflows
+On pushes to main, the Gitea action in `.gitea/workflows/deploy-twilio.yaml` handle deployments to Twilio / homeassistant.
+
+1. Builds change
+2. Run local integration tests
+3. Deploy to staging
+4. Run staging integration tests
+5. Promote to prod
+
+## .env.twiliotemplate
+
+This file is used for twilio Deployments in github actions. In short, it specifies all the env values that are supposed to be deployed with the Function.
+
+For any value that is specified, it uses that value.
+
+For any value that is in the file but not specified (= nothing), it would be loaded from the execution environment (as a Gitea secret). If the env var is not in the environment passed in, then the deployment would fail.
+
+## for twilio packages, what is going on in  src/index.ts (Bun.build)
+
+This file contains a Bun bundler for Javascript. In short, it compiles the Typescript code under ./src/functions into javascript, bundling all the local files together and optionally certain dependencies.
+
+This is necessary because Twilio Functions only supports Javascript, and doesn't allow importing code across files. This previously was really hard to maintain, because Javascript is not typed and the files were really long. Our deployed code is basically the package compiled into one file so it can be deployed in Twilio.
+
+Also, since Bun supports ESM (import . from .) and CommonJS (require) module syntax but Twilio only supports CommonJS, the bundler will explicitly bundle dependencies that only support CommonJS. This is because when deployed in Twilio, we cannot use ESM and Bun handles this conversion for us without the library needing dual support (though most libraries support both ESM and CommonJS).
+
+If we add a library that only supports ESM, then we need to add it as an explicitly bundled dep in src/index.ts
+
+### To deploy manually (not reccomended)
+
+First you need to have Twilio secrets in your env
+
+```
+export ACCOUNT_SID=AC...
+export AUTH_TOKEN=8e...
+```
+
+#### to deploy Doorman API / UI manually
 
 ```bash
-bun run index.ts
+bun run deploy-serverless<:staging>
 ```
 
-This project was created using `bun init` in bun v1.0.3. [Bun](https://bun.sh) is a fast all-in-one JavaScript runtime.
+#### to deploy Doorman Buzzer client 
-
-## Deployments
-
-They are configured to happen in gitea actions for CI/CD. If you need to deploy manually, it should be possible as long as you source the requisite envs first.
-
-## to deploy Doorman API / UI
 
 ```bash
-bun run deploy-serverless
+bun run deploy-buzzer-client<:staging>
 ```
-
-## to deploy Doorman Buzzer client 
-
-```bash
-bun run deploy-buzzer-client
-```
-
-## homeassistant integration poller
-
-in configuration.yaml
-
-```
-switch:
-  - platform: rest
-    name: Doorman
-    icon: mdi:door-closed-lock
-    state_resource: https://doorman.chromart.cc/api/door/info?door=buzzer
-    resource: https://doorman.chromart.cc/api/door/auth?door=buzzer&key=1991 
-    is_on_template: "{{ value_json.status == 'OPEN' }}"
-```

packages/doorman-api/.env.local

@@ -0,0 +1,36 @@
+# twilio stuff (not needed for dev)
+# ACCOUNT_SID=
+# AUTH_TOKEN=
+
+# aws creds (not needed for dev)
+# AWS_ACCESS_KEY=
+# AWS_SECRET_ACCESS_KEY=
+
+# discord stuff (needed for notify, edit, onboarding)
+# DISCORD_BOT_TOKEN=
+# DISCORD_CLIENT_SECRET=
+
+# use local ddb
+AWS_ENDPOINT=http://localhost:5000
+
+DISCORD_GUILD_ID=1299812960553795655
+DISCORD_CLIENT_ID=1299810962366398494
+
+# stage is used in metrics / logs, so just set it to username + dev suffix
+STAGE=$USER-dev
+
+# metrics
+PUSHGATEWAY_URL=https://metrics.chromart.cc
+PUSHGATEWAY_USER=doorman
+PUSHGATEWAY_PW=doormanmetrics
+
+# logs
+LOKI_URL=https://logs.chromart.cc
+LOKI_USER=doorman
+LOKI_PW=doormanlogs
+
+# use local UI endpoint for the redirects in dev
+DOORMAN_URL=http://localhost:3005
+
+# this isn't really a secret, its just to prevent spam on /notify in prod
+NOTIFY_SECRET_KEY=discordnotifyme

packages/doorman-api/.env.twiliotemplate

@@ -5,6 +5,7 @@ AUTH_TOKEN=
 # aws stuff
 AWS_ACCESS_KEY=
 AWS_SECRET_ACCESS_KEY=
+# AWS_ENDPOINT for local dev
 
 # discord notifs
 DISCORD_BOT_TOKEN=

packages/doorman-api/.gitignore

@@ -82,7 +82,6 @@ web_modules/
 .env.development.local
 .env.test.local
 .env.production.local
-.env.local
 
 # parcel-bundler cache (https://parceljs.org/)
 .cache

packages/doorman-api/README.md

@@ -1,4 +1,58 @@
-## deployment is serverless
+# doorman-api
+
+The backend API for Doorman. It is resposible for managing Door state, configuring Door options and sending Discord Notifications.
+
+## dependency services
+
+The Doorman API has hard dependencies:
+- DynamoDB
+- Discord (notify, edit, and onboarding)
+- Cloudflare (only in Production)
+- Twilio Functions (deployed compute in staging / prod)
+
+It has some soft dependencies on other services, mostly self hosted monitoring solutions:
+- Pushgateway (prometheus metrics)
+- Loki (logs)
+
+## running locally
+
+First install the necessary dependencies `bun install`. 
+
+When running locally, Doorman API will use [DynamoDBLocal](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.DownloadingAndRunning.html)
+
+You will likely need java installed on your machine and in your path in order for DynamoDBLocal to work. 
+
+To run the dev server: `bun run start:local-db`
+
+On startup, the main "doorman" DDB table is created and seeded with the a test Door.
+
+Note: DynamoDBLocal is configured to use in memory store only, so any changes will not persist when you exit the command.
+
+The API will run on port 8080 and DynamoDBLocal will use port 5000.
+
+You can verify the seeding worked by calling `http://localhost:8080/api/door/info?door=test`
+
+If you need to test Discord or against real DDB, you should set those env vars in your environment in a seperate .env file and source it before running
+
+## troubleshooting
+
+### Port 5000 is already in use after restarting server
+
+When exiting `bun run start:local-db`, we try to close the previous DDB local run, though this is best efforts it may have failed to close the child process. You can manually find and kill the forked process as follows:
+
+```
+ps -ef | grep DynamoDBLocal
+
+martin    243091  243061 93 17:57 pts/4    00:00:13 java -Xrs -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -port 5000 -inMemory
+
+kill 243091
+```
+
+or if you want to kill all java processes (likely only this DDB local thing)
+
+`kill $(pidof java)`
+
+## deployment
 
 this project deploys the UI and API to twilio functions https://doorman-6741-prod.twil.io
 

packages/doorman-api/package.json

@@ -3,14 +3,16 @@
   "version": "0.0.0",
   "private": true,
   "scripts": {
-    "integ-test:local": "bun run build && export $(grep -v '^#' .env.example | grep -v '=$' | xargs) && concurrently --success first --kill-others \"bun run start-twilio\" \"bun test --timeout 30000 ./tst/integ-local.test.ts\"",
+    "integ-test:local": "bun run build && export $(grep -v '^#' .env.twiliotemplate | grep -v '=$' | xargs) && concurrently --success first --kill-others \"bun run start-twilio\" \"bun test --timeout 30000 ./tst/integ-local.test.ts\"",
     "integ-test:staging": "STAGE=staging bun test --timeout 30000 ./tst/integ-staging.test.ts",
-    "start-twilio": "twilio-run --load-local-env --live --port 8080",
+    "start-twilio": "twilio-run --load-local-env --env .env.local --live --port 8080",
     "watch-build": "bun run --watch src/index.ts",
     "start": "concurrently \"bun run watch-build\" \"bun run start-twilio\"",
+    "local-db": "bun run src/local/localDdb.ts",
+    "start:local-db": "concurrently \"bun run local-db\" \"bun run start\"",
     "build": "bun run src/index.ts",
-    "deploy": "twilio-run promote --from=staging --to=prod --load-system-env --env .env.example",
+    "deploy": "twilio-run promote --from=staging --to=prod --load-system-env --env .env.twiliotemplate",
-    "deploy:staging": "twilio-run deploy --load-system-env --env .env.example --service-name doorman --environment=staging --override-existing-project"
+    "deploy:staging": "twilio-run deploy --load-system-env --env .env.twiliotemplate --service-name doorman --environment=staging --override-existing-project"
   },
   "dependencies": {
     "@aws-sdk/client-dynamodb": "^3.821.0",
@@ -30,8 +32,9 @@
     "zod_utilz": "^0.8.4"
   },
   "devDependencies": {
-    "twilio-run": "^3.5.4",
+    "concurrently": "^9.1.2",
-    "concurrently": "^9.1.2"
+    "dynamodb-local": "^0.0.35",
+    "twilio-run": "^3.5.4"
   },
   "engines": {
     "node": "22"

packages/doorman-api/src/local/localDdb.ts

@@ -0,0 +1,91 @@
+import { CreateTableCommand, DynamoDBClient, KeyType, PutItemCommand, ScalarAttributeType } from "@aws-sdk/client-dynamodb";
+import { DynamoDBDocument } from "@aws-sdk/lib-dynamodb";
+import { DoorAlias, DoorAliasEntity, DoorAliasSchema, getDoorAliasID } from "../schema/DoorAlias";
+import { DoorConfig } from "../schema/DoorConfig";
+
+import DynamoDbLocal from "dynamodb-local";
+import { sleep } from "bun";
+
+console.log("starting local DDB");
+
+const localDdb = await DynamoDbLocal.launch(5000, null, [], true, true);
+
+process.on("SIGINT", async code => {
+  console.log("exiting DDB local");
+  await DynamoDbLocal.stopChild(localDdb);
+});
+
+// wait 5s so we are available
+await sleep(5_000);
+
+// seed ddb
+const client = new DynamoDBClient({
+  endpoint: 'http://localhost:5000',
+});
+
+const tableName = "doorman";
+
+const createTableCommand = new CreateTableCommand({
+  TableName: tableName,
+  KeySchema: [
+    { AttributeName: 'PK', KeyType: KeyType.HASH },
+    { AttributeName: 'SK', KeyType: KeyType.RANGE }
+  ],
+  AttributeDefinitions: [
+    { AttributeName: 'PK', AttributeType: ScalarAttributeType.S },
+    { AttributeName: 'SK', AttributeType: ScalarAttributeType.S }
+  ],
+  ProvisionedThroughput: {
+    ReadCapacityUnits: 5,
+    WriteCapacityUnits: 5
+  },
+});
+
+try {
+  await client.send(createTableCommand);
+} catch (e) {
+  console.error("failed creating table", e);
+}
+
+const document = DynamoDBDocument.from(client, {
+  marshallOptions: {
+    removeUndefinedValues: true,
+  }
+});
+
+const testDoorBuzzer = "6133163433";
+const testDoorName = "test";
+const testDoorPin = "1234";
+
+const doorAlias: DoorAlias = {
+  PK: testDoorBuzzer,
+  SK: "alias",
+  name: testDoorName,
+};
+
+const doorConfig: DoorConfig = {
+  PK: "door-test",
+  SK: "config",
+  buzzer: testDoorBuzzer,
+  pressKey: "4",
+  discordUsers: [],
+  fallbackNumbers: ["1231231234"],
+  pin: testDoorPin,
+  buzzerCode: testDoorPin,
+  timeout: 60,
+  greeting: "test door",
+};
+
+try {
+  await document.put({
+    TableName: tableName,
+    Item: doorConfig,
+  });
+
+  await document.put({
+    TableName: tableName,
+    Item: doorAlias,
+  });
+} catch (e) {
+  console.error("failed seeding table", e);
+}

packages/doorman-api/src/utils/ddb.ts

@@ -3,8 +3,24 @@ import { DynaBridge } from "dynabridge";
 import { DoorConfigEntity, EditDoorConfigEntity, OnboardDoorConfigEntity } from "../schema/DoorConfig";
 import { DoorAliasEntity } from "../schema/DoorAlias";
 import { LockStatusEntity } from "../schema/LockStatus";
+import { DynamoDBClientConfig } from "@aws-sdk/client-dynamodb";
 
 export const createDynaBridgeClient = (context: TwilioContext) => {
+  let config: DynamoDBClientConfig = {
+    region: "us-east-1",
+    credentials: {
+      accessKeyId: context.AWS_ACCESS_KEY,
+      secretAccessKey: context.AWS_SECRET_ACCESS_KEY,
+    },
+  };
+
+  // for local DDB
+  if (context.AWS_ENDPOINT && !context.AWS_ACCESS_KEY && !context.AWS_SECRET_ACCESS_KEY) {
+    config = {
+      endpoint: context.AWS_ENDPOINT,
+    }
+  }
+
   // register all entities here
   return new DynaBridge({
     doorConfig: DoorConfigEntity,
@@ -26,11 +42,5 @@ export const createDynaBridgeClient = (context: TwilioContext) => {
       );
       return convertedObj;
     }
-  }, {
+  }, config);
-    region: "us-east-1" ,
-    credentials: {
-      accessKeyId: context.AWS_ACCESS_KEY,
-      secretAccessKey: context.AWS_SECRET_ACCESS_KEY,
-    },
-  });
 };

packages/doorman-client/.env.local

@@ -0,0 +1,18 @@
+# twilio (not needed for dev)
+# ACCOUNT_SID=
+# AUTH_TOKEN=
+
+DOORMAN_URL=http://localhost:8080
+
+# metrics
+PUSHGATEWAY_URL=https://metrics.chromart.cc
+STAGE=$user-dev
+PUSHGATEWAY_USER=doorman
+PUSHGATEWAY_PW=doormanmetrics
+
+# logs
+LOKI_URL=https://logs.chromart.cc
+LOKI_USER=doorman
+LOKI_PW=doormanlogs
+
+NOTIFY_SECRET_KEY=discordnotifyme

packages/doorman-client/.gitignore

@@ -80,7 +80,6 @@ web_modules/
 .env.development.local
 .env.test.local
 .env.production.local
-.env.local
 
 # parcel-bundler cache (https://parceljs.org/)
 .cache

packages/doorman-client/README.md

@@ -1,51 +1,30 @@
-# Make your apartment buzzer smart with Twilio Functions
+# doorman-client
 
-I wanted to make my apartment buzzer acessible without a physical key, so I created these Twilio Functions to make my buzzer smarter. Features include:
+This is the Twilio Function that is triggered when receiving a call from a buzzer. In short, it's job is to talk to the doorman-api to determine the buzzers configuration and to check if the door should be auto unlocked, or dial through to a phone number on file.
 
-* Voice password (make your friends shout stupid things in public to get into your building!)
+## dependency services
-* PIN password, a classic
+This service has a hard dependency on doorman-api, specifically on the /api/door/info Function. if this is down, the buzzer client will not work.
-* No obvious weirdness or extra waiting for first-time guests or delivery people
-* Notify multiple people until someone picks up the phone - as long as one of the roommates pick up, you won't miss your package
 
-I kind of went overboard I think, given my original goal. But this was actually *really* easy to develop and set up. And also really cheap.
+It has soft dependencies on self hosted monitoring solutions:
+- Pushgateway (prometheus metrics)
+- Loki (logs)
 
-## How to set this up
+## running locally
 
-1. You could `git clone https://github.com/jlian/smart-door-buzzer-twilio-functions.git`, but it's not critical.
+NOTE: You'll want to make sure that doorman-api is also running locally first. .env.local is configured to use http://localhost:8080 as the doorman-api endpoint
-2. Get a Twilio account and valid Twilio number.
-3. Go to https://www.twilio.com/console/runtime/functions/manage and hit **+**.
-4. Add each of the 4 `.js` files into its own function with names that you'd remember.
-5. Go to https://www.twilio.com/console/runtime/functions/configure and configure the environment variables:
-  * `TWILIO_PHONE` with the Twilio number you bought
-  * `PASSPHRASE` for voice password
-  * `PASSCODE` for PIN
-  * `JOHN_PHONE` and others for your cellphone number
-5. Go to https://www.twilio.com/console/phone-numbers/incoming and select the phone number you bought earlier.
-6. Scroll to where it says **A call comes in**, select **Function**, and then the function that corresponds to `buzzer-activated.js`.
-7. Contact your HOA to make the Twilio number your buzzer number - this might be the hardest step.
 
-## How this works
+First install the necessary dependencies `bun install`.
 
-[Twilio Functions](https://www.twilio.com/functions) is pretty sweet. It's completely serverless so you don't need a VM or computer to keep running an app. It's perfect for something small scale like your apartment buzzer. The flow of this program goes like this:
+Run the doorman-client
 
-1. A call comes to the Twilio phone number, `buzzer-activated.js` runs.
+`bun run start`
-1. The [Gather](https://www.twilio.com/docs/api/twiml/gather) verb is used to get either a voice password or a 4-digit PIN.
-   1. If correct, `door-open.js` dials a `9` to the buzzer (my building's buzzer code, yours may be different), which opens the door.
-   1. If incorrect, `call-residents.js` calls all the residents until someone picks up and manually dial `9` to open the door.
-1. When everything is finished, `text-me.js` texts a specified number with info on what happened.
 
-## How much this costs
+To test doorman-client head over to 
 
-According to Twilio docs, collecting speech is charged at $.02 per 15 seconds. A Twilio number costs $1/month. Looking at my own billing dashboard, it never exceeds $2/month - pretty reasonable. 
+`http://localhost:4500/buzzer-activated?From=6133163433`
 
-## Twilio tooling
+This will simulate a buzzer calling from 6133163433 (the test buzzer)
-Using the twilio serverless toolbox you can locally run the functions and test.
 
-```
+If you let it poll for 10s then it should respond with Twilio xml saying to dial fallback numbers.
-bun run start
-```
 
-Deploy to your account with 
+If doorman-api is not running, it would always return with a `<Reject/>`
-```
-bun run deploy
-```

packages/doorman-client/package.json

@@ -5,11 +5,11 @@
   "scripts": {
     "test": "echo \"Error: no test specified\" && exit 1",
     "watch-build": "bun run --watch src/index.ts",
-    "start-twilio": "twilio-run --live --port 4500",
+    "start-twilio": "twilio-run --live --load-system-env --env .env.local --port 4500",
     "start": "concurrently \"bun run watch-build\" \"bun run start-twilio\"",
     "build": "bun run src/index.ts",
-    "deploy": "twilio-run promote --from=staging --to=prod --load-system-env --env .env.example",
+    "deploy": "twilio-run promote --from=staging --to=prod --load-system-env --env .env.twiliotemplate",
-    "deploy:staging": "twilio-run deploy --load-system-env --env .env.example --service-name buzzer --environment=staging --override-existing-project"
+    "deploy:staging": "twilio-run deploy --load-system-env --env .env.twiliotemplate --service-name buzzer --environment=staging --override-existing-project"
   },
   "dependencies": {
     "@twilio-labs/serverless-runtime-types": "^4.0.1",

packages/doorman-schema/README.md

@@ -0,0 +1,5 @@
+# doorman-schema
+
+WIP: this will be the home for all the zod schemas in Doorman. The reason to make it a separate package, is so that we can share base validations between doorman-ui and doorman-api.
+
+Currently only home to the auth schemas.

packages/doorman-ui/README.md

@@ -1,46 +1,27 @@
-# Getting Started with Create React App
+# doorman-ui
 
-This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).
+The React frontend for Doorman.
 
-## Available Scripts
+## running locally
 
-In the project directory, you can run:
+NOTE: you should be running at least doorman-api before starting the UI
 
-### `yarn start`
+Install deps: `bun install`
 
-Runs the app in the development mode.\
+Start dev server
-Open [http://localhost:3000](http://localhost:3000) to view it in the browser.
 
-The page will reload if you make edits.\
+`bun run start`
-You will also see any lint errors in the console.
 
-### `yarn test`
+UI will be served on port 3005, visit the test door page http://localhost:3005?door=test
 
-Launches the test runner in the interactive watch mode.\
+The UI proxies /api to localhost:8080 (doorman-api) running locally so make sure this is also running
-See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information.
 
-### `yarn build`
 
-Builds the app for production to the `build` folder.\
+# sample e2e testing with all components
-It correctly bundles React in production mode and optimizes the build for the best performance.
+you can do e2e testing with the UI by running doorman-api doorman-client and doorman-ui
 
-The build is minified and the filenames include the hashes.\
+visit http://localhost:3005?door=test&key=1234 and click through until reaching the buzzer part
-Your app is ready to be deployed!
 
-See the section about [deployment](https://facebook.github.io/create-react-app/docs/deployment) for more information.
+in another tab simulate buzzer call to test buzzer with http://localhost:4500/buzzer-activated?From=6133163433
 
-### `yarn eject`
+UI should show the door was unlocked. doorman-client should return `<Play>` tags to simulate pressing keys on the phone to allow access
-
-**Note: this is a one-way operation. Once you `eject`, you can’t go back!**
-
-If you aren’t satisfied with the build tool and configuration choices, you can `eject` at any time. This command will remove the single build dependency from your project.
-
-Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except `eject` will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own.
-
-You don’t have to ever use `eject`. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it.
-
-## Learn More
-
-You can learn more in the [Create React App documentation](https://facebook.github.io/create-react-app/docs/getting-started).
-
-To learn React, check out the [React documentation](https://reactjs.org/).