Storage

Some parts of an app remember things between requests: a job and its progress, how many requests a client made this minute, a session, a refresh token, the answer to an idempotency key. In inkan all of them keep it in one kind of thing, a Store. In one process the built-in memoryStore() is all you need. Behind several processes or servers, hand every part the same shared store, from Redis or the database you already run, and they all see the same state.

import { inkan, rateLimit } from "@vxnsin/inkan";
import { redisStore } from "@inkanjs/redis";
import { session } from "@inkanjs/session";

const store = redisStore(redis); // your own client, connected

const app = inkan({ workers: "auto" })
  .register(rateLimit({ max: 100, store }))
  .register(session({ secret, store }))
  .job("/exports", { store }, buildExport);

Each part keeps its keys apart by a prefix of its own, so one store serves them all. New in 0.8.0.

The Store interfaceCopy Anchor Link

import type { Store } from "@vxnsin/inkan";

interface Store {
  get<T = unknown>(key: string): Promise<T | undefined>;
  set(key: string, value: unknown, ttl?: number): Promise<void>;
  delete(key: string): Promise<void>;
  add(key: string, value: unknown, ttl?: number): Promise<boolean>;    // set if absent: atomic
  take<T = unknown>(key: string): Promise<T | undefined>;               // get and delete: atomic
  increment(key: string, by?: number, ttl?: number): Promise<number>;   // count: atomic
  publish?(channel: string, message: unknown): Promise<void>;
  subscribe?(channel: string, fn: (message: unknown) => void): Promise<() => void>;
  readonly shared?: boolean;
}

What every store guaranteesCopy Anchor Link

• Keys are opaque strings, any characters (:, *, %, _, spaces, unicode). Two keys are the same only when they are equal; none is a pattern or a prefix of another.
• Values are JSON. What get returns equals the value after a trip through JSON.stringify: a Date comes back as a string, undefined properties are gone. undefined is what get and take return for a missing key, never null; null is only ever a stored null.
• TTLs are in milliseconds, and optional. A key is never gone before its TTL, and never returned once its TTL is over by a second or more, so a store that counts in whole seconds rounds up.
• add, take and increment are atomic across every process that shares the store: of fifty adds on one key at once exactly one wins, of fifty takes exactly one gets the value, and fifty increments count to fifty.
• publish and subscribe are both there or both missing. A message reaches every subscriber of its channel in every process, at most once, in the order one process published it. A message published while nobody listens is lost.
• shared: true means several processes see the same keys. inkan refuses to run jobs with workers on a store that is not.

These guarantees are what the parts of inkan build on: a refresh token used once, a login counter that counts every parallel guess, an idempotency key that only one process runs.

memoryStoreCopy Anchor Link

import { memoryStore } from "@vxnsin/inkan";

const store = memoryStore({ max: 50_000 });

The Store inkan has built in, in this process's memory: fast, gone when the process ends, not seen by other workers (shared: false).

TTLs are checked as keys are read, and expired keys are dropped once a minute on a write. add, take and increment are atomic in the process, and publish reaches the process's own subscribers at once. Values are kept as they are given, not copied: do not change one after setting it.

What uses a storeCopy Anchor Link

Without a store each of them keeps its state in a memoryStore() of its own.

Choosing an adapterCopy Anchor Link

Four official adapters implement Store on what you probably run already. Each takes your own client, has no dependencies, and passes the conformance suite.

All four are shared: true and have publish/subscribe, so they carry jobs across workers and servers. The SQL and MongoDB stores decide expiry by the database's clock, so app servers whose clocks disagree still agree on what is over. Every call is a round trip: for thousands of increments a second, Redis is the better store.

Workers and several serversCopy Anchor Link

memoryStore() lives in one process. Run more than one, with workers or behind a load balancer, and each process has its own:

• Jobs are refused: with workers, listen() throws for a job on a store that is not shared, since the request to follow or cancel a job could land on a process that never heard of it. With a shared store any worker answers, and progress and cancels reach the others through publish/subscribe.
• Rate limits count per process: four workers let four times max through. Give rateLimit the shared store and they count together.
• Logins, sessions and idempotency keys live where they were made. A session made in one worker is unknown to the next; a refresh token or an idempotency key could be used once per process. Give each the shared store.

The same store and prefix on every server is all it takes for several servers to agree. Two apps on one Redis or one table keep apart by the adapter's prefix.

Writing an adapterCopy Anchor Link

A store of your own (another database, a cloud key-value service) implements the interface above and proves it with the conformance suite from @vxnsin/inkan/testing:

// store.test.ts
import { randomUUID } from "node:crypto";
import { storeConformance } from "@vxnsin/inkan/testing";
import { myStore } from "./index.ts";

storeConformance(() => myStore(client, { prefix: `test:${randomUUID()}:` }), {
  close: (store) => store.close(),
});

node --test store.test.ts

It registers node:test cases every Store must pass: every method there, and publish/subscribe both or neither; get, set and delete; JSON values of every kind coming back as they went in; look-alike keys (user:1, user:10, user*, user%, USER) kept apart; two stores from make not seeing each other's keys; add, take and increment under fifty concurrent calls; TTLs (set replaces one, increment keeps the first); and publish/subscribe in order, until unsubscribed, when the store has them.

Each test makes its own store, and keys carry a random part, so a database that is not emptied between runs passes too. The TTL test waits about four seconds of real time unless wait moves the store's clock:

import { memoryStore } from "@vxnsin/inkan";
import { storeConformance } from "@vxnsin/inkan/testing";

let now = Date.now();
storeConformance(() => memoryStore({ clock: () => now }), { wait: async (ms) => void (now += ms) });

When you publish it, say which inkan it was tested with (peerDependencies "@vxnsin/inkan": ">=0.8.0 <1.0.0"), and list it on Hanko.