Sinji

A multi-tenant platform that turns a call recording, or a pasted transcript, into a diarised transcript, a fact-check against the team's knowledge base, a two-sided read of tone (what the words say and what the voice says) and a weighted rubric scorecard, all drawn on one audio timeline. Around it sit a knowledge-base manager with AI curation, a grounded support bot with an agent copilot that streams cited drafts, public voice tools, and an operator console with AI-usage and server-health reporting. It works in Georgian first, with English alongside. We built the whole product, from the Python pipeline to the Next.js front end; this demo is that front end under a fictional brand.

Year: 2026 · Category: AI & analytics

Our role

Full-stack. The analysis pipeline and its data model, the knowledge base and retrieval, the scoring and fact-check logic, the chat engine behind the copilot, the operator console, and the front end.

The problem

A support team that records its calls rarely listens to them. Quality review is a person with a spreadsheet and a few calls a week, so what gets measured is whatever happened to be sampled, and the score depends on who sampled it. In Georgian the usual shortcuts fail twice: transcription of a phone line is imperfect, and a transcript that is wrong in one word can make an agent look wrong. Once a speech model heard "up to 36 months" as "up to 36 years", and the only reason anyone noticed was that the knowledge base said otherwise.

The second gap is what the agent is allowed to say. The answers live in policy documents, product sheets and scripts that change, and some of them (a discount ladder, an internal script) must never be quoted to a customer. A reviewer needs to see, claim by claim, what the agent said, which passage supports or contradicts it, and how the tone of the words compares with the tone of the voice.

The approach

We treated a call as a document with a timeline. The pipeline transcribes with speaker labels, retrieves from the workspace's own knowledge base, extracts the factual claims, verifies each one against its best passages, reads tone from the text and, when the sidecar is available, from the voice, and only then scores. Every analyser is handed the transcript as numbered segments and answers with segment indices, never seconds, so findings stay attached to the right line whatever the timings turn out to be.

Scoring is split on purpose: the model scores each rubric dimension, and code applies the weights. Two dimensions are measured, not judged (accuracy comes from the fact-check, courtesy from the tone read), and a reviewer can edit any score with a note, which creates a revision instead of overwriting the model. The same knowledge base then grounds a customer-facing bot and a copilot that drafts answers for the agent, and a nightly curation job turns what customers keep asking, and what the fact-check keeps flagging, into proposed edits a human accepts or declines.

Highlights

  • A call becomes a timeline: the player, a sentiment ribbon, one finding lane per rubric dimension, and a transcript whose claims are marked supported, contradicted or not in the knowledge base, each linked to the passage behind the verdict.
  • Text tone and voice tone are shown side by side and never averaged. When the words are polite and the voice is not, the banner says so and follows the voice.
  • The model scores, the code weighs. Two rubric dimensions are measured rather than judged, any score can be edited with a note, and every edit is a revision with the model's original kept immutable.
  • The knowledge base can mark a document internal, and the bot then refuses to quote it. The retrieval playground shows exactly why: ranked passages, a score, a margin and a confidence level.
  • The copilot shows its work: stage chips with measured milliseconds (grounding, tier-1 passages, drafts), cited drafts that stream word by word, and Insert, Edit-then-send or Ignore, each recorded as feedback.
  • The operator console covers everything a platform needs: workspaces and their users, server health with live charts, AI providers and per-workspace assignment, a chat-credentials diagnoser, and token usage that drills from workspace to call to single model call.

Architecture

  • Web app: A Next.js static export served by nginx, which proxies `/api` to the API on the same origin. One workspace page serves both customers and the operator, who acts as any workspace through a request header. Uploads report progress over XHR; copilot turns and digests stream over server-sent events.
  • API: A FastAPI service with one principal resolver: operator, workspace (bearer token or API key), registered user, or anonymous by IP. Tenant identity is taken from the token, and every query filters by the workspace id. Many endpoints exist twice, once tenant-scoped and once as an operator twin, over the same implementation.
  • Data and retrieval: PostgreSQL 16 with pgvector. Knowledge-base chunks carry a cosine HNSW index, with a trigram index as keyword fallback; a self-hosted embedding model produces 1024-dimension vectors. One row per recording holds the transcript, analysis, fact-check, tone and scorecard as JSON columns. Migrations are idempotent SQL applied at start-up.
  • AI layer: A provider registry with one default connection per capability (language model, speech-to-text, text-to-speech), per-workspace assignments and bring-your-own keys sealed at rest. Every structured output is a forced tool call against a strict JSON schema. All model calls pass through one service that applies admission control and meters tokens into a usage table, keyed by feature.
  • Analysis pipeline: Speech-to-text with diarisation, retrieval over the knowledge base, analysis, claim extraction with per-claim retrieval and verification, a tone analyser (text, plus an optional self-hosted voice-emotion sidecar whose absence degrades the result to text-only), then rubric scoring. Inputs are rendered as a numbered timeline and outputs are segment indices.
  • Knowledge base and curation: Upload, parse, chunk, embed, with retrieval confidence metrics, an exact and near-duplicate scan, and re-embedding as an async job. A nightly miner clusters customer questions from web, social and phone channels into one proposal per cluster; a proposal is applied only after a person accepts it.
  • Chat engine: An async generator of transport-neutral events, serialised to server-sent events for an external chat service that authenticates with scoped, rotating credentials. A triage step and a grounding gate decide whether the bot may answer; the copilot streams several cited drafts to the agent; a global and per-workspace kill switch can stop the bot.
  • Worker and operations: A worker process on the same image runs curation, queued re-embeds and retention purges. In the API process, a sampler records server health and a flusher records per-workspace load. Everything runs under Docker Compose on one Linux server behind nginx with TLS; a push to the main branch triggers a webhook that pulls, rebuilds and applies the schema.

In numbers

Mocked API routes: 134 · Lazy-loaded pages: 46 · Demo accounts, one per role: 4 · Seeded calls, with 83 fact-checked claims: 17 · Interface languages: 2 · Front-end lines of TypeScript: 40k

Results

  • Review stopped being a sample. Any call can be opened as a timeline in which each claim, each score and each tone reading points at the line it came from.
  • Fact-checking against the team's own documents caught what a transcript alone cannot: a number that sounds right and is wrong, and a transcription error that would have been blamed on the agent.
  • The same knowledge base grounds the QA scorecard, the customer bot and the agent copilot, so a correction made once, through the curation queue, improves all three.
  • Operators run many teams from one console, with their own AI providers and keys per workspace and a visible, token-level account of what every workspace spends.

What is used

  • Front end: Next.js (static export), TypeScript, XHR upload progress, Server-sent events, Web Audio API (editor)
  • API and worker: Python 3.11, FastAPI, Background worker (same image), Scoped bearer tokens and API keys, Fernet-sealed provider keys, psutil (health sampler)
  • Data and retrieval: PostgreSQL 16, pgvector (HNSW, cosine), pg_trgm (keyword fallback), Self-hosted embedding model (1024 dimensions), JSON columns per recording
  • AI layer: Provider registry (language model, speech-to-text, text-to-speech), Forced tool calls with strict JSON schemas, Self-hosted voice-emotion sidecar, Bring-your-own keys per workspace
  • Infrastructure: Docker Compose, nginx with TLS, Webhook-driven deploy script, Idempotent SQL migrations
  • Demo build: React 18, TypeScript (strict), Vite 6, Tailwind CSS 4, React Router 6, TanStack Query 5, i18next (EN/KA), Recharts 2, Radix UI, framer-motion, cmdk, @portfolio/demo-kit mock API

The demo

A fictional re-skin of a real product Boulder built, with the backend replaced by an in-browser mock: seed data compiled into the bundle, answered from memory. Every model result is precomputed (or computed in the tab by small pure functions such as retrieval and the scoring formula), speech synthesis plays pre-recorded clips, and an uploaded file is matched to the closest canned call. Nothing leaves the page, nothing is stored, and a reload restores the pristine data. Sign in as the operator, a workspace owner or member, or a contact-centre agent from the top bar.

[ visit project ↗ ]