Wearing the Agent: Engineering a Family-and-Friends Personal Agent, from Group Chats to Glasses
Sai Krishna Rallabandi
- When
- Thursday, July 23:45 PM – 4:05 PM · 20 min
- Where
- Track 3San Francisco, CA · imported from ai.engineer's public schedule feed
About this session
Judith is a personal AI agent that has run in daily production for a year, used by more than a dozen of my family and friends across three WhatsApp group chats, Telegram, and Discord. This talk walks through how it's built, in two parts. The first part is the engineering that makes one agent safe for many people to share: a multi-tenant permission model (read-only for my mom, exec for me), a memory stack — FAISS + Neo4j + curated long-term notes — that stays useful over a year instead of bloating into noise, cron-scheduled subagents that scout and act on their own, and the guardrails it enforces on every message — redact personal info before posting to a group, never reply to the wrong person, and screen attacker-controllable text for prompt injection before acting on it. The second part takes the agent off the screen and onto a $50 pair of smart glasses. It captures what I see, describes and stores it as a running visual memory, sets destination path on maps before I get onto car, finds and tells me which aisle in the store to go to first, etc. I cover the latency budget that keeps it conversational — on-device Whisper for speech, cloud reasoning, sub-one-second round trips — and the custom neural voice it speaks in rather than stock TTS, drawn from my speech-synthesis background. Both parts are shown live, including a candid look at the pieces that don't work yet. Audience takeaways: A multi-tenant architecture for a personal agent multiple people actually share A memory design that survives real long-term use (not just a vector store) A defensive checklist for any agent that ingests untrusted text A blueprint for an ambient, vision-aware wearable interface on commodity hardware, with a real latency budget
Speaker
Director, Data Science, Fidelity Investments
Sai Krishna Rallabandi is Director, Data Science at Fidelity Investments, where he leads applied LLM and AI-agent work. He has been awarded Meta fellowship for his PhD in Computer Science from Carnegie Mellon University. His applied work has taken first place at multiple Finance for NLP challenges over the past 4 years. His research spans speech and language processing with a focus on financial data.
More in AI in Finance
- ALPHALAB: Autonomous Multi-Agent Research Across Optimization Domains with Frontier LLMsThursday, July 2 · 10:45 AM – 11:05 AM · Track 3
- Why Off-the-Shelf AI Doesn't Understand MoneyThursday, July 2 · 11:10 AM – 11:30 AM · Track 3
- Let's integrate AI Agents in Event-Sourced SystemsThursday, July 2 · 11:40 AM – 12:00 PM · Track 3
- How Kepler Built Verifiable AI for Financial ServicesThursday, July 2 · 12:05 PM – 12:25 PM · Track 3
- Build for the Memo, Not the Demo — Notes from 200 Investment CommitteesThursday, July 2 · 1:30 PM – 1:50 PM · Track 3
For developers: this programme is open data — JSON, iCal, schedule XML and an MCP endpoint.Show endpointsHide
- JSONEvery published session and speaker, in one request./aie-worldsfair-2026-import/feed.json
- iCalSubscribe in Google, Apple or Outlook Calendar./aie-worldsfair-2026-import/feed.ics
- Schedule XMLfrab / pentabarf — the format conference apps import./aie-worldsfair-2026-import/feed.xml
- MCP + RESTPoint Claude at the programme. OpenAPI 3.1 included./agents
No key, no signup, CORS open. Everything here is generated from the same data the organisers edit.