[ 0.000] booting eshan.service on blr-prod-01
[ OK ] Migrated mysql → alloydb: 365/365 tables, 0 rows lost
[ OK ] Moved workloads aws → gcp in 10 days
[ OK ] Started livekit-recorder.service[ OK ] Reached target on-call.target
Eshan Pandey
Platform & backend engineer. 3 years running production infrastructure on AWS and GCP. I migrate databases, move clouds, and keep live video streaming up. Go and Ruby on Rails, Kubernetes underneath.
# HELP eshan_deploy_time_reduction_percent CI/CD on AWS ECS + CodePipeline# TYPE eshan_deploy_time_reduction_percent gaugeeshan_deploy_time_reduction_percent{pipeline="ecs"} 60eshan_incident_response_reduction_percent{src="cloudwatch"} 40eshan_cloud_migration_window_days{from="aws",to="gcp"} 10eshan_db_migrations_total{from="mysql",to="alloydb"} 1eshan_cutover_tables_verified{gate="fingerprint"} 365eshan_cutover_rows_lost0eshan_db_cutover_window_hours{db="core"} 3eshan_gas_reduction_percent{contract="cryptoballot"} 25eshan_awards_total{name="bravo",year="2025"} 1
$ cat docs/adr/mysql-to-alloydb.md# core app DB, planned and cut over solo
# ADR: move the core database from MySQL to AlloyDBstatusaccepted · shipped · 0 rows lostowner eshan (benchmark → schema → CDC → validation → cutover)
## why
The roadmap needed AI search: vector embeddings for natural-language course
discovery. AlloyDB is Google's PostgreSQL-compatible database with vector search
built in, next to the app data. Getting there meant leaving MySQL.
## evidence first
Benchmarked both on identical 4 vCPU / 32 GB instances with the real Rails app:
~1.5M requests. Reads were a tie. Under a 2,000-signup burst AlloyDB had
0 failed signups (MySQL: 0.5%) and 39% better p95 write latency.
## decision
Heterogeneous migration with a blue-green cluster:
· stream every change MySQL → AlloyDB with CDC while prod keeps running
· build a second GKE cluster running dark against AlloyDB
· at cutover: freeze, reconcile, fingerprint every table, flip DNS
· MySQL is only ever read, so rollback stays a DNS flip away
## consequences+ AlloyDB embeddings and RAG course discovery are unblocked (see wip/)+ better write tail latency and reliability under bursts
- two engines, different types and SQL semantics, synced live
- MySQL-isms hide in app code (so I built a scanner, see tools/)
$ stat pipeline/cdc# how both databases stayed in step
──▶──▶──▶──▶
› click a stage to inspect it
$ ./cutover --window 02:00-05:00 IST# replay the real cutover night
02:00 freeze: both clusters scaled to zero, zero writes verified (not assumed)
02:1x CDC drained, max ids identical on hot tables
02:24 schema migrations on AlloyDB (106s)02:36 reconcile 365 tables, 4 parallel workers (86 → 55 min)03:32fingerprint gate: 365/365 MATCH03:33 reset 367 sequences, ANALYZE full DB
03:54DNS flip to the new cluster, IPv4 + IPv6 verified
04:1x91% of traffic on AlloyDB. done inside the window
$ ls /var/log/postmortems/# bugs that threw no errors. click to open.
DATA The script that deleted live rows while reporting success
symptom a reconcile run finished "ALL DONE", yet ~10,400 rows were missing on the target
cause the MySQL client defaulted to autocommit off + REPEATABLE READ, so the first
SELECT pinned one snapshot for a multi-hour run. rows inserted mid-run were
invisible to the script but already replicated by CDC, so they looked like
"extras" and were deleted. the safety re-check read the same stale snapshot.
proof every missing row's created_at sat in one 30-min band starting the second
the script started
fix autocommit on. a re-verify that shares the snapshot is not a safety check.
outcome caught before cutover. cutover night: 0 rows lost
DATA cancel vs drain: tens of millions of rows gone in one click
symptom 10 streaming jobs silently ran pinned at 2 workers. relaunching them fixed
throughput, but ~183 tables came back short, some by 20M+ rows
cause "--max-workers" alone does not enable autoscaling. and "cancel" drops data
that was acked but not yet written. "drain" flushes it.
twist draining the 4 biggest jobs forced a flush their 15 GB workers couldn't
hold: OOM crash loop. moved them to 32 GB highmem workers.
fix full re-backfill of all 183 tables, verified with exact COUNT(*) on both sides
lesson drain, never cancel. estimated row counts are not evidence.
DATA Rows that vanished with no error, no dead-letter, no trace
symptom the same ~18k rows of one table failed to land on 4 backfills in a row.
then a sweep found 24 tables stuck the same way.
cause the streaming job's per-key dedup state was poisoned by earlier churn. it
silently dropped any row whose sort key didn't beat the stored state.
proof replayed one "lost" row by hand inside a rolled-back transaction: postgres
accepted it. a fresh-state job then recovered the whole table.
lesson recover with a fresh-state job, never by re-feeding the same one
DATA Counts matched, checksums matched, data was 3.5 months old
symptom found on cutover night: 177 records in the dead-letter queue were updates
whose row already existed. right id, stale content.
why hidden row count matches, SUM(id) matches, fingerprint says MATCH, id-based
reconcile never touches it. one row was 3.5 months stale.
fix whole-table force-upsert of the 5 affected tables, inside the window
lesson every check that compares ids is blind to content drift
INFRA Connection storm with nothing in the logs
symptom background workers hit "server closed the connection unexpectedly"
not it not the worker, not connection lifetimes, not the database
cause the connection pooler had its own autoscaler flapping 4→3→4 pods in two
minutes. each scale-down sent SIGTERM, which the pooler treats as
"exit now", killing every client on that pod. killed processes log nothing.
fix preStop sends SIGINT (its graceful shutdown), 60s grace period,
10-minute scale-down stabilization
INFRA Deploys failing: "IP space exhausted" with half the cluster idle
symptom a rolling deploy couldn't schedule pods, and the autoscaler couldn't add nodes
math one /19 pod range shared by six clusters. at 110 max pods per node, each
node reserves a /24 (256 IPs) → a hard ceiling of 32 nodes, total.
real usage: ~14 pods per node. 12× over-reserved.
fix new node pools at 32 max pods (/26 per node) on a fresh range: 4× denser
per node, room for 1,024 nodes. rehearsed in staging with zero downtime.
lesson max-pods is immutable on a pool. 33 costs the same as 64. 32 is the sweet spot.
MEMORY A worker that ate 1.3 GB while doing nothing
symptom messaging worker memory climbed 459 MB → 1,350+ MB and never came back
digging live heap dumps via rbtrace: 135,970 APM Layer objects retained at idle.
the APM agent stored each job's trace in a thread-local that was never
cleared, so 10 threads each kept the previous job alive.
fix APM off for background jobs + jemalloc decay tuned to 1s
result idle RSS 1,350 MB (growing) → ~730 MB (stable), spike-and-release pattern
$ quiz silent-bugs# MySQL ran this for years. what did it do?
where("exam_date >= #{Time.zone.today}") # note: no quotes around the date
[MIGR]Independently planned and executed the MySQL → AlloyDB (PostgreSQL) migration of the core application database, from schema design through a blue-green production cutover: 365/365 tables verified, zero data loss.
[MIGR]Built a real-time CDC pipeline with DataStream, Dataflow and Pub/Sub to keep data in sync during the move with minimal downtime.
[ARCH]Made the backend vendor-agnostic with the Strategy Pattern: authenticator and cloud provider became configuration, not hardcoded dependencies. That groundwork made the multi-cloud migration possible.
[MIGR]Migrated OpenSearch and AWS Lambda workloads in the team push that moved critical infrastructure from AWS to GCP in a 10-day window.
[SHIP]Build features on Superlive, the live class streaming platform, including automatic session recording in the Go-based LiveKit WebRTC media server. Primary point of contact for the area. The stack includes a Node.js/Fastify resource manager that schedules Linode VMs through BullMQ.
[MIGR]Moved backend services and async workers from instances to GKE, deployed with Helm.
[ARCH]Designing natural-language course discovery: RAG over embeddings stored in AlloyDB, resolving open-ended queries into courses, articles and assessments.
[SHIP]Built a Coding Challenge module with secure sandboxed code execution for interactive programming assessments.
[PAGE]On-call for production: API downtime, database issues, failed deploys. Wired CloudWatch monitoring into Sidekiq workloads, cutting incident response time by 40%.
[SHIP]Automated CI/CD on AWS ECS + CodePipeline, cutting deploy time by 60%.
EXL Service
Graduate Engineer TraineeOct 2023 → Mar 2024 · Remote
Python · Jenkins · data engineering pipelines
[INFO]Technical rotation in Python-based data engineering and Jenkins deployment automation.
LCEPL
Software Development InternApr 2022 → Oct 2022 · Pune
Flask · JavaScript · CSS
[SHIP]Built an automated email response engine in Flask, improving response latency by 50%.
[SHIP]Front-end work in JavaScript and CSS that lifted user engagement metrics by 25%.
$ kubectl describe system superlive# live class streaming, I own the feature area
──▶──▶──▶──▶
› click a component to inspect it
# design rules I work to1 the product backend never talks to video infrastructure. it asks the
resource manager, which owns placement, capacity, VMs and recording.
2 recording is intent vs observed state. the host sets intent, a reconciler
converges reality to it, so a crashed recorder heals itself. a human stop
flips intent, which is how "stopped on purpose" differs from "crashed".
3 a self-hosted server dies mid-class → users reconnect onto LiveKit Cloud
automatically, and recording resumes as cloud egress.
$ CLOUD_PROVIDER=gcp rails runner 'p CloudProvider.current'# strategy pattern, flip it
# illustrative sketch of the pattern, not production codeclassCloudProvider
STRATEGIES = { aws: AwsStrategy, gcp: GcpStrategy }
def self.current
STRATEGIES.fetch(ENV["CLOUD_PROVIDER"].to_sym).new
endend=> #<GcpStrategy># same code path, different vendor. this is what made the AWS → GCP move possible.
$ k6 run puma-ladder.js# tuning Rails concurrency on GKE. pick a config.
# why: Ruby's GVL lets one thread run Ruby at a time per process. threads
# beyond what one worker can feed just queue behind the lock. workers add real
# parallelism, threads add overlap on I/O. measure, don't guess.
$ ls ~/bin# tools I built to make the big moves safe
pg_compat_scan
Static scanner for MySQL-isms that break, or silently change behaviour, on PostgreSQL. Pure Ruby, no gems, no DB, no Rails boot. Sorts every finding into actionable, review or analytics-only.
Per-table fingerprints on both databases, compared before any traffic moves. Parallel reconcile on idle cores cut the in-window run from 86 to 55 minutes.
365/365 MATCH IDSET_DIFF=0 → go
live alerts framework (in review)
Error alerting for the live-class stack: environment tags, email per error type (not per occurrence, so nobody mutes it), and error-tracker faults pulled into the same incidents.
scope live servers · resource manager · rails · composer
$ git log --oneline -1 feature/course-discovery
(WIP) design natural-language course discovery with RAG over AlloyDB
# a learner types "how to build gundam" and gets courses from that school,
# each with a one-line reason grounded in lesson transcripts.rule 1 retrieval is always scoped to one school
rule 2 generated text may only restate what was retrieved. no world knowledge.
rule 3 below a relevance floor, return nothing. nearest-neighbour search on a
small corpus always finds something; confident wrong answers are worse.
──▶──▶──▶──▶
› click a stage to inspect it
$ pagerctl simulate # a replay of the kinds of incidents I handle. pick one.
pager idle. no active incidents. trigger one above.
Decentralized voting protocol on smart contracts for tamper-proof results. Refactor cut gas use by 25%.
$ tail /var/log/eshan/events.log
2025AWARD Bravo Award for the critical AWS → GCP migration, delivered in 10 days
2023EDU B.Tech, Information Technology, Bharati Vidyapeeth University, Pune · GPA 8.9/10(2019–2023)
$ ps aux | grep off-hours# when the pager is quiet
USER PID %CPU STAT COMMAND
eshan 110197.0Rskyrim --dragonborn --one-more-quest
eshan 100764.0Rjames-bond --licence-to-kill
eshan 204842.0Sread --books
I build and run the systems behind web products: databases, cloud infrastructure and live video. 3 years in production on Google Cloud and AWS, working in Go and Ruby on Rails.
Moved the company's main database from MySQL to AlloyDB (PostgreSQL). All 365 tables were checked before switching over, no data was lost, and the switch fit inside a 3-hour overnight window.
Helped move core infrastructure from AWS to Google Cloud in 10 days. Won the Bravo Award (2025) for it.
Built automatic recording for Superlive, the company's live-class video platform, and I'm the main contact for that area.
Deployments 60% faster and incident response 40% faster through better automation and monitoring.
Experience
Learnyst · Software EngineerMay 2024 – present
Database migrations, moving services onto Kubernetes (GKE), live video with LiveKit and WebRTC, on-call for production, and now designing AI-powered course search.
EXL Service · Graduate Engineer TraineeOct 2023 – Mar 2024
Python data engineering and Jenkins deployment automation.
LCEPL · Software Development InternApr 2022 – Oct 2022
Built an automated email response engine in Flask that answered 50% faster.
How I work
Measure before deciding. The database move started with a benchmark of about 1.5 million requests.
Always keep a way back. The old database stayed untouched, so undoing the switch was one DNS change.
Write up the bugs that fail quietly. The terminal version has seven short postmortems about bugs that never threw an error.
Skills
Languages
Go, Ruby, JavaScript, Python, SQL
Backend
Ruby on Rails, Node.js (Fastify), GraphQL, REST, Sidekiq, BullMQ