Genome Analysis Platform

Project Overview

An event-driven serverless system for large-scale genomic data processing

The platform separates genomic sequencing analysis into AWS Batch jobs and connects raw inputs, intermediate results, and reports through EFS and S3. Event-driven messaging and explicit retry boundaries orchestrate the long-running flow from analysis request to result generation.

Key Implementations

• Parallel analysis jobs: Run analysis as isolated AWS Batch jobs and manage container images in ECR
• Shared data paths: Separate EFS and S3 responsibilities for input data, intermediate results, and final artifacts
• I/O reliability: Investigate EFS throttling and adjust job and filesystem boundaries
• Lambda execution boundaries: Keep long-running analysis in Batch while Lambda handles triggers, state transitions, and report orchestration
• Message-driven reports: Separate analysis completion from report generation with EventBridge, SNS, and SQS
• Operational visibility: Observe job state, failures, and retries through CloudWatch

Technical Architecture

1. Compute & Storage

• AWS Batch + EFS: Container-based analysis jobs and shared intermediate results
• Lambda: Request triggers, state transitions, and report-generation orchestration
• S3 · EFS: Purpose-specific storage for raw data, intermediate results, and final artifacts

2. Event-Driven Pipeline

• EventBridge + SNS/SQS: Event triggers and decoupled processing stages
• AppSync · Cognito: APIs, authentication, and authorization
• CloudWatch · ECR: Monitoring and container-image management

Key Technologies

NextJS, NestJS, FastAPI, TypeScript, Python, AWS Batch, Lambda, Step Functions, AppSync, EventBridge, SQS, SNS, EFS, S3, Cognito, CloudWatch, ECR, CloudFormation

Developed with NestJS, AWS Batch, and an event-driven serverless architecture.