Software Engineer LLM Performance and Evaluation

You will build and maintain LLM benchmarks, evaluation pipelines, dashboards, and reports. You will measure latency, throughput, memory use, utilization, and model accuracy; automate reproducible evaluations; investigate regressions; and communicate performance and accuracy trade-offs.

Responsibilities

• Design and maintain LLM inference benchmarks
• Measure latency, throughput, memory use, and accelerator utilization
• Evaluate model accuracy and compare results with reference implementations
• Build automated and reproducible benchmark and accuracy evaluation pipelines
• Maintain performance and accuracy dashboards and reports
• Define regression thresholds and release acceptance criteria
• Investigate regressions through profiling and controlled experiments
• Improve measurement quality through statistical analysis and validation
• Communicate performance and accuracy findings and recommendations

Requirements

• Python
• Automation
• Data pipeline
• Performance analysis
• Profiling
• Latency analysis
• Throughput analysis
• Transformer
• LLM inference
• PyTorch
• Hugging Face Transformers
• Experimental design
• Data analysis
• Reproducibility
• vLLM
• SGLang
• TensorRT-LLM
• lm-evaluation-harness
• Quantization
• Mixed precision
• Speculative decoding
• Distributed inference
• GPU
• NPU
• TPU
• Linux
• Container
• Cluster
• CI
• Rust
• C++