ML System Design interview questions

How would you design an offline metric for text generation quality?

Describe the complete ML pipeline from data collection to inference: what stages and tools did you use?

Do you have experience optimizing inference with TensorRT, quantization, or similar approaches?

What tools do you use for experiment tracking and model versioning?

Do you have an on-call rotation? How is service monitoring organized?

How does the IVF (Inverted File Index) work in FAISS?

Model training takes 12 hours. How would you identify the bottleneck and speed it up?

How would you keep ML model inference latency below 100 ms?

Describe your experience with MLOps and model deployment.

How did you handle the embedding dimensionality limit of 256 in OpenSearch?

Have you written production code? How automated was the deployment process?

What tests did you write for the ML model or assistant, and how did you evaluate quality?

Will precomputed scores be stored in a cache, or will inference run at request time?

Is the solution you are describing already complex, or is it still a simple one?

Estimate how long this solution would take to implement with a team of three: two ML engineers and one backend engineer.

What problems do you think we need to anticipate at the architecture level?