Deployment & Production Considerations

Deploying Models in Production

Model Export Formats

PyTorch → ONNX → Works everywhere
TensorFlow → SavedModel → TensorFlow Serving
Keras → .h5 or SavedModel → Any framework

Quantization (Make Models Smaller)

Problem: Neural network weights are 32-bit floats (4 bytes each).

Solution: Use 8-bit integers (1 byte)!

Original: 100M parameters × 4 bytes = 400 MB
Quantized: 100M parameters × 1 byte = 100 MB  (4x smaller!)

Speed: 2-4x faster on mobile!
Accuracy: Usually only 0.5-1% drop!

Batch Inference

Single prediction:

Input: One image → Model → Output
Latency: 100ms (slow)

Batch prediction:

Input: 32 images → Model → 32 outputs
Latency: 150ms (only 50% slower!)
Throughput: 32/150ms = 213 images/sec

Huge efficiency gain!

Monitoring in Production

Track:
- Accuracy on real data
- Inference latency
- Memory usage
- Error rates

Alert if:
- Accuracy drops (model drift)
- Latency spikes (resource issue)
- Error rate increases

Data Drift

Problem: Production data different from training!

Training: 2020 data
Production 2024: Different distribution!

Model's accuracy degrades over time.

Solution: Retrain periodically on new data

Deployment Platforms

Production Checklist

✓ Model accuracy validated (>95%?)
✓ Tested on diverse data (edge cases?)
✓ Latency acceptable (<100ms?)
✓ Memory footprint reasonable (<100MB?)
✓ Quantized for mobile (if needed)
✓ Error handling implemented
✓ Monitoring set up
✓ Retraining pipeline ready
✓ Documentation complete
✓ Ethics/bias reviewed

Common Pitfalls

Ethics & Fairness

Before deployment, ask:

• ✓ Does model work equally for all groups?
• ✓ Are predictions explainable?
• ✓ Are there unintended biases?
• ✓ Is data used ethically?
• ✓ Can users understand why they were rejected/approved?