The current multilingual XTTS service is implemented entirely in PyTorch and has the following limitations:
High latency:
The time-to-first-byte (TTFB) is approximately 1.2 seconds on a single A100 GPU, which is suboptimal for real-time conversational use cases.
Concurrency instability:
Under concurrent request load, the existing implementation frequently triggers runtime failures, typically manifesting as CUDA kernel assertion errors (e.g., tensor dimension mismatches). This indicates insufficient concurrency safety in the current execution path.
Implemented Improvements:
To address the above issues, I implemented the following enhancements:
Re-engineered XTTS-v2 on top of vLLM with true streaming support.
Achieved a TTFB of ~0.4 seconds.
Text preprocessing enhancements
Updated preprocessing logic to correctly normalize numbers, dates/times, and special formatted tokens prior to passing text into the XTTS GPT module.
This improves pronunciation accuracy and reduces downstream inference ambiguity.
OpenAI-compatible API interface
Exposed the service through a fully OpenAI-compatible API format, enabling seamless integration with existing clients and tooling.
API Information
openai format api
The current multilingual XTTS service is implemented entirely in PyTorch and has the following limitations:
High latency:
The time-to-first-byte (TTFB) is approximately 1.2 seconds on a single A100 GPU, which is suboptimal for real-time conversational use cases.
Concurrency instability:
Under concurrent request load, the existing implementation frequently triggers runtime failures, typically manifesting as CUDA kernel assertion errors (e.g., tensor dimension mismatches). This indicates insufficient concurrency safety in the current execution path.
Implemented Improvements:
To address the above issues, I implemented the following enhancements:
Re-engineered XTTS-v2 on top of vLLM with true streaming support.
Achieved a TTFB of ~0.4 seconds.
Text preprocessing enhancements
Updated preprocessing logic to correctly normalize numbers, dates/times, and special formatted tokens prior to passing text into the XTTS GPT module.
This improves pronunciation accuracy and reduces downstream inference ambiguity.
OpenAI-compatible API interface
Exposed the service through a fully OpenAI-compatible API format, enabling seamless integration with existing clients and tooling.
API Information
openai format api