Speech Integration
Explore how to integrate speech capabilities in AI by managing audio input and output through explicit content type boundaries. Learn to handle audio formats, enforce payload size limits, and reduce latency via streaming techniques. Understand debugging strategies and the contract design for reliable STT and TTS communication in production.
A complete speech loop has two separate calls that pass different types through our app. The /stt call accepts audio bytes and returns a transcript string, then the /tts call accepts a string and returns audio bytes we can play in the UI. If we keep the handling ephemeral, the server reads the uploaded body, sends it to the provider, returns the result, and discards the raw audio by default.
The key is to make the boundaries explicit with content types and response types, so the client never has to guess. For STT, the request is typically multipart/form-data with a file field and the response is application/json containing text. For TTS, the request is JSON and the response is audio like audio/mpeg or audio/wav, which Streamlit can play directly from bytes.
The diagram below shows the flow of bytes and strings through /stt, /tts, and the Streamlit UI, highlighting ephemeral in-memory handling that persists nothing by default.
The /stt endpoint signature makes the input a file-like payload and the output a small JSON object, so the UI reads transcript[“text”] and never handles provider-specific fields. The /tts endpoint returns raw audio bytes with an explicit media_type, which is why st.audio(...) can render a player without decoding logic, as long as the bytes and declared format match.
Audio formats, payload size, and hard limits
Once the loop works, the first production failure usually comes from sending the wrong encoding or too much audio. A longer clip ...