Android Voice

Android Voice (OpenAI STT/TTS)

Discrete pipeline: record → transcribe → text, and text → synthesize → play. Each stage independently testable; keep a typed-text input path so agent/network debugging never requires a microphone.

The Realtime speech-to-speech API is the production upgrade (latency, barge-in) — don't build it in a timed interview; design the voice layer so it can slot in (interfaces below already isolate it).

Endpoints (verified Jul 2026)

Known issue: current gpt-4o-mini-tts snapshots sometimes truncate the final sentence (~5–20%, silent 200). Mitigation for short spoken summaries: end input with a final period and avoid trailing short questions; don't chase it further in a demo.

Permission (Compose)

AndroidManifest.xml: <uses-permission android:name="android.permission.RECORD_AUDIO" />

val permissionLauncher = rememberLauncherForActivityResult(
    ActivityResultContracts.RequestPermission()
) { granted -> if (granted) onMicGranted() else onMicDenied() }

// on mic button press:
when {
    context.checkSelfPermission(RECORD_AUDIO) == PERMISSION_GRANTED -> onMicGranted()
    else -> permissionLauncher.launch(RECORD_AUDIO)
}

Denied → keep the app usable: fall back to the text input path, show a short rationale. Don't loop the request dialog.

Recording — MediaRecorder → m4a

Simplest reliable capture; AAC in an .m4a container is accepted by the transcription endpoint directly. (AudioRecord is for raw PCM / streaming — only if the prompt demands it.)

class VoiceRecorder @Inject constructor(@ApplicationContext private val context: Context) {
    private var recorder: MediaRecorder? = null
    private var output: File? = null

    fun start(): Result<Unit> = try {
        val file = File(context.cacheDir, "turn_${System.currentTimeMillis()}.m4a")
        recorder = MediaRecorder(context).apply {
            setAudioSource(MediaRecorder.AudioSource.MIC)
            setOutputFormat(MediaRecorder.OutputFormat.MPEG_4)
            setAudioEncoder(MediaRecorder.AudioEncoder.AAC)
            setAudioSamplingRate(16_000) // speech; keeps upload small
            setOutputFile(file.absolutePath)
            prepare()
            start()
        }
        output = file
        Result.success(Unit)
    } catch (t: Throwable) {
        Result.failure(t.toAppError())
    }

    fun stop(): Result<File> = try {
        recorder?.apply { stop(); release() }
        recorder = null
        Result.success(requireNotNull(output))
    } catch (t: Throwable) {
        recorder?.release(); recorder = null
        Result.failure(t.toAppError())
    }
}

• MediaRecorder(context) ctor needs API 31+; use the deprecated no-arg ctor below that
• stop() throws if called immediately after start() (no audio yet) — treat as "too short, try again"
• Cache dir files: delete after successful transcription

STT — multipart upload (OkHttp)

suspend fun transcribe(file: File): Result<String> = withContext(Dispatchers.IO) {
    try {
        val body = MultipartBody.Builder().setType(MultipartBody.FORM)
            .addFormDataPart("model", "gpt-4o-mini-transcribe")
            .addFormDataPart("response_format", "text")
            .addFormDataPart("file", file.name, file.asRequestBody("audio/m4a".toMediaType()))
            .build()
        val request = Request.Builder()
            .url("https://api.openai.com/v1/audio/transcriptions")
            .header("Authorization", "Bearer ${BuildConfig.OPENAI_API_KEY}")
            .post(body)
            .build()
        client.newCall(request).execute().use { response ->
            if (!response.isSuccessful) return@withContext Result.failure(AppError.Http(response.code))
            Result.success(response.body.string().trim())
        }
    } catch (t: CancellationException) {
        throw t
    } catch (t: Throwable) {
        Result.failure(t.toAppError())
    }
}

Optional prompt form field improves domain vocabulary ("Kotlin, Gradle, Hilt, Jetpack Compose") — cheap accuracy win for coding speech.

TTS — synthesize + play

suspend fun speak(text: String): Result<Unit> = withContext(Dispatchers.IO) {
    try {
        val json = """{"model":"gpt-4o-mini-tts","voice":"alloy","input":${text.toJsonString()},"response_format":"mp3"}"""
        val request = Request.Builder()
            .url("https://api.openai.com/v1/audio/speech")
            .header("Authorization", "Bearer ${BuildConfig.OPENAI_API_KEY}")
            .post(json.toRequestBody("application/json".toMediaType()))
            .build()
        client.newCall(request).execute().use { response ->
            if (!response.isSuccessful) return@withContext Result.failure(AppError.Http(response.code))
            val file = File(context.cacheDir, "speech.mp3")
            file.outputStream().use { response.body.byteStream().copyTo(it) }
            play(file) // suspends until playback completes
        }
        Result.success(Unit)
    } catch (t: CancellationException) {
        throw t
    } catch (t: Throwable) {
        Result.failure(t.toAppError())
    }
}

private suspend fun play(file: File) = suspendCancellableCoroutine { cont ->
    val player = MediaPlayer().apply {
        setDataSource(file.absolutePath)
        setOnCompletionListener { it.release(); cont.resume(Unit) }
        setOnErrorListener { mp, _, _ -> mp.release(); cont.resume(Unit); true }
        prepare() // file is local; prepareAsync unnecessary
        start()
    }
    cont.invokeOnCancellation { player.release() } // barge-in / stop speaking
}

• Write bytes to a file then MediaPlayer — simplest reliable path; skip ExoPlayer/streaming playback unless latency polish time remains
• Cancellation of the speak coroutine stops playback → this is your "barge-in": cancel speaking when the user presses the mic
• Use Android's built-in TextToSpeech only as an offline fallback if asked; quality is noticeably worse

VoiceSession state machine

One enum drives the whole screen; every phase change is a UiState copy.

enum class VoicePhase { Idle, Listening, Transcribing, WaitingOnAgent, Speaking }

// transitions:
// Idle          --mic press-->        Listening   (recorder.start)
// Listening     --mic release-->      Transcribing (recorder.stop → transcribe)
// Transcribing  --transcript-->       WaitingOnAgent (send turn)
// WaitingOnAgent --terminal event-->  Speaking (if speak text) else Idle
// Speaking      --playback done-->    Idle
// any           --stop/cancel-->      Idle (cancel jobs; recorder/player released)

• Push-to-talk (press-and-hold or tap-to-toggle) beats always-listening for a demo: deliberate, no VAD tuning, no open-mic questions
• Speak summaries at turn boundaries, never stream tokens into TTS
• Big touch targets + high contrast if the prompt is glanceable/car-mode

Emulator

• Extended controls (⋯) → Microphone → enable "Virtual microphone uses host audio input" — the laptop mic feeds the emulator
• Audio output plays through host speakers by default
• Rehearse once: mic levels through the emulator are quieter than device mics; the prompt field on STT helps

Pitfalls

• MediaRecorder.stop() immediately after start → RuntimeException (treat as "too short")
• Forgetting Authorization header on the second OkHttp client if you build a fresh one — share the client, add the key via interceptor
• TTS input not JSON-escaped (quotes in summaries) — serialize with Moshi, don't hand-build JSON in real code
• Playing TTS while still recording (release order: recorder before player)
• Blocking the main thread with MediaPlayer.prepare() on remote sources (local file is fine)
• Testing voice before the text path works — voice bugs and agent bugs become indistinguishable

Related

• App structure / Result / AppError: ../android-interview-bootstrap/SKILL.md
• Agent loop / harness-server client: ../android-agent-harness/SKILL.md