From Peter Watts’ Blindsight (p.261), on dark interspecies cryptographic game-theory:

Hurt them.
It may take a while to figure out how. Some may shrink from fire, others from fire, others from toxic gas or liquid. Some creatures may be invulnerable to blowtorches and grenades, but shriek in terror at the threat of ultrasonic sound. You have to experiment; and when you discover just the right stimulus, the optimum balance between pain and injury, you must inflict it without the remorse.
You leave them an escape hatch, of course. That’s the very point of the exercise: give one of your subjects the means to end the pain, but give the other the information required to use it. To one you might present a single shape, while showing the other a whole selection. The pain will stop when the being with the menu chooses the item its partner has seen. So let the games begin. Watch your subjects squirm. If — when — they trip the off switch, you’ll know at least some of the information they exchanged; and if you record everything that passed between the, you’ll start to get some idea of how they exchanged it.
Whe they solve one puzzle, give them a new one. Mix things up. Switch their roles. See how they do at circles versus squares. Try them out on factorials and Fibonnacis. Continue until Rosetta Stone results.
This is how you communicate wuith a fellow intelligence: You hurt it, and keep on hurting it, until you can distinguish the speech from the screams.