Would you rate an audio as difficult if the speaker/speakers is/are very fast?

Yes, an audio where the speaker/speakers is/are very fast would be rated as difficult audio.