This book constitutes the refereed proceedings of the International Workshop MUSCLE 2011 on Computational Intelligence for Multimedia Understanding, organized by the ERCIM MUSCLE working group in Pisa, Italy on December 2011