Scarry was born in Boston, Massachusetts, where his parents ran a shop. The Scarry family enjoyed a comfortable life even during the time of the Great Depression. Following high school graduation, Scarry enrolled in a business college but soon dropped out, finding it not to his liking. He then studied at the School of the Museum of Fine Arts, Boston, where he remained until being drafted into the U.S. Army during World War II. After the war, Scarry worked for the art departments of various magazines before making a career breakthrough in 1949 with Little Golden Books.