Authorities in George, a city on the southern coast of the nation, said that 21 people have been sent to different hospitals, with at least 11 of them suffering from serious injuries, and that 49 people are still stranded.
Three teams of rescuers have been operating at several locations near the fallen structure where they thought construction workers might be.
The five-story structure collapsed on Monday afternoon, local time, in the city, around 250 miles (400 km) east of Cape Town. It is thought that some people are under massive slabs of concrete that fell on them.
To remove the rubble, large cranes and other heavy lifting equipment have been brought in.
Colin Deiner, the chief director of the province Western Cape Disaster Management and Fire and Rescue Services, stated that rescuers had made contact with 11 personnel who were stuck under the debris and were hopeful that they would be rescued.