At least five buildings in Hualien City caved in and were shown dangerously tilting on their sides in the tourist port city.
Rescuers are working to reach five people they know are trapped in buildings while more than 140 are unaccounted for.
The worst hit was a military hospital which rescuers have propped up with cranes as it leans at about 45 degrees.
A hotel employee died after the bottom three floors of the Marshal Hotel caved in while another person died in a residential building, the government disaster centre reported.
The shallow quake hit just before midnight on Tuesday night, injuring 225 people, with two dozen in a critical condition, Taiwan's Central News Agency reported.
Video footage and photos showed several large buildings leaning at sharp angles.
Their lower floors were crushed into heaps of concrete, smashed glass, bent iron beams and other debris.
Residents were seen lowering their children from balconies in slanted buildings before climbing down themselves or being helped down by firefighters.
The tremor was felt shortly before midnight on Tuesday, forcing the closure of a busy road nearby.
The epicentre was very shallow at just 10.6km deep and was the 94th tremor since a 6.1 magnitude quake struck nearby on Sunday.
Roads were seen with large cracks in them while the tremor disrupted electricity and water supplies to thousands of households, the National Fire Agency said.
The government said a bridge in the city on Taiwan's eastern coast could not be used, and the structure of an inn had tilted during the earthquake.
Chen Chih-wei, 80, said he was sleeping on the top floor of his apartment building when the quake struck.
"My bed turned completely vertical, I was sleeping and suddenly I was standing," he said.He crawled to a balcony to wait to be rescued, and said it was the strongest tremor he had felt in more than five decades of living in Hualien.
The President's office said in a statement following Tuesday's earthquake: "The President (Tsai Ing-wen) has asked the cabinet and related ministries to immediately launch the disaster mechanism and to work at the fastest rate on disaster relief work."
In a post on her official Facebook page, Ms Tsai said she had arrived in Hualien on Wednesday morning to review rescue efforts.
She said she "ordered search and rescue workers not to give up on any opportunity to save people, while keeping their own safety in mind".
"This is when the Taiwanese people show their calm, resilience and love," she wrote. "The government will work with everyone to guard their homeland."
Chen Tzai-Tung, a worker with the government disaster centre, said it was not safe for rescuers to enter the 12-storey Yunmen Cuiti apartment building where most of the missing are thought to be trapped because it was continuing to lean farther.
A maintenance worker who was rescued after being trapped in the Marshal Hotel's basement said the initial hit was not big.
"At first it wasn't that big ... we get this sort of thing all the time and it's really nothing. But then it got really terrifying," Chen Ming-hui said after he was reunited with his son and grandson.
"It was really scary."
As aftershocks came through the night 830 people went to shelters, including a newly built baseball stadium where beds and hot food were provided.
Taiwan lies along the Ring of Fire, the seismic faults surrounding the Pacific Ocean where the majority of the world's earthquakes occur..
The island's worst tremor in recent decades was a 7.6 magnitude quake in September 1999, which killed about 2,400 people.
