The ship was carrying 48 passengers and five crew members when it capsized in the UNESCO World Heritage-listed bay due to sudden heavy rain, the Dan Tri website reported.
Most of the passengers were families traveling from the capital Hanoi, and more than 20 children were among the passengers, according to the news website VNExpress.
Border guards rescued 12 people and recovered 34 bodies, reports"VNExpress".
Rescue operations are expected to continue at night to find the missing people.
The storm began after three days of heatwaves, when temperatures reached 37 degrees Celsius in some areas.
Ha Long Bay is one of Vietnam's most popular tourist destinations, with millions of people visiting its blue-green waters and rainforest-covered limestone islands every year.