In a statement Tata Sons, the holding company of the massive $100 billion TataGroup, said Monday its board had voted to replace Mistry, four years after he became its first chief from outside the immediate Tata family.
It added that 78-year-old Ratan Tata, one of India's most famous industrialists, would return to the helm of the company until a successor is found, which would likely to take four months.
Shortly after the Bombay Stock Exchange opened Tata Steel sank 2.89 percent, car manufacturing giant Tata Motors fell 1.17 percent and IT giant TataConsultancy Services was down 0.75 percent.
The announcement came as Tata Steel struggles to offload its loss-making British assets while Tata Motors struggles with weak sales.
Mistry had been unable to resolve a long-running dispute with NTT Docomo, which is demanding the giant coughs up a $1.17 billion arbitration payment awarded to the Japanese mobile phone operator at an international hearing. The issue threatens to damage its international reputation.
Mistry was declared heir to Tata a year before he took over the top position in December 2012. Analysts predicted that the abruptness of Monday's decision, uncharacteristic for the company, would impact on Tuesday's trade.
Tata, who took over as chairman of Tata Sons in 1991 and led the company for 21 years, is credited with building it into a global behemoth.
During his time at the helm, the organisation went on a global purchasing spree, acquiring major names ranging from Tetley Tea to Land Rover and the Anglo-Dutch steel firm Corus in 2007 for $13.7 billion.
However, Mistry's time in charge has been rocky and the group, founded under British colonial rule in 1868, has hit headwinds with lacklustre performances at several companies including Tata Motors, Tata Power and Tata Steel.
Profits at India's IT outsourcing companies are also being squeezed as the sluggish global economy sees clients rein in their spending.
The Tata Group spans at least 100 companies in as many countries. Its brands feature daily in the lives of Indians, with products ranging from salt to watches.
