In the Middle Ages, Óbuda was under the direct influence of the royal court as part of the Queen's estate. The town lost its significance during the Ottoman occupation. After the Turkish wars, the town was acquired by the Zichy family. Significant constructions were carried out in the area during the Baroque period, when the old town of Óbuda was built.
The one-storey, baroque-style, modestly ornate castle was built by Count Miklós Zichy (1709–1758) between 1746 and 1752. The building was built by stonemason Henrik Jäger and was decorated by sculptor Károly Bebó. Its northern and southern façades are symmetrical, divided into three parts, its central avant-corps is decorated with characteristic columns supporting balconies. The castle originally had a mansard roof, which was replaced by a tent roof during the 19th century. Its former garden included orange trees and a greenhouse with tropical plants. The castle had a library, a billiard room, a salon, women's and men's sleeping chambers, guest rooms and a chapel. Miklós Zichy maintained his own orchestra and organized theater performances of fashionable plays of the time. Queen Maria Theresa visited the castle twice.
Due to the extinction of the Óbuda branch of the Zichy family, the Viennese court acquired the commercially and strategically important settlement. The castle then served as a military base, and underwent large-scale reconstruction. Due to the strengthening of the industrial life of Óbuda, the building operated as a silk manufactory for some time, and was declared an emergency apartment due to the lack of housing after the Second World War.
The castle was renovated and utilized for cultural purposes in 1976, when it became a youth cultural center. Today, the Kassák Museum is housed in the castle building, while the outbuildings are home to the Óbuda Museum and the Vasarely Museum, the Térszínház (Space Theater) and the workshops of Hello Wood.
The Kassák Museum was renovated in 2010 with the support of the Stiftung Ahlers pro Arte and the Deutsch-Ungarische Gesellschaft. The ground floor of the castle was used by the Hungarian Gallup Institute from the 1990s until 2014, and has been managed by the Petőfi Literary Museum ever since. The castle building is now owned by the Magyar Nemzeti Vagyonkezelő Zrt. (Hungarian National Asset Management Inc.) and maintained by the Magyar Kultúráért Alapítvány (Foundation for Hungarian Culture).