Plans for £300,000 of improvements to Silsden Town Hall have been revealed by Bradford Council.
Changes could include performance and exhibition space, and disabled access including a new entrance and a lift to the upper floor.
Silsden library would be moved back into the town hall and its current building in nearby Wesley Place put on the market.
Sale proceeds, estimated at £300,000, would fund the town hall improvements.
A Silsden community group campaigning for the changes claims it has support from other local organisations, but Silsden Town Council fears the plans could jeopardise its own work to improve community facilities in the town.
Councillors say Bradford's proposals could affect their wider strategy to develop the town hall, protect the library building and build a new community hall at Silsden Park.
The town hall plans were revealed last Thursday by Bradford asset management and libraries officers.
Silsden Community Arts and Heritage group, which organised the meeting, said there was overwhelming support from local groups and activities.
A spokesman said the radical plan would revitalise the town hall, ensure its survival and enable it to meet the needs of today's townspeople.
Supporters also believe moving the library would attract people to both the town hall and nearby shops.
The town council has now issued a statement saying it was disappointed it had not been invited to the meeting.
It said Bradford Council had recently assured councillors there were no plans to sell the town hall or move the library.
A spokesman said: “There is currently a review underway into Silsden's wider needs given the potential of an additional 17,000 houses and pressure on existing resources that will bring.
“Due to the large number of user groups requiring space in the town, the town council is looking at a number of options including lottery funding for the town hall, a new facility in the park, and a new community building on the footprint of Wesley Place in the library.”
Rita Farmer, who chaired last Thursday's meeting, told representatives there was no clash of interest between the town hall project and Silsden Town Council's proposed community hall.
Coun Andrew Mallinson told the Keighley News he supported the town hall plans, but pledged to keep a “watchful eye” on any planning applications for Wesley Place.