I have played the role of "build specialist" in the past, and I think it depends on what is meant on a team by "build specialist." If there is only one person on a team that can touch the build scripts, that can fix the continuous deployment system, or that can configure the continuous integration server, that is definitely a problem. It is inevitable, however, that one person acts as the "build architect." The build system should have collective ownership, but there will always be one person who owns the vision for long term improvement, who acts as the domain expert in the field of builds, and who holds everyone accountable for having ownership over the build.
If you have a team of "generalizing specialists," (http://www.agilemodeling.com/essays/generalizingSpecialists.htm), having one person on the team that specializes in builds, while everyone else "generalizes" in builds, can work well.