No, you don't have to be connected.
The JSF taglib URI is simply what it is: Universal Resource Identifier.
If your IDE recognizes JSF META-INF descriptors properly, then you should be able to have some code completion support when developing.
Otherwise, those references (to taglib URIs and corresponding namespaces) are only resolved at runtime.
And even then, the META-INF descriptors' config will be used to resolve those URIs (that's supported by all application servers), you don't have to be online while developing/debugging the app.