Layered Service Providers in Vista/64-bit

Layered Service Providers (LSPs) are user mode dlls that allow one (most usefully) to be in network packet chain and thus can change (think encryption/compression) or inspect (think sniffing) buffers from winsock applications. They are chained along with base providers by registering in LSP System Configuration Database (aka catalog).

On 64-bit Windows, there are 2 catalogs, one for 32-bit and one for 64-bit. 32-bit LSPs generally need to be ported to 64-bit in case network buffers of 64-bit applications are of interest. One can register the 32-bit LSP on 64-bit Windows (in the 32-bit catalog of course) and get at buffers in 32-bit applications (running under wow64). But the 32-bit LSP  will not be able to get at buffers in 64-bit winsock applications. Similarly 64-bit LSP  will not be able to get at buffers in 32-bit winsock applications.

In other words for all practical purposes, both 32-bit and 64-bit LSP versions are needed for complete functionality in 64-bit Windows. Installation of 32-bit and 64-bit versions of LSP can be accomplished by WSCInstallProviderAndChains64_32. To install just the 64-bit version WSCInstallProvider can be used. There is no WSCInstallProviderAndChains  on 64-bit Windows to install 64-bit only LSP.

LSP categories were introduced in Vista and Windows Server 2008, to give order to chaotic world of layering LSPs correctly (for example LSP_CRYPTO_COMPRESS LSP  that encrypts data should be generally below LSP_INSPECTOR LSP so the inspector can look at unencrypted data and make sense out of it). Before LSP categories, programmers, in order to stay functional and free of interop issues, were left to do manual layering, whitelisting etc whenever another LSP  (presumably from a different company) gets installed in or removed from winsock catalog. Frequently winsock catalog corruption is the end result of we-should-be-above-them-but-below-these-other-guys (or oops-when-did-they-install) games, leaving the system with broken network connectivity.

In general LSPs are easier to write but hard to debug and even harder to uninstall/remove from system because they get loaded into all socket applications and reboot is often the only recourse. Buggy/malware LSPs can load into pretty much all processes (since most useful applications end up doing network operations) and cause instability or compromise security. LSP categories address system stability (and performance) issues by introducing Application categories. Here is how Vista RTM sets up Application category for system binaries.

With LSP categories and Application categories Windows can match bits and decide not to load an LSP into a system critical process like lsass.exe. When an application indicates its Application category to winsock (by calling WSCSetApplicationCategory ), only relevant subset of LSPs installed end up getting loaded into the process.

This can be a good thing or a bad thing, depending on what you are trying to do. For example, if an LSP provides security services in Vista, it needs to setup its application category correctly and be aware of how Vista and third party applications set their categories so that it is loaded where it needs to be loaded to provide security services. Also of interest would be how malware LSPs and malware applications set up categories for themselves or change categories on others to attempt to bypass or break security or detection technologies in place.