You can invoke Windbg two ways. One way is from the Windows Start menu:

    Start | All Programs | Debugging Tools for Windows | Windbg

The other is from the DOS command prompt:

    C:\ > windbg

In Windbg, use the File pulldown menu to select Open Crash Dump, specifying the location of the dumpfile. This can be accomplished in one step from the command prompt by using the -z option:

    C:\> windbg -z memory.dmp

Be sure to watch out for any warnings from Windbg indicating a truncated or inconsistent set-bit count. Messages like this may indicate the dumpfile is corrupt or missing data:

********************************************************************************
WARNING: Dump file has been truncated. Data may be missing.
********************************************************************************
WARNING: Dump file has inconsistent set-bit count. Data may be missing.
********************************************************************************

Windbg does a good job of pointing out problems with asterisks (*), so be sure to pay particular attention whenever you see them in the output. By default, the debugger output is displayed in the main window with a one-line command prompt at the bottom.

No matter what sort of hang your server has encountered, the first command that should be used in Windbg is this:

    !analyze -v -hang

The !analyze command will perform a preliminary analysis of the dump and provide a "best guess" for what caused the crash. In the case of a forced dump, the analysis will typically point to the i8042prt.sys or kbdhid.sys driver because that is the driver that initiated the crash. You will also notice the bugcheck type is a 0xE2, indicating a manually initiated crash as seen in Figure 1.

Figure 1.

In addition to providing a best guess for the cause of the crash, the !analyze command will also check for blocking locks and set the processor, process, thread and register context to the current ones at the time of the crash. Subsequent commands will use this context for their execution.

Once you have executed the !analyze command, the commands in Table 1 will help determine the footprint or circumstances that existed when the crash was forced. Be sure to focus on the current process, current thread, stack trace, virtual and physical memory usage, and locking information. We will take a closer look at these commands in subsequent sections.

Windbg commands for analyzing server hangs.

!process - Display current process information
!thread - Display current thread information
!running -it - Display currently executing threads on all CPUs
!vm - Display virtual memory usage
!poolused - Display paged and non-paged pool usage
!memusage - Display physical memory usage
!locks - Display kernel locks held
!stacks - Display summary of threads, states and function
kv - Display current threads stack trace

High-priority compute-bound threads

Identifying the current process (!process) and the current thread (!thread) can prove useful if the server hung because of a high-priority runaway, compute-bound thread. Use the !running -it command, as it will list all the currently executing threads across all the processors. Processes and threads can be assigned various levels of priorities that can preempt other processes and threads.

System resource depletion

If you suspect a system resource depletion caused the hang, use the !vm, !poolused and !memusage commands. These commands display the virtual and physical memory usage at the time of the hang. Be sure to watch for any asterisks flagged by Windbg as illustrated in Figure 2.

Figure 2.

To determine if paged pool or non-paged pool has been depleted, compare the "usage" to the "maximum" value as circled in red above. If the usage is relatively close to the maximum value, then there is a high likelihood that pool depletion caused the hang. You would then use the !poolused command to focus in on which pool data structure was responsible. The !poolused command has several flags to sort the paged or non-paged data structures according to their usage (see the online debugger help for more information on the command syntax and usage).

It is worth mentioning that pool statistics can also be acquired by several tools without the need for a memory dump. You can use Perfmon to collect general paged and non-paged performance statistics. Poolmon and Poolsnap are free tools from Microsoft that capture more granular specifics on the actual pool data structures. Finally, note that it is possible to tune paged pool on x86 servers by tweaking two registry values (PagedPoolSize and PoolUsageMaximum). For further details on tuning paged pool, check out Microsoft KB article 312362.

Deadlock and spinlock hangs

Use the !locks command if you suspect a deadlock hang. As explained earlier, a deadlock exists when one thread owns an exclusive lock on a resource that another thread wants, and that thread exclusively owns a resource that the initial thread wants. There are several variants of a deadlock scenario, but there must be waiter threads that stall as a result. In Figure 3, you can see a potential deadlock scenario where we have an exclusively owned lock on a resource that has numerous waiters.

You will notice under the list of threads for the resource that one has an asterisk next to it. This thread is the one that owns the exclusive lock for the resource. So, the question to be answered is, what is causing the owning thread to stall and not release the lock for the other waiters to acquire? Therefore, the next command to issue would be a !thread command on the owning thread to determine why it is stalled.

Figure 3

Figure 4 shows the output of a !thread command on the owner. It reveals that it is stalled waiting for an I/O request packet (IRP) to complete from the QAFilter.sys driver. This particular case is a known issue caused by a deadlock with the QAFilter driver documented in Microsoft KB article 906194. Note that QAFilter and NmSvFsf are not standard Microsoft drivers, so symbols are not available for them from the Microsoft symbol server.

Figure 4

A spinlock hang is very similar to a deadlock condition except that processors are involved instead of threads. A data structure called a spinlock is used to synchronize access to other data structures or a critical section. Only one processor can own a particular spinlock at a time. The other processors that want to acquire the spinlock will wait (or spin) until the spinlock is released. In a spinlock scenario, multiple processors all want to acquire the same spinlock at an elevated IRQL, causing a perceived system hang.

To troubleshoot a spinlock hang, examine each processor to determine what function is executing at the time. Use the ~# command -- where # is the processor number (0, 1, 2 …) -- to change context between processors. You will notice that the debugger's kd prompt changes to reflect the processor number that currently has context.

Then use the !thread or kv command to determine the stack trace of the current thread to see what function was executing. In a true spinlock scenario, all processors except one will be executing a spinlock acquire function. Finally, to determine the culprit (driver) responsible for the spinlock condition, look down the stack trace for the last driver to call the spinlock acquire function. See Figure 5 for an example of a stack trace illustrating a spinlock hang initiated by the XYZDrv.sys driver.

Figure 5

Finally, the command !stacks is very useful to determine which threads are executing and the states of those threads (running, ready, blocked, etc.). In the example of the spinlock hang, !stacks was extremely useful in illustrating how threads currently running on the various processors were all trying to acquire spinlocks except for the current thread that was executing the bugcheck code. Figure 6 shows an example of the !stacks command and the pertinent output.

Figure 6

And there you have it. Troubleshooting non-responsive Windows servers can be very perplexing. Fortunately, the Windows operating system has matured over the years and now offers a variety of features and tools to help determine what causes servers to hang. By forcing a crash dump and using Windbg to analyze it, you can typically isolate the hang to a particular application or system resource. Plus, if the problem requires further analysis from Microsoft, you will have the memory dump they will need to troubleshoot the issue.

When troubleshooting a hung Windows server, there are several things that need to be done up front to prepare for collecting data. A forced crash dump may only be necessary if other means of troubleshooting prove unsuccessful. The first thing administrators should always do is run MPS Reports to collect event logs and other pertinent information. Close examination of system and application event logs may reveal a pattern of particular entries occurring prior to each hang. If the problem starts with a slow down or performance issue, you should collect Perfmon data as described in Microsoft Knowledge Base article 248345.

Once you determine that a forced crash dump is necessary, update the appropriate registry entries per KB article 244139 or 927069 and reboot the server. Also, ensure you have properly configured the dump file type as previously mentioned in KB article 254649. Finally, be sure that your pagefile.sys is sufficiently sized to accommodate a memory dump and that you have enough free space on the disk where the memory.dmp will be located, per KB article 886429.

Installing Windbg and setting the symbol path

In addition to configuring the server to generate a memory dump, you have to install the Windows Kernel Debugger and establish the symbol path. Do that on a separate server from the one that you are troubleshooting. You can download the Windbg kit free from Microsoft, and the kind of kit you choose depends on the architecture you are installing it on: (x86 or x64/IA64). Each is capable of reading a dump from a different architecture (i.e., 32-bit Windbg can read a 64-bit dump and vice versa).

Once Windbg is installed, be sure to establish the symbol path as documented in KB article 311503. Setting up the symbol path allows the debugger to translate memory references to meaningful functions and variable names. This will allow you to look at a stack trace and determine what routines were executing at the time of the hang.

Once you have all this set up, you are ready to analyze a crash dump. Use the appropriate keystrokes, Web GUI, Management Processor TC command or NMI button to initiate the forced crash as previously described. Be sure to allow sufficient time for the contents of memory to write to the pagefile.sys. If you have trouble getting the crash dump created, be aware that there are several reasons why a crash dump may not be captured as expected (see KB article 130536).

Now you are ready to determine the cause of the server hang. In the final part of this series, I'll explain how to use Windbg to analyze a forced crash as a means of resolving the problem.

When Microsoft released the early versions of its server operating system (Windows NT 3.5x and NT4), there was no easy way to troubleshoot a hung server. Other mainstream operating systems, such as Digital Equipment Corp.'s VAX/VMS, offered ways to manually intervene by forcing a crash dump whereby the server's state could be captured at the time of the hang. This dump could then be analyzed to determine why the server hung. The only option for early Windows platforms, however, was to reset the box.

As Windows servers became more predominant in the business world, hitting the reset button became unacceptable. As a result, in Windows 2000 Server and later versions, it became possible to force a crash dump to assist with determining why the server hung. Microsoft introduced this feature in Knowledge Base article 244139. It allows a keystroke combination (right CTRL+SCROLL LOCK twice) to generate a crash dump on PS/2-type keyboards. Microsoft extended this feature in Windows Server 2003 with a hotfix to the Kbdhid.sys driver to accommodate USB-type keyboards.

Several other options now exist to force a crash dump. Microsoft provides the Windows Special Administrative Console (SAC) Crashdump command as part of Windows Emergency Management Services (EMS), which allows for "headless" servers with no local graphical console. Vendor-specific options also exist to force a crash dump including the HP Integrity server's Management Processor TC (transfer of control) command, an NMI (non-maskable interrupt) button on some Integrity models, or the Integrated Lights Out (iLO) virtual NMI button. We'll take a closer look at each of these options later in the series.

Why a server hangs

There are a variety of reasons why a server may hang, including both hardware and software issues. The most common hardware reason for a server hang is spurious interrupts by a failing device. For example, a network interface controller (NIC) may have a bad component or be attached to a bad cable causing false interrupts to occur. These interrupts occur at an elevated interrupt request level (IRQL) dominating the attention of the processor(s), leaving lower priority requests (user level) unanswered. As a result, the server appears to be hung.

Another example of a hardware-induced hang involves storage requests going unanswered. For example, consider a case where a disk drive fails, causing outstanding I/O requests to be queued up. Eventually, these pending requests trigger a cascading effect of user and system threads to hang, leading to a system-wide outage.

More often, however, server hangs are a result of software issues. These issues come in several flavors, including:

System resource depletion (e.g., out of memory pool) -- The most common type of software hang, this typically is the result of a memory leak by a driver or kernel mode thread. Resource depletion can also result from exceeding architectural limits of paged and nonpaged memory pools (typically experienced on an x86 32-bit operating system).

Deadlock conditions -- A deadlock occurs when contention exists for common resources between two or more threads. For example, a deadlock exists when one thread owns an exclusive lock on a resource that another thread wants, and that thread exclusively owns a resource that the initial thread wants.

Spinlock conditions -- Spinlock hangs are similar to deadlocks, but involve contention for a spinlock that is used to synchronize access to data structures in a multi-processor environment. Other permutations of these conditions include a driver holding a lock while performing other activities for an extended period of time. Actual examples of deadlock and spinlock hangs will be provided later.

High-priority, compute-bound threads -- A software hang can also occur if high-priority, compute-bound thread(s) are dominating the processors. Since the Windows operating system permits varying levels of thread priority, one or more threads may execute at a higher priority than typical user threads. The result is that applications and users at normal priority are starved for CPU time, causing a perceived software hang.

The big picture

So, as you can see, there are numerous reasons why a server may hang. To give you a better idea of what happens when you force a crash to generate a memory dump, and subsequently analyze the crash to determine what caused the hang, see Figure 1 below.

Starting on the left-hand side, you can see the server crashes or hangs. In the event of a crash, the server would generate a memory dump if the dumpfile and pagefile are properly configured (see Microsoft Knowledge Base articles 254649, 197379 and 889654).

In the event of a hang, manual intervention would be required to force a crash dump as previously described. In either case, the content of memory is written to the pagefile.sys before the server is rebooted. During the reboot, the pagefile.sys is written to the memory.dmp file. Finally, once the server has rebooted, you can use the Windows Kernel Debugger (Windbg) to analyze the memory dump using a symbol server (as documented in KB article 311503) to translate memory references to meaningful functions and variables.

Figure 1: Overview of memory dump process and analysis

Now that you have a better idea of why server hangs occur, the next article in this series will look at the preparation process for troubleshooting a hung Windows server.

One of the first things you should reach for when troubleshooting a Windows XP boot problem is a Windows startup disk. This floppy disk can come in handy if the problem is being caused when either the startup record for the active partition or the files that the operating system uses to start Windows have become corrupted.
To create a Windows startup disk, insert a floppy disk into the drive of a similarly configured, working Windows XP system, launch My Computer, right-click the floppy disk icon, and select the Format command from the context menu. When you see the Format dialog box, leave all the default settings as they are and click the Start button. Once the format operation is complete, close the Format dialog box to return to My Computer, double-click the drive C icon to access the root directory, and copy the following three files to the floppy disk:

• Boot.ini
• NTLDR
• Ntdetect.com

After you create the Windows startup disk, insert it into the floppy drive on the afflicted system and press [Ctrl][Alt][Delete] to reboot the computer. When you boot from the Windows startup disk, the computer will bypass the active partition and boot files on the hard disk and attempt to start Windows XP normally.

2. Use Last Known Good Configuration

You can also try to boot the operating system with the Last Known Good Configuration feature. This feature will allow you to undo any changes that caused problems in the CurrentControlSet registry key, which defines hardware and driver settings. The Last Known Good Configuration feature replaces the contents of the CurrentControlSet registry key with a backup copy that was last used to successfully start up the operating system.

To use the Last Known Good Configuration feature, first restart the computer by pressing [Ctrl][Alt][Delete]. When you see the message Please select the operating system to start or hear the single beep, press [F8] to display the Windows Advanced Options menu. Select the Last Known Good Configuration item from the menu and press [Enter].

Keep in mind that you get only one shot with the Last Known Good Configuration feature. In other words, if it fails to revive your Windows XP on the first attempt, the backup copy is also corrupt.

3. Use System Restore

Another tool that might be helpful when Windows XP won't boot is System Restore. System Restore runs in the background as a service and continually monitors system-critical components for changes. When it detects an impending change, System Restore immediately makes backup copies, called restore points, of these critical components before the change occurs. In addition, System Restore is configured by default to create restore points every 24 hours.

To use System Restore, first restart the computer by pressing [Ctrl][Alt][Delete]. When you see the message Please select the operating system to start or hear the single beep, press [F8] to display the Windows Advanced Options menu. Now, select the Safe Mode item from the menu and press [Enter].

Once Windows XP boots into Safe mode, click the Start button, access the All Programs | Accessories | System Tools menu, and select System Restore. Because you're running in Safe mode, the only option on the opening screen of the System Restore wizard is Restore My Computer To An Earlier Time, and it's selected by default, so just click Next. Then, follow along with the wizard to select a restore point and begin the restoration procedure.

4. Use Recovery Console

When a Windows XP boot problem is severe, you'll need to use a more drastic approach. The Windows XP CD is bootable and will provide you with access to a tool called Recovery Console.

To boot from the Windows XP CD, insert it into the CD-ROM drive on the problem system and press [Ctrl][Alt][Delete] to reboot the computer. Once the system begins booting from the CD, simply follow the prompts that will allow the loading of the basic files needed to run Setup. When you see the Welcome To Setup screen, shown in Figure A, press R to start the Recovery Console.

Figure A

You'll then see a Recovery Console menu, like the one shown in Figure B. It displays the folder containing the operating system's files and prompts you to choose the operating system you want to log on to. Just press the menu number on the keyboard, and you'll be prompted to enter the Administrator's password. You'll then find yourself at the main Recovery Console prompt.

Figure B

5. Fix a corrupt Boot.ini

As the Windows XP operating system begins to load, the Ntldr program refers to the Boot.ini file to determine where the operating system files reside and which options to enable as the operating system continues to load. So if there's a problem rooted in the Boot.ini file, it can render Windows XP incapable of booting correctly.
If you suspect that Windows XP won't boot because Boot.ini has been corrupted, you can use the special Recovery Console version of the Bootcfg tool to fix it. Of course, you must first boot the system with the Windows XP CD and access the Recovery Console as described in #4.

To use the Bootcfg tool, from the Recovery Console command prompt, type Bootcfg /parameter
Where /parameter is one of the required parameters listed in the table below.

• /Add -Scans the disk for all Windows installations and allows you to add any new ones to the Boot.ini file.
• /Scan - Scans the disk for all Windows installations.
• /List - Lists each entry in the Boot.ini file.
• /Default - Sets the default operating system as the main boot entry.
• /Rebuild - Completely re-creates the Boot.ini file. The user must confirm each step.
• /Redirect - Allows the boot operation to be redirected to a specific port when using the Headless Administration feature. The Redirect parameter takes two parameters of its own: [Port Baudrate ] | UseBiosSettings].
• /Disableredirect - Disables the redirection.

6. Fix a corrupt partition boot sector

The partition boot sector is a small section of the hard disk partition that contains information about the operating system's file system (NTFS or FAT32), as well as a very small machine language program that is crucial in assisting the operating system as it loads.

If you suspect that Windows XP won't boot because the partition boot sector has been corrupted, you can use a special Recovery Console tool called Fixboot to fix it. Start by booting the system with the Windows XP CD and accessing the Recovery Console as described in #4.

To use the Fixboot tool, from the Recovery Console command prompt, type Fixboot [drive]: where [drive] is the letter of the drive to which you want to write a new partition boot sector.

7. Fix a corrupt master boot record

The master boot record occupies the first sector on the hard disk and is responsible for initiating the Windows boot procedure. The master boot record contains the partition table for the disk as well as a small program called the master boot code, which is responsible for locating the active, or bootable, partition, in the partition table. Once this occurs, the partition boot sector takes over and begins loading Windows. If the master boot record is corrupt, the partition boot sector can't do its job and Windows won't boot.

If you suspect Windows XP won't boot because the master boot record has been corrupted, you can use the Recovery Console tool Fixmbr to fix it. First, boot the system with the Windows XP CD and access the Recovery Console as described in #4.

To use the Fixmbr tool, from the Recovery Console command prompt, type Fixmbr [device_name] where [device_name] is the device pathname of the drive to which you want to write a new master boot record.

For example, the device pathname format for a standard bootable drive C configuration would look like this: \Device\HardDisk0

8. Disable automatic restart

When Windows XP encounters a fatal error, the default setting for handling such an error is to automatically reboot the system. If the error occurs while Windows XP is booting, the operating system will become stuck in a reboot cycle - rebooting over and over instead of starting up normally. In that case, you'll need to disable the option for automatically restarting on system failure.

When Windows XP begins to boot up and you see the message Please select the operating system to start or hear the single beep, press [F8] to display the Windows Advanced Options Menu. Then, select the Disable The Automatic Restart On System Failure item and press [Enter]. Now, Windows XP will hang up when it encounters the error and with any luck, it will display a stop message you can use to diagnose the problem.

9. Restore from a backup

If you can't seem to repair a Windows XP system that won't boot and you have a recent backup, you can restore the system from the backup media. The method you use to restore the system will depend on what backup utility you used, so you'll need to follow the utility's instructions on how to perform a restore operation.

10. Perform an in-place upgrade

If you can't repair a Windows XP system that won't boot and you don't have a recent backup, you can perform an in-place upgrade. Doing so reinstalls the operating system into the same folder, just as if you were upgrading from one version of Windows to another. An in-place upgrade will usually solve most, if not all, Windows boot problems.

Performing a Windows XP in-place upgrade is pretty straightforward. To begin, insert the Windows XP CD into the drive, restart your system, and boot from the CD. Once the initial preparation is complete, you'll see the Windows XP Setup screen (shown earlier in Figure A). Press [Enter] to launch the Windows XP Setup procedure. In a moment, you'll see the License Agreement page and will need to press [F8] to acknowledge that you agree. Setup will then search the hard disk looking for a previous installation of Windows XP. When it finds the previous installation, you'll see a second Windows XP Setup screen, as shown in Figure C on the next page.

This screen will prompt you to press R to repair the selected installation or to press [Esc] to install a fresh copy of Windows XP. In this case, initiating a repair operation is synonymous with performing an in-place upgrade, so you'll need to press R. When you do so, Setup will examine the disk drives in the system. It will then begin performing the in-place upgrade.

Keep in mind that after you perform an in-place upgrade or repair installation, you must reinstall all updates to Windows.

Figure C

Many Active Directory (AD) tools can be used at various levels to gather information about the current state. CSVDE is a favorite of mine because it is relatively quick and can be iterated through a scheduled task and the output is workable in Excel. I export certain organizational units (OUs) monthly as a record of the membership of user accounts. With this record, I can see what has changed in diagnosing issues that initially may not make much sense. This is especially helpful with a large AD environment.

#2: Copy for the quick test

I have had great success copying users to temporary user accounts to test permissions within Active Directory. This makes one key assumption that most, if not all, permissions are assigned via group membership. When group membership is the main mechanism for deploying access, copying users to test access in roles like service accounts, task accounts, and other restrictive operations, the copied user account is a quick and easy way to test without affecting the user in question. Be sure to perform the requisite housecleaning and remove the copied accounts.

#3: Have a copy of the domain available for testing

Core changes to an Active Directory installation are difficult to test and simulate. A development domain is good, but usually not configured exactly as in use on the live domain. Having a domain for testing that is exactly like your live domain makes testing schema extensions, Group Policy changes, and new security polices a breeze.

There are two principle ways to get this test environment created. The first is to create a new domain controller in the domain, then move it to a test network. Once in the test network, remove the domain controller from the live domain. When it is in the test network, the other domain controllers won't be available, but if it has all of the requisite roles, it can process logon requests and be the test environment for the changes or policies in question. The other strategy is to use a system conversion tool, such as Symantec BackupExec System Recovery, Symantec Ghost, Acronis True Image, VMware Converter, or PlateSpin PowerConvert, to take a snapshot of the domain controller in varyious ways and transport it to a physical or virtual test environment.

#4: LDIFDE for the whole thing!

The LDIFDE export tool can be helpful to move the entire domain out and have it available for importing. I would not recommend this as a backup and restore mechanism. But to create an exact replica of the live domain as described above, the LDIFDE tool can be the vehicle to export your domain to the test environment and keep it up to date. My issue with test domains is that they stray from the live environment, and keeping them current is important. You can export your entire domain as is with this easy one-liner:

LDIFDE -f C:domain-out.file

LDIFDE can interpret this file in an import, and it's readable in a text editor. It's easy to blur the differences between LDIFDE and CSVDE when you read their descriptions, but I like CSVDE because you can export by a particular organizational unit (OU). This is handy, as LDIFDE will take the entire directory, which includes user accounts as well as printers, computer accounts, domain controllers, and other Active Directory objects. LDIFDE will tend to have a larger export file because of its scope.

#5: Save queries in Active Directory

Don't we all breeze right through this first option of Active Directory Users And Computers? Having a saved query can help administrators repeat mundane tasks and easily detect policy violations. I frequently run queries for disabled user accounts that have not logged in within 60 days.

An AD query result set is a list of objects that meet the selected criteria. With this set, you can perform large scale account operations, such as deleting accounts, adding a group, moving to an account, and enabling or disabling an account. You can also perform mass operations on Exchange accounts from the results of a query.

#6: Use DSGET for AD object details

While CSVDE and LDIFDE are good for large collections of data, the DSGET command is the detail-oriented alternative. DSGET is the object tool for the Active Directory service command series, including DSADD, DSQUERY, DSMOVE, DSRM, and DSMOD. DSGET fits well in the space of documenting and benchmarking your AD installation because you can get information specifically for objects within the domain. Each object type in the directory is available to run from DSGET. You may find yourself wanting to use DSQUERY in conjunction with DSGET to save the hassle of working with the directory distinguished names.

#7: Export Group Policy objects

Managing Group Policy objects in AD is a challenging feat as well. How difficult is it to determine an issue with a complicated Group Policy? Exporting the Group Policy is a way to benchmark the configuration from a point in time. The Windows Resource Kit tool ADMX.EXE allows for an export of Group Policy objects from AD for archival and comparison purposes.

#8: Export your AD-integrated DNS zone

If your IP addressing is managed or tracked within Active Directory, you can export the zone that contains your domain systems. This will enable you to see how the addresses are used and where your domain systems are addressed across all networks in the domain. The DNSCMD command is the best utility to perform this export. The command to export a DNS zone for the sample WS2K3DEV.LOCAL zone from the DC001 server would be:

DNSCMD DC001 /zoneprint WS2K3DEV.LOCAL

You can optionally direct the command to a file for the archival. While you can also use DNSCMD for importing and modifications, the output functionality is very useful in the course of benchmarking the AD environment. The relevant output from this command is about the third line from the bottom. The output for individual systems and their addressing (in the form of DNS A records) is shown below:

DC001 [Aging:3569020] 3600 A  192.168.1.100

#9: Document with ADFind.exe

ADFind.exe provides a great way to take a quick snapshot outside Active Directory Users And Computers and outside of normal administrative rights situations. ADFind does not require special domain privileges or permissions through the Delegation Of Control wizard. So you can comfortably have your AD environment documented by computer operators, temporary employees, junior administrators, or anyone else whom you are not 100 percent comfortable giving additional rights.

Using ADFind is a little different than using the normal tools, as it is not a Microsoft tool. But a quick jog through the usage section of the Joeware Web site will have you making queries in no time at all. Here is an example I performed on a test domain (WS2K3DEV.LOCAL):

adfind -b dc=WS2K3DEV,DC=LOCAL -f "objectcategory=computer"

All computer accounts are returned, and they have a format like the following sample result:

dn:CN=VM-SERVER1,OU=VServers,DC=WS2K3DEV,DC=LOCAL

>objectClass: top

>objectClass: person

>objectClass: organizationalPerson

>objectClass: user

>objectClass: computer

>cn: VM-SERVER1

>distinguishedName: CN=VM-SERVER1,OU=VServers,DC=WS2K3DEV,DC=LOCAL

>instanceType: 4

>whenCreated: 20071109010719.0Z

>whenChanged: 20071109010838.0Z

>displayName: VM-SERVER1$

>uSNCreated: 98317

>uSNChanged: 98336

>name: VM-SERVER1

>objectGUID: {305864AA-98F3-4F0C-A813-5832F73F7BD1}

>userAccountControl: 4096

>badPwdCount: 0

>codePage: 0

>countryCode: 0

>badPasswordTime: 0

>lastLogoff: 0

>lastLogon: 128390526426562500

>localPolicyFlags: 0

>pwdLastSet: 128390440401406250

>primaryGroupID: 515

>objectSid: S-1-5-21-1529256218-1546654017-687563949-1123

>accountExpires: 9223372036854775807

>logonCount: 5

>sAMAccountName: VM-SERVER1$

>sAMAccountType: 805306369

>operatingSystem: Windows Server 2003

>operatingSystemVersion: 5.2 (3790)

>operatingSystemServicePack: Service Pack 2

>dNSHostName: VM-SERVER1.WS2K3DEV.LOCAL

>servicePrincipalName: HOST/VM-SERVER1

>servicePrincipalName: HOST/VM-SERVER1.WS2K3DEV.LOCAL

>objectCategory: CN=Computer,CN=Schema,CN=Configuration,DC=WS2K3DEV,DC=LOCAL

>isCriticalSystemObject: FALSE

>dSCorePropagationData: 20071109010838.0Z

>dSCorePropagationData: 20071109010838.0Z

>dSCorePropagationData: 20071109010838.0Z

>dSCorePropagationData: 16010108151056.0Z

As with any good tool or procedure, I recommend learning the ropes in a test environment. While it is generally a query and lookup tool, you want to be sure of any load placed on your domain controllers for big queries or exports. Using these tools in a test environment first can ensure no surprises while running in the live domain.

#10: Minimize security group sprawl

We all agree that assigning permissions via group membership is the best practice for most situations. However, having too many groups in your AD environment poses a management challenge of its own. I have found it useful to determine which groups have either no members or very few members and to consider removal or consolidation. I generally do this with the CSVDE command within the organizational unit that contains the groups in question for a quick view of the membership inventory. In this fashion, the lesser groups will pave the way for simpler administration.

At the very least, this leaves IIS 5.x users vulnerable to data interception. And while the exploit hasn't been used to take over systems to date, that could well change, according to Swa Frantzen of the Internet Storm Center.

The ability to execute code is "unexplored, but hinted about," Frantzen wrote in a blog post on SANS' Internet Storm Center security alert site.

The ISC has tracked public exploits that apparently focus on leaking protected information.

According to Microsoft, which has written up the issue in its Knowledge Base article 328832, hit-highlighting with Webhits.dll only relies on the Microsoft Windows NT ACL (Access Control List) configuration on 5.x versions.

Microsoft "strongly [recommends] that all users upgrade to IIS (Internet Information Services) version 6.0 running on Microsoft Windows Server 2003. IIS 6.0 significantly increases Web infrastructure security," the company wrote in its KB article.

Microsoft is currently shipping IIS 6.0 of the Internet Information Services Server for Windows Server 2003. Microsoft is up to IIS 7.0 for Windows Vista and IIS 5.1 for Windows XP Professional.

What are the security issues with Microsoft's "Surface"?

Yet, in spite of urging upgrading in order to gain improved security, Microsoft is treating the bug as a nonissue, providing no workaround nor indications that it will patch versions 5.0 and 5.1. "This behavior is by design," the KB article asserts.

Rather than supply a patch or workaround, Microsoft published six steps to reproduce the exploit—a response that is "a bit atypical," according to Frantzen. "Microsoft is telling the world how to exploit their products being used by their customers. Not that the worst of those interested in it did not already know, but the one thing we need from Microsoft is not the exploit, but the patch or at least a decent work-around," Frantzen wrote.

The only defensive information Microsoft gives is to urge users to upgrade to 6.0—an upgrade that's neither free nor easy, Frantzen pointed out. He provided these possible workarounds:

If you don't use the Web hits functionality, a simple workaround would be to remove the script mapping for .htw files. Without a script mapping, IIS should treat the file as static content.

Try to use application-level firewalls (filters). If you have the infrastructure it can be a temporary measure till you can upgrade IIS, solving the actual problem.

URLScan, a URL filter by Microsoft can be used to stop access to .htw files and is reported by some SANS-ISC readers as being effective.

Manage rights on the confidential files or directories themselves.

Upgrade to Apache or another Web server, with or without a (cross) upgrade of the OS.

Scramble an upgrade to Windows 2003, potentially on more potent hardware.

Frantzen advised IIS 5.x users that failing to find "null.htw" in a document root directory doesn't mean much—the exploit doesn't need the file.

Microsoft hadn't delivered a statement by the time this story posted.

Source: Lisa Vaas - eWEEK

OSPF is an internal routing protocol. (While primarily used inside a single company, it can span multiple sites.) Based on RFC 2328, it's an open standard. Because of this, OSPF is available on Microsoft's Windows Server 2003 OS, Linux, and many other network devices - unlike Cisco's EIGRP routing protocol. Like other dynamic routing protocols, OSPF enables routers to disclose their available routes to other routers.

OSPF is a link-state routing protocol that runs Dijkstra's algorithm to calculate the shortest path to other networks. Taking the bandwidth of the network links into account, it uses cost as its metric. OSPF works by developing adjacencies with its neighbors, periodically sending hello packets to neighbors, flooding changes to neighbors when a link's status changes, and sending "paranoia updates" to neighbors of all recent link state changes every 30 minutes.

While OSPF is an excellent routing protocol for networks of all sizes, one of its weaknesses is that it can be quite complex to configure. On the other hand, it offers more features than simpler protocols such as RIP.

Here are some of OSPF's strengths:

* It converges quickly, compared to a distance-vector protocol.
* Routing update packets are small, as it doesn't send the entire routing table.
* It's not prone to routing loops.
* It scales very well for large networks.
* It recognizes the bandwidth of a link and takes this into account in link selection.
* It supports variable-length subnet masks (VLSM) or Classless Inter-Domain Routing (CIDR).
* It supports a long list of optional features that many others don't.

Configure OSPF

Some may find OSPF configuration intimidating, so let's look at how to make it easy. Let's start with a basic network: Our network example has two routers - one in San Diego (192.168.1.0 /24) and one in Dallas (192.168.2.0 /24). Between these two routers, there's a point-to-point T1 circuit with IP network address 1.1.1.0/30. The San Diego router's WAN interface is 1.1.1.1, and the Dallas router's WAN interface is 1.1.1.2.

We'll begin by configuring the router in San Diego. The first step to configuring OSPF is to use the router ospf command when in Global Configuration Mode. Here's an example:

Router(config)# router ospf {process number}
Router(config-router)#

While it doesn't matter which process number you use, I recommend keeping it the same on all OSPF routers on your network. I usually use 100 to keep everything simple. However, if you use different process numbers, OSPF will still work and exchange all routes - unlike EIGRP.

After entering OSPF Configuration Mode, the most common next step is to specify which networks OSPF will advertise, which you can do using the network command. Here's an example:

Router(config-router)# network 192.168.1.0 0.0.0.255 area 0
Router(config-router)# network 1.1.1.0 0.0.0.3 area 0

The first parameter is the network ID, and the second parameter is the inverse mask. The inverse mask - or wildcard mask - is the inverse of the subnet mask. It tells OSPF what range of interfaces the IP addresses given will apply to. Therefore, you can have one network statement that covers multiple interfaces.

You also need to specify the area, which is how OSPF organizes networks. All traffic must flow through area 0. In a small network, it's logical to put all networks in area 0, as we did in the example.

After you've configured each side of the network, the routers will exchange routes and form adjacencies. You should see a statement in the log file or console that looks something like the following:

*Mar  1 02:53:33.370: %OSPF-5-ADJCHG: Process 100, Nbr 1.1.1.1 
  on Ethernet0/0 from LOADING to FULL, Loading Done

To make sure you see these types of messages, use the log-adjacency-changes command in your OSPF router configuration. This command causes OSPF to enter information into the router's log file whenever it loses or regains connectivity with its neighbors. Here's an example:

Router(config-router)# log-adjacency-changes

Check the status of OSPF

After you've configured OSPF, you need to know how to check its status. Here are some common OSPF commands, along with links to their Cisco documentation and sample output from our example:

* show ip ospf
* show ip ospf neighbor
* show ip ospf interface
* show ip route ospf

For more information on OSPF, see Cisco's OSPF Design Guide and Cisco's OSPF Documentation.

Know the EIGRP basics

Let's start with the fundamentals of EIGRP and discuss how to configure this protocol.

What type of routing protocol is EIGRP?
EIGRP is a hybrid-distance-vector routing protocol. It's primarily a distance-vector routing protocol, but it also uses the same composite metrics as Interior Gateway Routing Protocol (IGRP). EIGRP uses the Diffusing-Update Algorithm (DUAL) to perform look-free routing and calculate the shortest path.

How does EIGRP work?
With EIGRP, two routers form a neighbor relationship and exchange routes. Hello packets ("keepalives") are present between the two routers; they serve to let each side know if the other goes down or if the link between them goes down.

Typically, these keepalives between neighbors are multicast packets. The type of multicast used is Reliable Transport Protocol (RTP), and communication takes place using the reserved IP address 224.0.0.10.

How do I configure EIGRP?
Like OSPF, EIGRP uses autonomous system numbers to identify areas of the network that are under a single administrative domain. In other words, these network areas are under the control of a single part of the company or a certain group.

To activate EIGRP on your router and enter its Configuration Mode, use the router eigrp command while in Global Configuration Mode. Here's an example:

Router(config)# router eigrp {AS number}
Router(config-router)#

It doesn't matter which Autonomous System (AS) number you use - as long as it's the same on all routers that will be talking to each other. Valid options for the AS number are 1 to 65535. While you can configure more than one AS on a single router, Cisco doesn't recommend this approach

After entering the EIGRP Configuration Mode, a network administrator's most common task is to specify which networks EIGRP will advertise. You can accomplish this using the network command. Here's an example:

Router(config-router)# network 10.0.0.0 0.255.255.255

The first parameter is the network IP address; the second parameter is the inverse mask. The inverse mask (or wildcard mask) is the inverse of the subnet mask.

This command is similar to the OSPF network command. It tells OSPF which range of interfaces the specified IP addresses will apply to. So, you can have one network statement that covers multiple interfaces. However, unlike OSPF, EIGRP does not use areas.

How do I see what's going on with EIGRP?
After you've configured EIGRP, you need to know how to check its status. Here's a list of the most common EIGRP commands as well as links to their Cisco documentation:

* show ip eigrp neighbors
* show ip eigrp interfaces
* show ip eigrp topology
* show ip eigrp traffic

Study the vocabulary

Now that you know how to configure EIGRP and you're familiar with common commands, let's define some words you may run across while working with EIGRP.

What is the topology table?
Essentially, the topology table is the EIGRP database of available routes received from neighbors. It shows the metric for these routes as well as the feasible distance to these networks. The topology table contains a lot of information about successors, feasible successors, and feasible distance.

What is a successor?
A successor is the neighbor with the best path to a destination.

What is a feasible successor?
A feasible successor is the neighbor or neighbors that have other loop-free paths to a destination that aren't a preferred as the successor's path.

What is the feasible distance?
The feasible distance is the metric of a network advertised by the connected neighbor plus the cost to get there.

What is an adjacency?
An adjacency is when two neighbors form a relationship and are exchanging routes.

Get more specific

Now let's take a look at some specifics about using EIGRP.

Does EIGRP use split horizon?
Split horizon is a loop-prevention method. Essentially, when using split horizon, a routing protocol tries to prevent a routing loop. It does this by not advertising a route from an interface from which it received an advertisement for that route.

EIGRP uses split horizon, but you can disable it if necessary. To do so, use the no ip split-horizon eigrp {AS number} command. Keep in mind that the no ip split-horizoncommand doesn't affect EIGRP, as it would RIP. Link-state routing protocols such as OSPF and the Intermediate System-to-Intermediate System (IS-IS) protocol don't use split horizon.

Does EIGRP support VLSM or CIDR?
EIGRP carries the subnet mask in the routing update, and it does support both variable length subnet masks (VLSM) and Classless Inter-Domain Routing (CIDR). In other words, you can subnet your network from the classful boundaries (where a class A network is 10.0.0.0 with a 255.0.0.0 subnet mask, etc.), and EIGRP will work fine. (Lack of support for VLSM and CIDR are limitations of RIP and IGRP.)

By default, EIGRP summarizes networks at the classful boundaries. You can disable this by using the no auto-summary command in Router Configuration Mode.

What is the administrative distance (AD) and routing table code for EIGRP?
An entry in the routing table for EIGRP looks something like the following:

D       10.93.103.0/24 [90/5542656] via 10.226.100.1, 00:30:54, Serial5/0

The D at the beginning tells you that this is EIGRP. The 90 is the administrative distance for this EIGRP route. This is the default administrative distance for EIGRP.

What happens when EIGRP is "stuck in active"?
"Stuck in active" is a common issue with EIGRP. In fact, it's so common that Cisco has an acronym for it: SIA. Cisco has also created a support page for SIA; however, Cisco login information is required.

Essentially, SIA occurs when an EIGRP router doesn't receive a query reply sent to its neighbor after three minutes. When this happens, you'll see "DUAL-3-SIA" in the log file. Troubleshooting this can be quite complex, so I would refer to the Cisco documentations.