The Raven Report

While troubleshooting excessive client login times, we identified an issue where the clients in one site would authenticate with DCs on the other side of the country.  

In our case a few of our sites are very well connected to our core site via high speed low latency links, so there are not any local DCs at site that users report slow logins.   

Initial test were inconclusive as a well connected DC was selected by the client, but we got lucky and one of our test system started experiencing the slow login issue.  After login set logonserver was run from the command line and a DC in the remote location with > 115ms latency was selected by this particular client PC.  At this point we were pretty sure the issue was in the AD site configuration, but did a couple more tests to confirm the suspicion.

We then repeatedly ran the following command multiple times to confirm our suspisions:

nltest /dsgetdc:domainname /force

The nltest command with the dsgetdc uses the same API the client does to select a DC.  In our case after multiple runs of the command DCs on the other side of the country were returned.

After notifying the directory services team of the issue it was determined that at the direction of an AD Consultant the site links between the core site and remote site were removed, and since no DCs were present on the site and the clients didn’t have enough information to determine the proper cost of a DC and were randomly selecting any DC in the environment. 

The AD team recreated the site links and the nltest  command  now only returns DCs in the core site.

While adding a new disk to a server we ran into an issue where we were getting  the following error message when trying to delete a folder from a mounted drive.

Well it turns out that this is a know issue and only happens when try to delete a folder that is on a mount point.  Apparently the Recycle Bin not understand mounted volumes and attempts to store the deleted folder in the Recycle Bin of the parent drive which doesn’t work since the folder doesn’t exist on that drive.  Since the Recycle Bin is not used when deleting folders over the network, this issue only appears when deleting folders locally.

To work around this you can bypass the Recycle Bin with a Shift+Delete or delete it via a network share. 

For more details check out this KB Article http://support.microsoft.com/kb/243514

I regularly have requests to allow a regular user of a system access the event viewer remotely.   What would seem like a trivial task ends up taking some thought as there is no built in way to easily allow this access on Windows 2003.

So say we want to allow Jim Bob user access to the System event log on our server.

First we need to open Regedit and browse out to the following key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\System

Next dsd well want to copy the CustomSD value into a text editor and add access for Jim Bob

Original Value:

O:BAG:SYD:(D;;0xf0007;;;AN)(D;;0xf0007;;;BG)(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x5;;;SO)(A;;0x1;;;IU)(A;;0x1;;;SU)(A;;0x1;;;S-1-5-3)(A;;0x2;;;LS)(A;;0x2;;;NS)

Now since we only want to give him read access we’ll mirror the Interactive User’s (IU) default permissions.

O:BAG:SYD:(D;;0xf0007;;;AN)(D;;0xf0007;;;BG)(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x5;;;SO)(A;;0x1;;;IU)(A;;0x1;;;SU)(A;;0x1;;;S-1-5-3)(A;;0x2;;;LS)(A;;0x2;;;NS)(A;;0x1;;; S-1-5-3-3127463467463)

Just copy your newly minted SDDL string back into the CustomSD key and Jim Bob will be good to go.

Now say you just want to allow all Authenticated Users (AU) access you could just modify your SDDL as follows:

O:BAG:SYD:(D;;0xf0007;;;AN)(D;;0xf0007;;;BG)(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x5;;;SO)(A;;0x1;;;IU)(A;;0x1;;;SU)(A;;0x1;;;S-1-5-3)(A;;0x2;;;LS)(A;;0x2;;;NS)(A;;0x1;;;AU)

Rinse and repeat for any additional event logs that you want to grant access to.

Windows 2008 is much easier as long as you are ok giving the user/group read access to all event logs. If that is the case just add them to the Built in Event Log Readers group.

Now if you want to customize things like give someone access to the Application and System log but not the security log you still have to dig into the SDDL.

The location on the SDDL has changed in Windows 2008 and is no longer set it via the CustomSD in the registry. You now have to use the wevtutil utility. Ok so let’s say Jim Bob now needs access to just the System event log on our Windows 2008 Server.

First we need to open the command prompt, and run the following command to dump out the SDDL for the System log out to a txt file.

wevtutil gl system > C:\temp\out.txt

Open the text file and copy out the channelAccess: entry

channelAccess: O:BAG:SYD:(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x5;;;SO)(A;;0x1;;;IU)(A;;0x1;;;AU)(A;;0x1;;;SU)(A;;0x1;;;S-1-5-3)(A;;0x2;;;LS)(A;;0x2;;;NS)(A;;0x2;;;S-1-5-33)(A;;0x1;;;S-1-5-32-573) )

Now once again copy the Interactive User (IU) rights and add Jim Bob to them.

O:BAG:SYD:(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x5;;;SO)(A;;0x1;;;IU)(A;;0x1;;;AU)(A;;0x1;;;SU)(A;;0x1;;;S-1-5-3)(A;;0x2;;;LS)(A;;0x2;;;NS)(A;;0x2;;;S-1-5-33)(A;;0x1;;;S-1-5-32-573) (A;;0x1;;; S-1-5-3-3127463467463))

Last we need to apply the new SDDL. Just replace the O:BAG:XXXX with your SDDL String you created in the previous step.

wevtutil sl System /ca:O:BAG:XXXX

There you have it.  As another option, you can remove access for the Event Log Readers group from event log in question by removing the (A;;0x1;;;S-1-5-32-573) entry from the respective log SDDL String.

Seems like a simple thing, copying files from one system to another, but sometimes it’s the little things that make us think.

Case in point, a few years back we ran into an issue when copying many large (20GB+) files for one system to another. The system would actually run out of kernel memory after a couple of hours of copying the files and freeze up. A few tweaks and tests later we had found an acceptable number of concurrent processes and the copies went on without a hitch.

That was until recently, when the actual speed of the copy was brought into question.

Speed of the copy that’s fixed right? Not much we can do here outside of using robocopy or some third party tool, and even then we are still limited to by the fact that the transaction is buffered.  Buffered you say, what’s that?  Well in a Buffered transaction Windows keeps track of file reads and writes and buffers these transactions. Normally this works great speeding up future reads and writes to the files, but when dealing with large files the overhead of buffering can be staggering, not only in performance, but in kernel memory usage, hence the original problems with kernel pool exhaustion we experienced in the past.

Ok, so buffered transactions are slow down large file copies, how would we go about doing and unbuffered copy?  As far as I can tell there isn’t a native Windows tool that does un-buffered copy, but there is an Exchange tool ESEUTIL that can be used to accomplish this.  Just copy the ESEUTIL.EXE and ESE.DLL files from your friendly Microsoft Exchange system. The syntax for the command follows:

Eseutil /Y [Sourcefile] /D [Destinationfile]

One caveat with the Eseutil is that you can only do one file at a time and the utility does not accept wild cards so you’ll have to use the full path and file names when copying files.

So is it worth all this? You be the judge. In testing we had the following results for a 50GB  file copy  over a 1Gb connection:

All in all not too shabby,  For more details  on this and other performance related issues check out  the Askperf Blog

Ok so now we know what to look for how to we look for it?

Tools

Task Manager

Task manager can quickly provide general memory information. On the performance tab, you can quickly determine how much physical memory is in the system, how much physical memory is available, Kernel memory usage and how much memory the VMM has promised to running processes. With this information you can determine if further investigation is required.

The processes tab also has a wealth of information, but first you have to enable additional columns via the View à Select Columns option. On this tab you can quickly determine the amount of memory a process is using and how much of the paged & nonpaged pool is being used.  There many addition columns available, so don’t settle for the default view.

Process Explorer does everything task manager does and more. You can really dig into a process and figure exactly what it’s doing and what kind of resources it’s consuming.

Get it here:  http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

Poolmon is a great tool for tracking down kernel memory leaks or just determining what is consuming kernel resources .

More info here: http://support.microsoft.com/kb/177415  
& here: http://technet2.microsoft.com/windowsserver/en/library/0d302498-c947-4655-95af-719ae75acfb51033.mspx?mfr=true

Performance Counters

Memory\Available Mbytes (MB):  Indicates the amount of physical memory (in MB) immediately available for allocation to a process or for system use. Used with the Pages/Sec to determine if the system is low on physical memory.

Memory\Pages/sec: Pages/sec is the number of pages read from the disk or written to the disk to resolve memory references to pages that were not in memory at the time of the reference (hard page faults), and the number of pages that were redirected to memory outside of the working set.  This counter also includes paging traffic on behalf of the system Cache to access file data for applications, so a large number alone is not reason for concern just further investigation. This is the primary counter to observe if you are concerned about excessive memory thrashing.

Memory\Pool Nonpaged Bytes: This counter shows the current size of the Nonpaged pool in Bytes. Anything over 200MB should be investigated, and unless memory tuning options are in place (/3GB) then anything over 100MB.

Memory\Pool Paged Bytes: This counter shows the current size of the paged pool in Bytes. In general anything over ~250MB should be investigated unless memory tuning options are in place (/3GB) then anything over 150MB.

Memory\Free System Page Table Entries: This counter shows the number of page table entries that are available for use. Anything under 8,000 needs a closer look.  If the system is getting low consider using the /userva=XXXX with something like 3030 to start to bump these up.  

Memory: Cache Bytes: This is the size of the system cache. You can use this counter with the Pool nonpaged/paged & PTEs to determine who kernel memory is being allocated.

Registry Keys 

As with all registry modifications, use caution. Under normal circumstances you should not need  to modify these keys, but if you are running into kernel memory issues proper use of these keys can resolve the problem.

NonPagedPoolLimit: Specifies the maximum amount of system VA space that can be used by the nonpaged pool.

PagedPoolLimit: Specifies the maximum amount of system VA space that can be used by the paged pool.

SystemCacheLimit: Specifies the maximum amount of system VA space that can be used by the system cache.

SystemPtesLimit: Specifies the maximum amount of system VA space that can be used by I/O mappings and other resources that consume system page table entries (PTEs).

SessionSpaceLimit: Specifies the maximum amount of system VA space that can be used by session space allocations

PoolUsageMaximum: Determines the Max Poll usage before pruning starts.  The default is 80%. A value of 40-60 provides a workable solution to some paged pool exhaustion issues.

Windows 2003 64Bit can support a theoretical limitation of 16 exabytes or 2⁶⁴

·         Kernel Mode Address space - 8 TB

·         Paged Pool – 128GB

·         NonPaged Pool – 75% of RAM up to a maximum of 128 GB.

·         System Cache/PTEs - 1 TB regardless of physical memory.

As you can see, with 64bit Windows all of the previous memory constraints are blow out of the water.   For just this reason alone upgrading to a 64bit OS should be seriously considered.

Troubleshooting 32bit memory problems

For those who have legacy applications and 32bit hardware upgrading may not be an option. So in the following I will go over some tips and tools to identify memory bottlenecks a 32bit system.

Physical Memory Exhaustion - This is pretty much the straight forward if you are out of physical memory and the system is excessively paging to disk. More physical memory is most likely the answer.

Kernel Memory Exhaustion -  Memory problems in  kernel memory are a little more challenging to nail down.  Here are some of the signs:

Paged Pool & Nonpaged Pool

·         Sluggish or unresponsive user interface

·         Server has message or client processing failures

o   RPC Service Unavailable

o   Not enough Storage to process this command

·         Pool allocation failures:

System PTEs

·         Server fails to respond to I/O requests

·         Server fails to respond to network requests

·         Server has message or client processing failures

If you are experiencing these problems or other unexplained issues kernel memory could be your issue.

Your first stop should be the performance tab in Task Manager. From here you can quickly gauge if you are approaching the kernel memory limits of the system. Take a look at the Kernel Memory section and look at the amounts used by the page pool ( > 250MB or > 150 with /3GB ) and nonpaged pool( >200 or >100 with /3GB).  If we come up higher than these numbers further investigation is required.   We’ll start there next time.

Since a 32 bit OS can only access 4GB of memory, some magic needs to happen to allow the system to see additional memory.  Say hello to Physical Address Extension (PAE).  PAE allows the operating system itself to address more that 4GB of RAM. Each Process is still bound by the 4GB limits, but the VMM can utilize the all the Physical memory up to 64GB. How does PAE do it?  Starting with the Pentium Pro processor the x86 platform actually allows 36bits of address space, so  2³⁶ bytes  = 64GB addressable. PAE is automatically enabled on computers running Windows Server 2003 with Service Pack 1 (SP1) and Windows XP with Service Pack 2 (SP2) when DEP is enabled on a computer with a processor that supports the no-execute page protection feature.

The 2GB dividing line between user memory and kernel memory is completely arbitrary. 4GB memory tuning (4GT) is enabled by putting a /3GB switch in the boot.ini. Using the /3GB switch allocates 1 GB to the kernel and 3 GB to the User-mode space. But be warned, we are stealing from Peter to pay Paul here, and since Nonpaged Pool, Paged Pool & PTEs all reside in Kernel memory,  If the memory reduction in the pools is too great in a specific server installation, the server or the applications may generate an error or appear to stop responding.

You shouldn’t use 4GB Tuning in the following scenarios. The system has more than 16GB of memory, the server is a Terminal server, large file servers and Exchange are good examples when the /3GB alone does more harm than good

Exchange?? What do you mean Exchange? All the docs say to use the /3GB switch. What gives? Please welcome /userva to the party. The /userva=xxxx switch is designed to allow for more precise tuning of User-mode address space for applications that require more than 2 GB of User-mode space but do not require all the space that is provided by the /3GB tuning switch alone. Use the /userva switch with the /3GB switch in the Boot.ini file to tune the User-mode space to a value between 2 and 3 gigabytes (GB), with the difference (3,072 less xxxx) being returned to Kernel mode.

OK, so now the OS can see more than 4GB of Memory, but my application is still limited to 2-3GB of memory.  How can my applications use more memory?

As stated earlier by using PAE we give the OS the ability to address memory over the 4GB mark. Windows 2003 Enterprise and Datacenter Editions make use of the x86 36bit memories addressing for applications with an API called Address Windowing Extensions (AWE). The AWE API allows an application to transfer memory pages above the 4GB limit into the addressable memory area where it can make changes to the pages. This permits applications such as SQL Server that use large datasets to manage them in RAM rather than a slow paging file. AWE locks physical memory address space to an application. No other application can use this memory and it can’t be pages to disk.

What’s new in Vista and Server 2008 32bit?

Let say we have a Windows 2003 Server running Terminal Server  with the following Kernel memory utilization:

–      Paged pool: 50%

–      Non-paged pool: 70%

–      System PTEs: 98%

Even though there is still fair amount of memory in the paged & nonpaged pools, essentially this system is loaded to its maximum capacity because the System PTEs are fully exhausted. 32bit versions of Windows have relatively low maximum values for these memory areas, that all need to be allocated out of the 1-2GB of kernel virtual memory space. Up until Vista this allocation was only done at boot up.

 In Windows Vista/2008, kernel virtual address space is dynamically allocated. The sizes and locations of the paged pool, nonpaged pool, System Cache & PTEs are no longer fixed, but are dynamically adjusted according to the current operational requirements. This simple but effective change resolves the single pool exhaustion issue described above.  Now, not only can Peter steal from Paul, but Paul can steal back from Peter.

Virtual memory is the memory that applications use and can be much greater than the Physical memory installed.

Ok, so virtual memory isn’t limited by physical memory then how does it work? The Virtual Memory Manager (VMM) is responsible for mapping each virtual address into a corresponding physical address. The VMM performs the mapping by dividing the RAM into fixed-size page frames, creating system page table entries (PTEs) to store information about these page frames, and mapping them. VMM also manages kernel resources, such as the paged pool, nonpaged pools and the system cache.

User Mode Memory is the address space where all nonsystem (kernel) processes and applications run. Each process has its own virtual memory address space up to the maximum allowable on the OS Architecture. If you have 10 user mode process running on a 32bit windows system each will have its own 2GB address space

Kernel Memory, unlike the User Mode Address space, shares the same address space (2GB by default on a 32bit system), and all kernel allocations share the same memory space.

Kernel memory consists of four main areas:
     Paged Pool consists of virtual memory addresses that can be paged to disk.
     Nonpaged pool consists of virtual memory addresses that are guaranteed to reside in physical memory as long as the corresponding kernel objects are allocated.
     System Cache which is used to cache disk I/O processes for faster file access.
     System Page Table Entries (PTE)

A little more on PTEs, every process that runs on a 32bit Windows system has 4GB  of virtual address space for its use. Of this, the upper 2 GB address (kernel) is common to all processes running in the system. The lower region is called user address space. System PTEs are small chucks of kernel memory that are used to map virtual address space to physical memory. From the process perspective, each element of virtual address maps to a byte of physical memory. It is the responsibility of the Virtual Memory Manager to do this translation, and each PTE represents a page, and contains information necessary for the VMM to locate that page.

That's it for now. We'll get into the 32bit architecture in part 2 

Background

A few months back we had an issue with our clustered DHCP server. It appears that the virus scanner lost its exceptions on the DHCP logs & database files.  During the built in DHCP backup, which normally happens once and hour, corrupted not only the backup, but the DHCP database itself.  Since the issue didn’t actually kill the DHCP service, our SCOM didn’t catch the failure.  Actually SCOM doesn’t monitor DHCP in a cluster very well, since the management pack doesn’t detect cluster installs.  Because the problem occurred in the middle of the night no one was aware of the joy that waited on arrival to the office.  Long story short, we had about 2 hours of real live, no one working downtime. All the servers and applications were running clients they just couldn’t get addresses to connect.

The Question

How do we restore to a new none clustered server in the event that the entire cluster is unavailable, and do it quickly with minimal impact?

The Answer

To restore the DHCP database in a clustered environment or where the DHCP database is not installed in the default location, your new system must the same path available that was on the original DHCP server.  

So say you have DHCP in a cluster and the data base and log files are in the S:\DHCPDATA you will not be able to restore, unless you have a S:\DHCPDATA available on the new box, since the database in the backup folder isn’t a real database and has the path hard coded into the dhcp.mdb, dhcp.pat & log file.  You can actually mount the dhcp.pat in regedit and modify it, but  since the path is also hardcoded in the database, in the end it doesn’t help us in this situation.

This is all well in good except if you are restoring to a server that has an S:\, but what if that isn’t the case?

Workaround

Use the following command on the DHCP server as part of your backup process

netsh dhcp server export C:\dhcp.txt all

Then use the following command to restore on the new server

netsh dhcp server import C:\dhcp.txt all

This option looks great at first, but when you run this command the DHCP server service stops and then restarts. In a cluster this causes the resource to go offline and back online, which in turn triggers alerts. So either we put the system in maintenance mode or look for other options. Also what if I didn’t implement this, and all I have are the backups from tape?

Now to the heart of the problem:

All we have is the built in backup files, and we need to restore to a server with a S:\

1.    Create a folder called DHCP and share on the C:\ giving system full control

2.    Make sure the file structure matches what you had on your original server.  In our case: dhcpdata\back\new.

3.    Restore your backup files in this directory

4.    Create a persistent drive mapping from the server to itself as the system account

a.    How/Why the hell do I do this?
Well DHCP runs as system so we have to have the drive S:\ available for it
This can be accomplished by doing the following

                                       i.   Open a Command prompt

                                       ii.    Type:
AT <future Time> /interactive cmd /k
this will open a cmd window running  as system

                                       iii. Type the following in the system command window:
Net use s:  \\servername\DHCP  /PERSISTENT:YES

5.    Now that we have a S:\ available to system all we need to do is initiate a restore from the DHCP management tool and browse to the backup files.

Additions Info:

Technet Managing DHCP Databases

How to move a DHCP database