The Raven Report

Wow, it’s been quite a while since I last posted, kind of changed focus, but hopefully I can find some time a post a little more in the near future. As of late I’ve been doing a lot of performance analysis and tweaking, and I’d like to share some exciting results with all of you.

Over the past few months system performance especially disk has been the bane of my existence. Working with my team we have uncovered a few elusive bottlenecks is disk subsystems of a couple of our critical systems and really learned a lot.  I have quite a bit written up on this topic and hope to share that in the coming posts.

One of the big things we found was the dramatic performance increases one can obtain with the use of SSD disks when it pertains to DAS and SAN storage, the big issue here is cost being that 1TB of usable SSD Storage is over $150k. That is quite the pill to swallow, but certain applications that can justify the expense can see remarkable improvements.

Well until next time.

Oh yeah, Burger King finaly sent my wife a check for $20 they stole from her.

 

It appears Microsoft may have dropped DHCP Failover from the Windows 2008 R2 release.  Hopefully they come to their senses, but we’ll see. Feel free to continue reading to see what might have been.

 

About time, Windows 2008 R2 will now support DHCP Failover.

DHCP server services are used to provide automated IP configuration for network endpoints.  Traditionally DHCP lease information is stored in a single database, and this single computer is prone to be a single point of failure. Now you can Cluster the DHCP service, but this still leaves you venerable if the DHCP database gets corrupted.   In the past the only way to elevate this was to use something like a split of the DHCP Scope between 2 servers. This did improve the reliability, but this option didn’t fit in many environments do to the need to have many extra addresses in every subnet.

In Windows Server 2008 R2, the DHCP Failover feature has been included to allieviate outages due to single server failure. The DHCP Failover feature is an implementation of the DHCP Failover protocol.

With DHCP Failover, the servers providing DHCP Server services synchronize DHCP lease information between each other. One computer is designated as the primary DHCP server and the other as the secondary DHCP server.

When computers request IP configuration, the primary DHCP server will respond. In the event that the primary DHCP server is unavailable, the secondary DHCP server will service the request. The big difference here is the secondary server knows about the current leases that the primary server made, and will be able to renew existing leases, and not be forced to give out a new address. This also eases administration and the DHCP scopes do not need to be split between servers.  The DHCP Failover feature will also support two-way syncing to support load-balancing between the primary and secondary DHCP servers.

Now granted there have been some appliances in the market that supported this functionality for the past few years, but this is a great addition to the Window Server feature set. This was something that was lacking from the original Windows 2008 release, but appears to be making it in the R2.

 

Performance Analysis of Logs (PAL) is a nifty utility that will read in a performance counter log, analyze the data against customizable XML files with many built in counter thresholds, and then display the information in a HTML report.

You can find PAL Here:

http://www.codeplex.com/PAL

PAL only needs to be installed on the system that you are going to process the logs for not the System you are monitoring.  Go ahead download and install PAL and it’s prerequisites (Log Parser 2.2 & Microsoft Office Web Components 2003).  I wait till you’re done.

Now that PAL is installed we need to setup some Counter Logs on the system we want to analyze.  To do this go to Start then run and type Perfmon.  Go into Performance Logs and Alerts right-click Counter Logs and choose New Log Settings…

 

 

For a general system overview add the following counter objects:

LogicalDisk
Memory
Network Interface
PhysicalDisk
Process
Processor
System
TCPv4

Change the interval to something like 60 Seconds and click OK.

After saving the counter log settings, it should automatically start collecting data.  You can tell if it’s running because the icon will be green. 

Once the Counter Logs has been running for a sufficient amount of time go ahead and stop the logger.  At this point, you’ll want to copy the log file to the system that you installed PAL on.

Sidenote: 

PAL does come with VBScripts for creating these counters on the fly.  These are located C:\Program Files\PAL\PAL v1.3.4.2\PerfmonLogScripts .

Syntax:

cscript CreateAndStartPerfmonLogs.vbs <computer[;computer]> <ServerType> <CounterListFilePath>

<computer[;computer]>  List of computers to create and start the perfmon log.

<ServerType>           Text description to be added to the log name.

<CounterListFilePath>  File containing the list of perfmon counters.

Example:

Open a command window and change directory to C:\Program Files\PAL\PAL v1.3.4.2\PerfmonLogScripts.

cscript  CreateAndStartPerfmonLogs.vbs myserver sysoverview CounterList_SystemOverview.txt 

 

OK now back to Analyzing the Counter File:

Go ahead and open the PAL GUI and Click Next

On the Counter Log Screen you’ll want to browse out to the Log file we created earlier and click Next.

 

On the Threshold File screen Verify that the System Overview threshold file is selected. After and answering  the Question Variables Names  section in the lower left, Click Next.

 

Click Next to accept the defaults on the Analysis Interval screen:

 

Customize  the name and location of the output report on the Output Options Screen and click Next

 

 

The Queue Page displays the queued commands that the wizard has built Click Next to move to the final screen

 

 

 

On the Execute Screen verify Execute: Execute what is in the queue. is selected and choose Finish.

 

 

At this point PAL will kick off a VBScript that will slice and dice up the log file and generate a detailed report.  The report will highlight any thresholds that were exceeded, at the top of the report in the alerts section, and contains charts and suggested resolutions information for many of the counters analyzed. 

While adding a new disk to a server we ran into an issue where we were getting  the following error message when trying to delete a folder from a mounted drive.

Cannot delete Folder: Access is denied. The source file may be in use

Well it turns out that this is a know issue and only happens when try to delete a folder that is on a mount point.  Apparently the Recycle Bin not understand mounted volumes and attempts to store the deleted folder in the Recycle Bin of the parent drive which doesn’t work since the folder doesn’t exist on that drive.  Since the Recycle Bin is not used when deleting folders over the network, this issue only appears when deleting folders locally.

To work around this you can bypass the Recycle Bin with a Shift+Delete or delete it via a network share. 

For more details check out this KB Article http://support.microsoft.com/kb/243514

I was looking for the advanced network settings on a Windows 2008 Server I noticed that the menu bar (file, edit, view, Advanced, etc) is no longer visible by default.  Guess I hadn’t really needed it until this point.  How to get it back you ask? Well there are two ways.

1.       Just like in DOS days , press the ALT key and the bar will pop back up   

2.   Now say you are ALT key challenged and don’t want to use the ALT key. Click on Organize then Folder and Search Options  from there switch to Use Windows classic folders

 

 

I have looked forward forward to Windows 2008 Remote App and Single Sign On (SSO) for some time now. Shortly after the Server 2008 release, I looked into publishing a few troublesome application to our XP SP2 workstations. Well this kind of worked. I could get the application down to the system, but dragging the application across two screens did not work and SSO did not work. At the time SP3 was in beta, so I gave it a try and with a few reg changes got SSO and dual monitors to work. Since this was beta, and it wasn’t worth pushing a beta sp to the desktops I shelved the project until SP3 released.

Along comes SP3,  and I install it on a test machine, push my Reg changes and… nothing.  Well not exactly nothing, the application  does open and it now works correctly on dual screens, but SSO doesn’t work at all. Now to the naked eye it appears Microsoft left the feature out, but after a little digging and a couple more Registry modifications SSO is working. Well mostly working SSO to a TS farm/alias from XP SP3  still doesn't work.

That little setback aside here how to do it:

First off to get SSO working, we need to enable credssp and add tspkg to the security packages on the client system.

1.       Start Regedit

2.       Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa

                Double click the Security Packages and add tspkg to the bottom on the multi string value.

            

3.       Now navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders

4.       Double click the SecurityProviders and add credssp.dll to the end of the string data.

            CREDSSP.reg (1.07 kb)

 

Next we need to enable pass-through Kerberos authentication to our Windows 2008 Terminal Server on our client systems. This is easily done with group policy with Vista, but we have to make registry changes in XP.

1.       Navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

2.       Create the following values:
"AllowDefaultCredentials"=dword:00000001
“ConcatenateDefaults_AllowDefault"=dword:00000001

3.       While still at: HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

4.       Create a new key  AllowDefaultCredentials

5.       Now inside your new key create as string values for each server you want to connect to.
"1"="TERMSRV/Server1.domain.com"
"2"="TERMSRV/Server2"
"3"="TERMSRV/*.domain.com"    (This will enable SSO to all

6.       Now reboot and you should be good to go

             AllowDefaultCredentials.reg (768.00 bytes)

You can also enable NTLM pass through. In general I’d recommend using Kerberos, besides it doesn’t appear that NTLM pass-through is working in XP SP3 when connecting to a TS, but if you insist here’s how:  

1.       Navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

2.       Create the following values:
"AllowDefCredentialsWhenNTLMOnly"=dword:00000001
"ConcatenateDefaults_AllowDefNTLMOnly"=dword:00000001

3.       While still at: HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

4.       Create a new key  AllowDefCredentialsWhenNTLMOnly

5.       Now inside your new key create as string values for each server you want to connect to.
"1"="TERMSRV/Server1.domain.com"
"2"="TERMSRV/Server2"
"3"="TERMSRV/Server3"

            AllowDefCredentialsWhenNTLMOnly.reg (804.00 bytes)

Now if everything goes well you should be able open a RDP connection, be it full remote desktop or Remote App without having to retype your login credentials. There is one big caveat with SSO and XP Sp3 though. As mentioned above, at this point SSO does not work from XP sp3 to a TS Farm or alias, even when you allow NTLM pass through.  You can easily get this working in Vista with server certificates, but no amount of banging your head against the keyboard will get it to work in XP.  I will post an update if I ever figure out how to get it working though.

 

Since a 32 bit OS can only access 4GB of memory, some magic needs to happen to allow the system to see additional memory.  Say hello to Physical Address Extension (PAE).  PAE allows the operating system itself to address more that 4GB of RAM. Each Process is still bound by the 4GB limits, but the VMM can utilize the all the Physical memory up to 64GB. How does PAE do it?  Starting with the Pentium Pro processor the x86 platform actually allows 36bits of address space, so  2³⁶ bytes  = 64GB addressable. PAE is automatically enabled on computers running Windows Server 2003 with Service Pack 1 (SP1) and Windows XP with Service Pack 2 (SP2) when DEP is enabled on a computer with a processor that supports the no-execute page protection feature.

The 2GB dividing line between user memory and kernel memory is completely arbitrary. 4GB memory tuning (4GT) is enabled by putting a /3GB switch in the boot.ini. Using the /3GB switch allocates 1 GB to the kernel and 3 GB to the User-mode space. But be warned, we are stealing from Peter to pay Paul here, and since Nonpaged Pool, Paged Pool & PTEs all reside in Kernel memory,  If the memory reduction in the pools is too great in a specific server installation, the server or the applications may generate an error or appear to stop responding.

You shouldn’t use 4GB Tuning in the following scenarios. The system has more than 16GB of memory, the server is a Terminal server, large file servers and Exchange are good examples when the /3GB alone does more harm than good

Exchange?? What do you mean Exchange? All the docs say to use the /3GB switch. What gives? Please welcome /userva to the party. The /userva=xxxx switch is designed to allow for more precise tuning of User-mode address space for applications that require more than 2 GB of User-mode space but do not require all the space that is provided by the /3GB tuning switch alone. Use the /userva switch with the /3GB switch in the Boot.ini file to tune the User-mode space to a value between 2 and 3 gigabytes (GB), with the difference (3,072 less xxxx) being returned to Kernel mode.

OK, so now the OS can see more than 4GB of Memory, but my application is still limited to 2-3GB of memory.  How can my applications use more memory?

As stated earlier by using PAE we give the OS the ability to address memory over the 4GB mark. Windows 2003 Enterprise and Datacenter Editions make use of the x86 36bit memories addressing for applications with an API called Address Windowing Extensions (AWE). The AWE API allows an application to transfer memory pages above the 4GB limit into the addressable memory area where it can make changes to the pages. This permits applications such as SQL Server that use large datasets to manage them in RAM rather than a slow paging file. AWE locks physical memory address space to an application. No other application can use this memory and it can’t be pages to disk.

What’s new in Vista and Server 2008 32bit?

Let say we have a Windows 2003 Server running Terminal Server  with the following Kernel memory utilization:

–      Paged pool: 50%

–      Non-paged pool: 70%

–      System PTEs: 98%

Even though there is still fair amount of memory in the paged & nonpaged pools, essentially this system is loaded to its maximum capacity because the System PTEs are fully exhausted. 32bit versions of Windows have relatively low maximum values for these memory areas, that all need to be allocated out of the 1-2GB of kernel virtual memory space. Up until Vista this allocation was only done at boot up.

 In Windows Vista/2008, kernel virtual address space is dynamically allocated. The sizes and locations of the paged pool, nonpaged pool, System Cache & PTEs are no longer fixed, but are dynamically adjusted according to the current operational requirements. This simple but effective change resolves the single pool exhaustion issue described above.  Now, not only can Peter steal from Paul, but Paul can steal back from Peter.

Today we’ll start a multipart post on the Windows memory architecture. I will explain the different address spaces, cover 32bit and 64bit platforms, and provide some options for tuning and troubleshooting memory issues.  Let’s start with the aspects that are common between Windows 32bit and 64bit systems. Physical memory is truly the physical memory in a system. The amount of available physical memory is limited only by the OS maximum.

Virtual memory is the memory that applications use and can be much greater than the Physical memory installed.

Ok, so virtual memory isn’t limited by physical memory then how does it work? The Virtual Memory Manager (VMM) is responsible for mapping each virtual address into a corresponding physical address. The VMM performs the mapping by dividing the RAM into fixed-size page frames, creating system page table entries (PTEs) to store information about these page frames, and mapping them. VMM also manages kernel resources, such as the paged pool, nonpaged pools and the system cache.

User Mode Memory is the address space where all nonsystem (kernel) processes and applications run. Each process has its own virtual memory address space up to the maximum allowable on the OS Architecture. If you have 10 user mode process running on a 32bit windows system each will have its own 2GB address space

Kernel Memory, unlike the User Mode Address space, shares the same address space (2GB by default on a 32bit system), and all kernel allocations share the same memory space.

Kernel memory consists of four main areas:
     Paged Pool consists of virtual memory addresses that can be paged to disk.
     Nonpaged pool consists of virtual memory addresses that are guaranteed to reside in physical memory as long as the corresponding kernel objects are allocated.
     System Cache which is used to cache disk I/O processes for faster file access.
     System Page Table Entries (PTE)

A little more on PTEs, every process that runs on a 32bit Windows system has 4GB  of virtual address space for its use. Of this, the upper 2 GB address (kernel) is common to all processes running in the system. The lower region is called user address space. System PTEs are small chucks of kernel memory that are used to map virtual address space to physical memory. From the process perspective, each element of virtual address maps to a byte of physical memory. It is the responsibility of the Virtual Memory Manager to do this translation, and each PTE represents a page, and contains information necessary for the VMM to locate that page.

That's it for now. We'll get into the 32bit architecture in part 2