The Raven Report

Wow, it’s been quite a while since I last posted, kind of changed focus, but hopefully I can find some time a post a little more in the near future. As of late I’ve been doing a lot of performance analysis and tweaking, and I’d like to share some exciting results with all of you.

Over the past few months system performance especially disk has been the bane of my existence. Working with my team we have uncovered a few elusive bottlenecks is disk subsystems of a couple of our critical systems and really learned a lot.  I have quite a bit written up on this topic and hope to share that in the coming posts.

One of the big things we found was the dramatic performance increases one can obtain with the use of SSD disks when it pertains to DAS and SAN storage, the big issue here is cost being that 1TB of usable SSD Storage is over $150k. That is quite the pill to swallow, but certain applications that can justify the expense can see remarkable improvements.

Well until next time.

Oh yeah, Burger King finaly sent my wife a check for $20 they stole from her.

My wife and a couple friends frequent a local Burger King 2-3 times a month, and let the kids play in the play area.  On one such trip back in October my wife was overcharged $10 so instead of a $15 charge she was charged $25. Being an accountant the wife noticed the discrepancy fairly quickly. She also alerted her friends that were with her on that day of the issue, one of them was also charged $5 extra.

So the wife called Burger King and explained the issue. Seeing that her friend got his $5 back she didn’t think it would be an issue, but since she didn’t have her original receipt, store management was less than helpful.

If the customer says you charged them too much, and you know that another customer on the same day was over charged, then it would be advantageous to the business to err on the customer’s side and at least do a little investigation.  Neither of which were done in this case.

Now had this been me,, I would have broke the picket signs and besieged the Burger King. But due to the fact the Burger King in question is the only one with a play area and centrally located for the group, my wife decided to continue to frequent the establishment with the change of only paying with cash.

The cash basis Burger King went on for 2 months without a hitch, but then the mistake was made to use the debt card again toward the end of December. Low and behold an extra $10 was once again tacked on to the bill. Once again store management was unresponsive and it took days  even answer the phone.  Corporate was contacted and the number of the Owner was obtained.

After discussing the issue with the owner, records were faxed over. A couple days later security contacted my wife and explained that they would be investigating the issue.   Now fast forward a  couple weeks,  the security guy calls my wife and explains that they were able to isolate the employee and that employee was fired after admitting to overcharging unknown numbers customers over a 4 month period. They are also looking to press charges against the ex-employee.

Now Burger King still hasn’t refunded our $20 or provided any concessions even after it was found to be fraud on their part.  Which leads me back to question why wouldn’t management, owner, security or anybody refund the overcharge as soon as it looks plausible that the customer has a valid complaint?  Heck, they even asked my wife to be come down to the court house and file charges against the employee, which we promptly declined.  Come on we have already wasted much more than the $20 in the time it took to even contact them. Had this issue been taken care of and investigated in October countless customer would not had their hard earned money stolen from them when all they wanted was a Whopper!

The sad part here is this employee was overcharging 100s of customers over a 4 month period and no one caught on.  I know we couldn’t have been the only ones who noticed the incorrect charges, and attempted to contact management. How many customers just gave up? I mean what truly is $5-$10. Heck, the first time we ended up letting it slide after not getting anywhere with management.

Moral of the story: If someone contacts you and says they have a problem they probably do, or they wouldn’t waste their time. So take each inquiry seriously, and don’t ignore your customers or you soon will have no customers to ignore.  

While troubleshooting excessive client login times, we identified an issue where the clients in one site would authenticate with DCs on the other side of the country.  

In our case a few of our sites are very well connected to our core site via high speed low latency links, so there are not any local DCs at site that users report slow logins.   

Initial test were inconclusive as a well connected DC was selected by the client, but we got lucky and one of our test system started experiencing the slow login issue.  After login set logonserver was run from the command line and a DC in the remote location with > 115ms latency was selected by this particular client PC.  At this point we were pretty sure the issue was in the AD site configuration, but did a couple more tests to confirm the suspicion.

We then repeatedly ran the following command multiple times to confirm our suspisions:

nltest /dsgetdc:domainname /force

The nltest command with the dsgetdc uses the same API the client does to select a DC.  In our case after multiple runs of the command DCs on the other side of the country were returned.

After notifying the directory services team of the issue it was determined that at the direction of an AD Consultant the site links between the core site and remote site were removed, and since no DCs were present on the site and the clients didn’t have enough information to determine the proper cost of a DC and were randomly selecting any DC in the environment. 

The AD team recreated the site links and the nltest  command  now only returns DCs in the core site.

 

It appears Microsoft may have dropped DHCP Failover from the Windows 2008 R2 release.  Hopefully they come to their senses, but we’ll see. Feel free to continue reading to see what might have been.

 

About time, Windows 2008 R2 will now support DHCP Failover.

DHCP server services are used to provide automated IP configuration for network endpoints.  Traditionally DHCP lease information is stored in a single database, and this single computer is prone to be a single point of failure. Now you can Cluster the DHCP service, but this still leaves you venerable if the DHCP database gets corrupted.   In the past the only way to elevate this was to use something like a split of the DHCP Scope between 2 servers. This did improve the reliability, but this option didn’t fit in many environments do to the need to have many extra addresses in every subnet.

In Windows Server 2008 R2, the DHCP Failover feature has been included to allieviate outages due to single server failure. The DHCP Failover feature is an implementation of the DHCP Failover protocol.

With DHCP Failover, the servers providing DHCP Server services synchronize DHCP lease information between each other. One computer is designated as the primary DHCP server and the other as the secondary DHCP server.

When computers request IP configuration, the primary DHCP server will respond. In the event that the primary DHCP server is unavailable, the secondary DHCP server will service the request. The big difference here is the secondary server knows about the current leases that the primary server made, and will be able to renew existing leases, and not be forced to give out a new address. This also eases administration and the DHCP scopes do not need to be split between servers.  The DHCP Failover feature will also support two-way syncing to support load-balancing between the primary and secondary DHCP servers.

Now granted there have been some appliances in the market that supported this functionality for the past few years, but this is a great addition to the Window Server feature set. This was something that was lacking from the original Windows 2008 release, but appears to be making it in the R2.

 

Performance Analysis of Logs (PAL) is a nifty utility that will read in a performance counter log, analyze the data against customizable XML files with many built in counter thresholds, and then display the information in a HTML report.

You can find PAL Here:

http://www.codeplex.com/PAL

PAL only needs to be installed on the system that you are going to process the logs for not the System you are monitoring.  Go ahead download and install PAL and it’s prerequisites (Log Parser 2.2 & Microsoft Office Web Components 2003).  I wait till you’re done.

Now that PAL is installed we need to setup some Counter Logs on the system we want to analyze.  To do this go to Start then run and type Perfmon.  Go into Performance Logs and Alerts right-click Counter Logs and choose New Log Settings…

 

 

For a general system overview add the following counter objects:

LogicalDisk
Memory
Network Interface
PhysicalDisk
Process
Processor
System
TCPv4

Change the interval to something like 60 Seconds and click OK.

After saving the counter log settings, it should automatically start collecting data.  You can tell if it’s running because the icon will be green. 

Once the Counter Logs has been running for a sufficient amount of time go ahead and stop the logger.  At this point, you’ll want to copy the log file to the system that you installed PAL on.

Sidenote: 

PAL does come with VBScripts for creating these counters on the fly.  These are located C:\Program Files\PAL\PAL v1.3.4.2\PerfmonLogScripts .

Syntax:

cscript CreateAndStartPerfmonLogs.vbs <computer[;computer]> <ServerType> <CounterListFilePath>

<computer[;computer]>  List of computers to create and start the perfmon log.

<ServerType>           Text description to be added to the log name.

<CounterListFilePath>  File containing the list of perfmon counters.

Example:

Open a command window and change directory to C:\Program Files\PAL\PAL v1.3.4.2\PerfmonLogScripts.

cscript  CreateAndStartPerfmonLogs.vbs myserver sysoverview CounterList_SystemOverview.txt 

 

OK now back to Analyzing the Counter File:

Go ahead and open the PAL GUI and Click Next

On the Counter Log Screen you’ll want to browse out to the Log file we created earlier and click Next.

 

On the Threshold File screen Verify that the System Overview threshold file is selected. After and answering  the Question Variables Names  section in the lower left, Click Next.

 

Click Next to accept the defaults on the Analysis Interval screen:

 

Customize  the name and location of the output report on the Output Options Screen and click Next

 

 

The Queue Page displays the queued commands that the wizard has built Click Next to move to the final screen

 

 

 

On the Execute Screen verify Execute: Execute what is in the queue. is selected and choose Finish.

 

 

At this point PAL will kick off a VBScript that will slice and dice up the log file and generate a detailed report.  The report will highlight any thresholds that were exceeded, at the top of the report in the alerts section, and contains charts and suggested resolutions information for many of the counters analyzed. 

While adding a new disk to a server we ran into an issue where we were getting  the following error message when trying to delete a folder from a mounted drive.

Cannot delete Folder: Access is denied. The source file may be in use

Well it turns out that this is a know issue and only happens when try to delete a folder that is on a mount point.  Apparently the Recycle Bin not understand mounted volumes and attempts to store the deleted folder in the Recycle Bin of the parent drive which doesn’t work since the folder doesn’t exist on that drive.  Since the Recycle Bin is not used when deleting folders over the network, this issue only appears when deleting folders locally.

To work around this you can bypass the Recycle Bin with a Shift+Delete or delete it via a network share. 

For more details check out this KB Article http://support.microsoft.com/kb/243514

I was looking for the advanced network settings on a Windows 2008 Server I noticed that the menu bar (file, edit, view, Advanced, etc) is no longer visible by default.  Guess I hadn’t really needed it until this point.  How to get it back you ask? Well there are two ways.

1.       Just like in DOS days , press the ALT key and the bar will pop back up   

2.   Now say you are ALT key challenged and don’t want to use the ALT key. Click on Organize then Folder and Search Options  from there switch to Use Windows classic folders

 

 

Problem: 

PXE boot option will not map a drive (default F:) to the DS. All the configuration seems correct and the password has been verified.

Cause:

Altiris uses a LAN Manager Hash file to store  and user the password. Most security polices require the hashing of passwords to be disabled for the domain /server.  This is normally disabled via group policy in the Network security: Do not store LAN Manager hash value on next password change setting being enabled.

Solution:

In most environments it will not be possible to enable this setting across the domain due to security policies, but you can do it for local accounts if you move the server in question to an OU where a policy  over the Network security: Do not store LAN Manager hash value on next password change  to disabled. After the policy propagates reboot the server and create a new local account for PXE booting. You will now be able to use this account in your PXE Boot options just use workgroup as the domain.  Once the password is set you could re-enable the Network security: Do not store LAN Manager hash value on next password change   as this setting only applies during password changes.

 

If you ever wanted to remove a boot option from the PXE configuration utility that was flagged in use then you know it can be quite a pain figuring out what jobs are referencing the boot option and changing them. I have even had a couple occasions where the boot options were changed but I still couldn’t remove the old PXE option.

So do you just leave it and act like it doesn’t exist? No I say, to the database we must go.

First open the PXE Config utility select and choose edit on the Boot option in question.

Once the Edit Shared menu Option is open look at the File location and get the  number from the end of the path. 161 in our case

Now open SQL Query editor and point it to the Express DB

Run the following Query

select * from dbo.task
where bootoption_id = 161

This will return any jobs that reference the boot option. In this case Event_id 200735 references this boot option.  There is also a task_seq which is used if there are additional steps in a job that reference the boot option.

Now we need to run another query get  the Job info this si stored in the events table

select * from dbo.event

where event_id = 2000735

 

From this query we can get the name of the job. Once we have the name we can go into the DS console and easily find the job and change the boot option. Once these are updated we can once again go into the PXE Config Utility and delete the offending Boot option.

  

I regularly have requests to allow a regular user of a system access the event viewer remotely.   What would seem like a trivial task ends up taking some thought as there is no built in way to easily allow this access on Windows 2003.

So say we want to allow Jim Bob user access to the System event log on our server.

First we need to open Regedit and browse out to the following key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\System

Next dsd well want to copy the CustomSD value into a text editor and add access for Jim Bob

Original Value:

O:BAG:SYD:(D;;0xf0007;;;AN)(D;;0xf0007;;;BG)(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x5;;;SO)(A;;0x1;;;IU)(A;;0x1;;;SU)(A;;0x1;;;S-1-5-3)(A;;0x2;;;LS)(A;;0x2;;;NS)

Now since we only want to give him read access we’ll mirror the Interactive User’s (IU) default permissions.

O:BAG:SYD:(D;;0xf0007;;;AN)(D;;0xf0007;;;BG)(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x5;;;SO)(A;;0x1;;;IU)(A;;0x1;;;SU)(A;;0x1;;;S-1-5-3)(A;;0x2;;;LS)(A;;0x2;;;NS)(A;;0x1;;; S-1-5-3-3127463467463)

Just copy your newly minted SDDL string back into the CustomSD key and Jim Bob will be good to go.

Now say you just want to allow all Authenticated Users (AU) access you could just modify your SDDL as follows:

O:BAG:SYD:(D;;0xf0007;;;AN)(D;;0xf0007;;;BG)(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x5;;;SO)(A;;0x1;;;IU)(A;;0x1;;;SU)(A;;0x1;;;S-1-5-3)(A;;0x2;;;LS)(A;;0x2;;;NS)(A;;0x1;;;AU)

Rinse and repeat for any additional event logs that you want to grant access to.

Windows 2008 is much easier as long as you are ok giving the user/group read access to all event logs. If that is the case just add them to the Built in Event Log Readers group.

Now if you want to customize things like give someone access to the Application and System log but not the security log you still have to dig into the SDDL.

The location on the SDDL has changed in Windows 2008 and is no longer set it via the CustomSD in the registry. You now have to use the wevtutil utility. Ok so let’s say Jim Bob now needs access to just the System event log on our Windows 2008 Server.

First we need to open the command prompt, and run the following command to dump out the SDDL for the System log out to a txt file.

wevtutil gl system > C:\temp\out.txt

Open the text file and copy out the channelAccess: entry

channelAccess: O:BAG:SYD:(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x5;;;SO)(A;;0x1;;;IU)(A;;0x1;;;AU)(A;;0x1;;;SU)(A;;0x1;;;S-1-5-3)(A;;0x2;;;LS)(A;;0x2;;;NS)(A;;0x2;;;S-1-5-33)(A;;0x1;;;S-1-5-32-573) )

Now once again copy the Interactive User (IU) rights and add Jim Bob to them.

O:BAG:SYD:(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x5;;;SO)(A;;0x1;;;IU)(A;;0x1;;;AU)(A;;0x1;;;SU)(A;;0x1;;;S-1-5-3)(A;;0x2;;;LS)(A;;0x2;;;NS)(A;;0x2;;;S-1-5-33)(A;;0x1;;;S-1-5-32-573) (A;;0x1;;; S-1-5-3-3127463467463))

Last we need to apply the new SDDL. Just replace the O:BAG:XXXX with your SDDL String you created in the previous step.

wevtutil sl System /ca:O:BAG:XXXX

There you have it.  As another option, you can remove access for the Event Log Readers group from event log in question by removing the (A;;0x1;;;S-1-5-32-573) entry from the respective log SDDL String.