Practical Technology

I accidentally the config file

2012-01-17T16:13:00.000-08:00

Yes, today is the day that I made a fine boo-boo. A shameful one, to be honest.

I was deploying config files for the monitoring software and got lazy - I overwrote files instead of moving them or renaming them...all because I wanted to get that last little bit done before I left for the day.

Of course it turns out that some esoteric command was in the files. Not mission-critical, but I'll be spending time in the backup software tomorrow (assuming the servers were ever backed up) trying to figure it out. Or re-writing the command from scratch. Ouch.

The amount of time it will take to rename the files versus restore from backup can be described in orders of magnitude. Honest to goodness - at least this project is almost over.

Lesson: Even if you are 100% certain you'll be okay overwriting a file - move or rename it for insurance, because restoring from backup is tedious and painful and sometimes an endeavour in panic.

Sub-lesson: Are you in a hurry to go somewhere? Are you simultaneously working on production systems? Just stop. Stop right now.

VMware Workstation, ESXi VMs, and external iSCSI storage

2012-01-11T19:08:00.000-08:00

From a post I made here: http://communities.vmware.com/thread/342742?tstart=0

It started out simple, then got longer, and ended a saga. The iSCSI saga.

I would paste the whole thing here...but I'd imagine the VMware communities will be around well after this article is rendered obsolete.

Bottom line: If your switchport is accepting tagged packets on VLAN10 only, and your physical adapter is tagging packets for VLAN10, setting the virtual machine networking (bridged to said physical adapter) to ALSO tag packets for VLAN10 will break stuff. Leave the VLAN tagging for the physical devices.

After the saga came to a close, I then tried to add the iSCSI targets to my workstation, but ran into this: http://www.jeremycole.com/blog/2007/11/09/the-iscsi-name-specified-contains-invalid-characters-or-is-too-long/

Yes, my naming scheme was foiled by a subtle Microsoft...oversight. This is what the Initiator does: (http://technet.microsoft.com/en-us/library/ee338474(WS.10).aspx)
Generates an event log if a target has a node name that is not valid. It ignores that target, and in some cases, all targets that are discovered with it (for example, if a target in the SendTargets node has an invalid node name, Microsoft iSCSI Initiator ignores all targets within the SendTargets text response).

Found a little more 'best practice' stuff here: http://www.cognizant.com/InsightsWhitepapers/vmware_esx_wp.pdf
Hostname Naming Convention
It is recommended that the hostname should not
contain an underscore. Labels cannot start nor end
with a hyphen. Special characters other than the
hyphen (and the dot between labels) are not
advisable, although they are sometimes used. (1)

One other tidbit: https://forums.openfiler.com/viewtopic.php?id=815
The underscore is not a legal character int he iSCSI spec and the MS initiator docs does say that invalid paths will be ignored. [sic]

I rescind my previous comment about Microsoft's insights, and I thank them for following RFCs.

So there we have it - it's an RFC oversight...or intention. At any rate, everything is working wonderfully, and if I didn't have underscores in the names, I'd even be able to access it from the host.

Good times!

VCP410 - take two

2012-01-11T19:06:00.000-08:00

Well, misreading a forum post led me to book my VCP410 re-take for end of February - around the same time big things are happening. So Lord-willing I'll pass it this time. I have already begun studying in earnest, but definitely need to put effort into it.

A number of people I've spoken to have said the '410 is focused on min/maxes, and after taking it last spring, I'm inclined to agree. Memorization has never come easily to me, so I'll really have to put all the spare time I have into this. I had tried to go on working experience and book knowledge and got a 221 - pass is 300/400 - rather embarrassing. Now, to be fair, I did not study at all, nor do any of the recommended lab exercises, so that probably didn't help. Now, why did I wait so long to do a re-take? The aforementioned 'big thing' and rebuilding our garage over the summer.

I've been speaking to my employer about potentially taking the VCP5 course/exam, but would not be surprised if they turn me down. Last year I had to pay for the course first, then after seeing that I'd already paid for it tried to (and got) me some restitution (2/3s of the course). Finances this year won't cover even the extra exam, so it's up to our Provider, now.

Not to mention I have yet to even look at vSphere 5, so I have lots of time!

Key areas I need to focus on are...well frankly just about everything. I really should look into v3 to v4 upgrades/licensing as well (actually doing, not just reading). Also need to really memorize the min/max PDF, and truly understand/memorize the FC/iSCSI pathing stuff.

Linux/CentOS templates in VMware ESXi

2011-12-15T16:52:00.000-08:00

I am sure there are other posts like this out there, but last time I checked, there wasn't much info on this, so here goes.

I want to use kickstart to deploy virtual machines. Rather, wanted. Forgot that kickstart requires ethernet right off the bat, and the vmxnet3 NIC doesn't work until vmware tools is active. I could use the E1000, but then I'm still having to delete/reconfigure after the fact. This method means VM rollouts that are probably just as fast as kickstart, and possibly a little easier. I could probably write a script to do the hostname/network cleanup as well.

Anyways, here's my attempt at vCenter templates for CentOS. This is pulled from my wiki, posted here for searchability.

CentOS vCenter template creation

Created a new VM and gave it 1cpu/1GB RAM/40GB thin drive.
Installed CentOS 6 x86_64.
I configured LVM - do so according to your wishes. (I specifically added LVs for /var/log, /var, /tmp, and /home)
Set passwords, timezone, etc.
Once done, I installed VMware tools, deleted temp stuff afterwards
Configured the networking and gave the template IP (designated in my IP list)
Shut down the VM
Removed the VMware tools ISO and floppy drive
Created a template out of this VM (convert to template)

One last item I left out - remove the ethernet adapter. Why? If you clone this VM, it'll keep the mac, and confuse the new VM. There is a process to fix it, but it might be less work to just add a new adapter.

Process for fixing a cloned ethernet adapter is move the ifcfg-eth0 file to ifcfg-eth1, edit the file and change the device name to eth1 and the MAC address to suit, then edit the IP to match the new VM. Change the /etc/sysconfig/network file to suit the hostname. Reboot the host.

A few items I left off are doing something with SElinux and iptables. For my lab I've disabled them and will eventually work on getting them functional. At the moment, that's low priority.

Openfiler FC documentation

2011-12-08T19:13:00.001-08:00

All done! Very excited, and I will continue to update with screenshots as they come up.

http://wiki.practicaltech.ca/index.php/Openfiler_-_iSCSI/FC_config

This doc will help you:

Set up Openfiler FC for the first time
Tear down and re-create new FC devices
Use LVS commands to create your back-end storage
Set up Openfiler iSCSI (though, admittedly, that is not the primary purpose)

Next up...back to the cluster project!

VMware I/O Analyzer

2011-12-08T08:47:00.001-08:00

http://labs.vmware.com/flings/io-analyzer

"I/O Analyzer is a virtual appliance solution by VMware that provides a simple and standardized way of measuring storage performance in a VMware vSphere virtualized environments. I/O Analyzer automates the traditional storage performance analysis cycle and reduces the performance diagnosis time from days to hours."

Going to test this out in the lab, looks really interesting - might be able to help with benchmarking how the Openfiler box does (or at least VMs running on the OF).

LVS - Adding the 2nd array

2011-12-06T17:13:00.001-08:00

Update: Wiki article here: http://wiki.practicaltech.ca/index.php/Openfiler_-_iSCSI/FC_config

I had set up the FC array with only half the disks, thinking I would just add a 2nd disk later. Once I discovered that Openfiler uses LVM, I wanted to just extend the current LV.

First, I wanted to move the VMs off this LV just in case, and of course that was no simple matter. I had no vCenter server set up, so had to do that (from scratch), then discovered that when I migrated the VMs from Workstation 8 into ESXi 4.1, all VMware Tools bits broke. Disks went offline, network stuff had to be removed/re-added, OS type reset to 'unknown', etc.

With the vCenter install, discovered a few key points:

To use a remote SQL server, you need to install the SQL native client 10.0 (gives you a 64-bit DSN).
If you set the vCenter to auth by a set account, like a service account, you lose AD logon for other users.
Firewall! Don't forget about me! I just turned it off for now, but will be re-enabling once I have time to do a wiki article on it.
This guy has instructions for a scripted DSN config: http://derek858.blogspot.com/2010/08/automate-vcenter-41-64-bit-dsn-creation.html
This also came up: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001046
And finally, don't forget that it requires the .NET framework component installed (default option under the 'application server' feature).

Ok...vCenter done and VMs moved to the SATA array on iSCSI. Performance for the storage vMotion seems pretty good.

The LVS troubleshooting I've placed on the wiki. Quickly before I left this morning I tested things out - and all is good! Now have a single FC target that is 516GB! To be honest, I'm not sure that's a good thing - I will test things out again tonight by breaking the LV down and setting up multiple smaller targets, say two per RAID card. You can also do mirroring...maybe I could do two smaller targets mirrored across two arrays? Lots of fun stuff.

Subnet change for the lab = fun times

2011-12-03T20:58:00.001-08:00

I moved the Openfiler box and first ESXi host back down to the rack (off my desk), and so had to do a subnet change. So I changed things over (I thought) and tried to access the wiki. No go. Go and re-check all the CentOS box networking for wiki-01 and mysql-01 - seems ok. Checked more things out - now the pages load, but super slow

Well, after an hour of troubleshooting (this was a silly mistake), I discovered that it was DNS-related, and the ifcfg-eth1 file on mysql-01 had a DNS line!! Probably from the initial deployment, but shouldn't have been in there - I keep all the DNS stuff in /etc/resolv.conf.

Anyways, fixed it up and everything is back to normal. It is amazing how quiet things are without that!

Oh yeah, also had one of the IBM 73GB SAS drives fail on me (state in RAID BIOS reported as 'foreign').

I also discovered I know nothing about FC switching, or rather that FC switching is not automatic like Ethernet! So for now, the FC will be direct from Openfiler to ESXi host.

Tired. Bed.

Openfiler project

2011-11-29T18:06:00.001-08:00

As part of the lab rebuild, I am setting up an Openfiler box.

Some features I am using:

iSCSI block-level storage with path redundancy
FC block-level storage
Block-level replication (potentially have a 2nd host)

As I've been setting it up, I'm realizing that it uses LVM for storage which is kinda nice as it dovetails into what I've learned from the clusters at work. Also realized that fiber channel has way more curb appeal than iSCSI. I've not had a chance to integrate the FC switch yet, but that's on the list. What I have done is document (thanks to the internet and some testing) the steps involved in setting up/giving new access to the target and new servers. For the price, I'd say you can't go wrong with this setup for a home lab.

Openfiler has a 4-port FC card ($80 per card)
Each ESXi server has a 2-port FC card ($40/per card)
FC cables from monoprice ($10/each)

Now, the motherboards I am using in the ESXi servers do not have PCI-X slots, but being bus-limited to 133MB/s is probably the least of my worries. The Openfiler box has 133mhz PCI-X (~1050MB/s) slots, so no loss there. The alternative would be to use iSCSI MPIO, but I'm still limited to PCI slot speeds, as the other two slots on the ESXi boards are still PCI (two Intel PRO1000 dual port cards).

At any rate, I am running both FC and iSCSI from the Openfiler (SAS over FC, SATA over iSCSI), so there are two methods of accessing VMFS datastores.

The wiki has been updated with my install documentation - more to come......

Openfiler - Errno 104 (conary updateall)

2011-11-29T17:21:00.001-08:00

If anyone is getting this error:

/usr/lib64/python2.6/socket.py:381
error: [Errno 104] Connection reset by peer

It is probably caused by a proxy or transparent proxy blocking access. In my case it was my Astaro firewall - I had to add a 'Web Filtering/Exceptions' rule for the Openfiler host. I set it to allow everything, probably safe - no browsing will be done, just the conary updates.

I also found this bug on the Rpath site (even signed up to report!) from 2009, and the dev provided the proxy as a point of interest. ( https://issues.rpath.com/browse/CNY-1958 )

Once the fix was in place, I decided to run the Update utility from the web GUI (which had also failed before), and it provided me with errors for about 5% of the packages to update, although the rest installed correctly.

Service accounts & domain admin privileges

2011-11-16T15:53:00.001-08:00

Over the last few weeks I've had a good couple of lessons around service accounts and domain admin privileges (and who should have them). What came to mind was a kind of cascading failure caused by not following best practices.

Management team finally authorized the changes the windows admin had asked for - the removal of all 'regular' users from the Domain Admins group along with the creation of 'admin' accounts for people that required them.
Users were removed from Domain Admins group. Windows admin did not communicate this to anyone. Management did not communicate to users that this was going to happen.
Random things began to break. Small in-house-programmed websites stopped working, workflows were disrupted, ticket queue built up, etc.
After spending a lot of hours trying to figure out why these things were breaking, someone happened to mention that 'oh, admin removed domain admin privs for everyone'.
Light bulb.
Confirmed that each and every issue was caused by this change.

Further digging revealed that the custom sites had been programmed to just take the logged on user's account info. The sites would then access AD with abandon - something that only Domain Admins can do (in this case anyways). No Domain Admin, no AD access.

This all stemmed from two basic best practices for AD:

Never give your daily user accounts domain admin rights.
Always create a service account when an application or service requires access to AD authentication. Assign the service account a REALLY complex password and set the password to never expire. (In stricter environments this would probably not fly, but for the majority of companies this would be secure enough)

So the lesson is follow those best practices as you are able to. Sometimes management will rather give everyone freedom and accept the risks involved rather than lock things down, and you're stuck. That's okay, you've brought the issue before them and they can make an informed decision - it's their job to make these decisions! Some would harbour resentment (to some degree natural) with such a decision, but there is no point in holding a grudge. Bring the issue up as often as you can, and maybe the squeaky wheel will get the grease.

Apache clerestory

2011-11-08T18:24:00.000-08:00

As I get going with all this, it's becoming clear I should have a dedicated Apache box. The wiki box is the obvious example, but might as well start fresh so there's no wiki nonsense buried in there. Will also give me a chance to document the migration of sites from one server to another.

Further, because we're hardcore here, we'll be doing an Apache CLOISTER. I mean clerestory. Cluster.

From my day job I'm reasonably familiar with clusters, and I'll transfer my wiki info from there to the PTC wiki (yes this is okayed by them). Obviously info will be sterilized, and frankly a lot will change since I'll be running through the entire process and making corrections/addendums, so there.

Clusters. Uptime. Fo sho.

I am not making this up. The more you know!

Reasons why the wiki is down:

2011-11-08T12:34:00.000-08:00

Software issue (service crashed).
Hardware issue (like CPU broken).
Internet is out.
Power is out.
Meteor.

Update: In this case, option 3. Out for an hour. I should really call Bell and see what the heck.

Core i7 failure

2011-11-07T20:15:00.000-08:00

Well, there's a first time for everything. Actually had a CPU go bad on me. Thought it was mobo for quite some time, didn't even consider CPU failure as an option. Thank the Lord I had the spare lab box with another Core i7. Little upgrade I guess...i7-920 to -950. However, the CPU has a code that indicates 2008 manufacture, so might be out of warranty.

I'll be calling Intel tomorrow, so praying it'll be covered. If not...no cheap to replace!

For anyone interested, the symptoms of the failed CPU are just this: System will no longer post, or show any video. For all appearances it looks like a dead motherboard, or perhaps dead PSU.

update: Ah, was purchased in July 2009, so should be okay.

Constructive update:

Decided to post my troubleshooting process just in case someone isn't sure:

System BSOD'd.
I noted the BSOD error and codes via photograph, just in case they would help. No drivers were mentioned in the BSOD, so chances are it's hardware.
Once the memory dump completed (12GB of RAM takes a bit to dump), I reset the computer using the reset button.
The PC would no longer boot - no video; monitors went into powersave, but still indicated they were connected.
Immediately guessed motherboard was the issue, proceeded to colour my investigation with that assumption.
Disconnected power and removed the SAS card, tried booting again. Same symptoms.
Disconnected power and removed all PCI cards (NIC, GPU, SAS), tried booting again. Same symptoms.
Got frustrated.
Disconnected power and removed all RAM sticks, SATA cables, changed out GPU for a known good GPU, tried booting again. Same symptoms.
Disconnected power and removed motherboard/CPU/HSF from case, attached to another power supply from another system (PSU known good), tried booting again. Same symptoms.
Srsly. What.
...
Ok, maybe CPU is bad. Hahaha, silly notion.
Took 'spare' CPU from lab server and swapped it out, installed Intel OEM cooler for ease of use, tried booting using the known good PSU. System boots.
So a different CPU on the same motherboard allows me to boot. Therefore, the CPU is preventing the computer from booting (POSTing).
Contact Intel for warranty replacement. (which was handled like a consummate professional should - you go Intel)
It arrives the next day. Send bad one back to complete warranty.
Put replacement 920 into lab server because the 950 is prettier in my PC anyways.

Synology NFS 'access denied' and resolution

2011-11-05T20:38:00.000-07:00

Fun time figuring this out, I was being bad and not documenting, but here's what I recall:

Kept getting 'access denied by server while mounting' errors when using this command: mount -t nfs 10.0.0.14:/volume1/mysql_backup /srv/backup
Checked and re-checked the Synology settings to no avail. Thought it was something to do with root squash - was not.
Correct settings should be correct IP address, RW, No Mapping, Enable Async
SSH'd in to the Synology and after some messing about with /etc/exports, I set up tail -f /var/log/messages
Took me a while to notice it, but the IP it was registering was the Astaro gateway IP - the Synology and my PC are on different subnets!
Set the NFS rule to '*' and started working immediately.
Firewalls make it easy to overlook simple things. I imagine there is some sort of fancy NAT rule for the NFS traffic that would allow specific IPs, but seeing as how I'm technically behind two firewalls and this is a lab, the allow all rule will work for the time being.

If I were setting up a production environment and/or not using a NAS for my backups, I'd not leave such a wide security issue open. The issue is noted here, and maybe I'll fix it at some point, just to find out what's involved in getting it to work.

Fun Synology stuff:

/usr/syno/etc/rc.d/S83nfsd.sh restart/stop/start
cat /etc/exports

Reference: http://forum.synology.com/wiki/index.php/Basic_commands_to_get_around_the_Synology_Box_using_the_CLI#nfs

Yup.

Moving mysql databases into a central mysql server

2011-11-05T11:20:00.000-07:00

Today I'm working on moving the wiki database over to the central mysql server.

I still have to work through implementing best practices and whatnot, but everything is functional to this point:

Wiki database is being served from the central mysql server.
Central mysql server has the mysql database directory housed on a secondary disk.
Documentation is available that should enable moving other applications over to the central mysql server.

Yup, making progress. http://wiki.practicaltech.ca/index.php/Mysql

Next steps:

mysql backups are a necessity now, so have to configure that - could involve some scope creep as I have thought about setting up a backupPC server
Syslog server, also attached to central mysql server
NagiosXI server, again on the central mysql server, and will also require postgresql

vCenter server is on my priority list as well, but we have to wait on sorting out the hardware issues, and document the iSCSI Openfiler setup. I've finally moved the mail server off the last physical ESXi host, so Openfiler is next!

My desktop has now assumed the role of OrbitalHQ as it is running:

The only DC in the domain at the moment
Linux workstation
Mediawiki server (i.e. apache)
mysql server
syslog server
Exchange 2010 server

Yup.

Quick update - mysql & SNMP

2011-11-03T15:01:00.000-07:00

Last night I also got a mysql VM and syslog VM up. After the wiki issue, I figured now was a good time to get the database off the wiki server and onto a dedicated mysql server. That server will then back up onto the NAS - something that will help me sleep better at night. The dedicated mysql box is also pretty key for a number of other Linux applications.

I am not certain yet if I should be running postgresql and mysql on the same VM (even though there will be separate VMDKs for DBs, logs, temp, etc (wrong terminology??)). At any rate, once that's ready and documented, I can really get cracking. Can't wait to get the Nagios box online as well.

Speaking of which, I've learned a lot about SNMP lately. The process from trap to Nagios alert is quite involved:

Trap to Nagios host
snmptrapd receives trap
snmptrapd formats trap as per snmptt
snmptt logs the trap and looks up what happens
snmptt runs the EXEC line which submits the result to Nagios

I'd have to consult my documentation for more detail, but that's the gist of it. Also learned some handy SNMP troubleshooting tips:

Use the snmptt debug to get a clear picture of what's going on
If you get 'unknown' traps logged, chances are your MIB is outdated or not present
Ensure logging is turned on and log directories are correctly specified
You can use the net-snmp-tools to manually send a trap from your workstation

Yep, good stuff. I'll add an SNMP section to the wiki at some point.

Anyways, next step is configuring and documenting the mysql/postgresql VM.

Wiki down and up

2011-11-02T17:07:00.000-07:00

It was down for two reasons:

1. My PC is a new Win7 image and I'd forgotten to strip Windows Update of any reboot privileges. (it rebooted, halting the VMs)
2. When I powered the VMs back up, SElinux turned itself back on, blocking access to the mediawiki files.

Lesson: Either fix SElinux or turn it off. Since this is internal, and I have other stuff to do, we'll turn it off. Hosts directly facing the outside should have SElinux operational and properly configured.

Now, to finish planning all of this out.

I decided to move the Exchange 2010 box as it's 'sort-of' live (accepting junk mail at the moment), and it is the last running VM on the old 'big' ESX host. Once it's moved (250+ minutes to go - thank you ESX file-transfer speed restrictions) I can install Openfiler or whatever I decide on going with and get that tuned up. Only downside to that is I'm still using the one SAS bay in the desktop, but that's not critical, I suppose.

330 minutes to go.

Wiki update - Scope and Build Order

2011-10-22T19:18:00.000-07:00

I got a little sidetracked...I think I need a project flow document or something. Of course, I was sidetracked playing Minecraft....hosted on a VM here, no less - something else I'll end up setting as an available app.

The next wiki entry is going to focus on the order I'll be building the lab in, what milestones there are, etc.

http://wiki.practicaltech.ca/index.php/Lab_Build_Order
http://wiki.practicaltech.ca/index.php/Lab_Build_-_Project_Scope

Wiki integration into the blog

2011-10-21T18:15:00.000-07:00

This weekend I will focus on getting the wiki integrated into the blog so I don't have to deal with copy/pasting and formatting of the articles.

Ok, wiki is up, read-only. I'll probably install Lockdown or something to further help, but such low traffic, not a priority.

Cool!

As a side note, I'll be adding a wiki entry on setting up the NATing to get through.

First steps for the new lab

2011-10-19T18:46:00.000-07:00

First task is to install a Wiki server to hold my documentation. It will probably end up here as a 2nd page or something.

I will need to start making choices soon about when to blow away the old domain as I need the name.

The wiki is up, and the first two pieces of documentation are done - MediaWiki 1.17 install and config and CentOS 6 install/config.

Ew...formatting...definitely have to do something soon.

1. Create new VM,

a. Do not set aninstaller (it auto-installs the GUI)

b. Give disk as20GB, single file.

i. Note that loggingwill eventually all flow to the syslog server, so no need for a huge amount ofspace.

c. Assign an IP anddocument in the spreadsheet.

d. Choose the ‘basicserver’ spec.

2. Set up VMwaretools

a. Mount /dev/ /mnt

b. Copy thevmware-tools file into /tmp

c. Un-tar it

d. Run the .pl file- default install stuff.

3. Configure thenetworking

a. Change the DNS ifnecessary.

b. Set onboot toyes.

c. ifup etho

4. Set up theRPMforge repo.

a. http://wiki.centos.org/AdditionalResources/Repositories/RPMForge

5. Set up the user

a. Adduser chris

b. Passwd chris

c. Vi /etc/group

i. Add chris to sshdand wheel

d. Visudo

i. Uncomment wheel –all

MediaWikiinstallation/configuration

1. Wget http://download.wikimedia.org/mediawiki/1.17/mediawiki-1.17.0.tar.gz

2. Yum –y install php mysql-server httpdphp-mysql

3. Change the httpd.conf file(/etc/httpd/conf/httpd.conf)

a. Point the root dir to/srv/mediawiki/mediawiki-1.17.0

4. setenforce = 0

5. stop iptables

6. Connect to the server/index.php and runthrough the media

7. Save the LocalSettings.php file into the rootdir.

New lab design and direction

2011-10-18T16:52:00.000-07:00

After a summer of working on the garage and generally ignoring the lab I am back into it.

I've decided that my initial goal of having a VM sharing storage for the other physical hosts won't work, hardware is limited and, I must be honest, I think that it's pretty much impossible.

So...I'll be moving hardware around, making some sacrifices, and getting some more hardware. Initially I thought that I'd have to get a dedicated Openfiler box, but after pricing it out, and thinking some more, I have a better way.

The 'main' ESXi host (24GB i7 machine) will change into an Openfiler box. 24GB of RAM will go into my desktop (also i7) which will run all of the 'workstation' VMs in VMware Workstation. My 12GB of RAM currently in the desktop will go into the Openfiler machine. The RAID cards will stay in there, along with all the disks. I will, however, scrounge up a boot volume to add so I'm not relying on the arrays for the OS. It will then serve iSCSI to the three Core i5 ESXi boxes.

Each of those boxes has 16GB RAM, so plenty of memory space for the VMs I have planned.

I will be completely blowing away the old environment and starting fresh.

VM list
•nagios XI
•wsus/print
•apache
•iis
•dc1
•scom
•dns/winbind/ntp
•repo/samba - iscsi
•vcenter
•dc2
•mail relay (postfix)
•hub transport
•vMA
•syslog
•SQL Server
•mysql
•sharepoint
•exchange

(not in any particular order)

Notable new features
Nagios XI monitoring. We are a Nagios client at my employment so I asked for a lab license that will cover this lab - they generously acquiesced and gave me a 50-node license for lab use! I will be running this alongside a SCOM deployment for comparison sake.

SSL. I will be trying to use SSL properly wherever possible in this lab. I find it's a task that's becoming more and more common in the workplace for me, so experience in the lab is key. A wildcard cert might be a bit much though - even the cheapest I've found is $120 per annum. If we can swing it, I'll try a year.

Linux. I have been working with Linux-based hosts a lot lately at work, so this new environment will reflect that. I think there is a place for Linux and a place for Microsoft, and lab experimentation will help me determine the balance.

Sharepoint. I've worked a little with Sharepoint, but since it seems to be a significant part of Microsoft's future, I should probably get some more experience in it.

Exchange 2010. This will be a full deployment using the new cluster features (DAG), hub transport, and other stuff I still need to learn about.

Why so complicated?
A few of the items on this list could be done on my Synology, or in one VM, but I've learned that a good best-practice is to separate functions whenever possible, so why not here? Why is this lab environment so complicated? Simple answer: In real environments, things are complex more often than not. Learning to deal with a complex environment will help keep my mind sharp and open to the random stuff that can come up.

Another point of the complexity is to develop my own documentation for each task's deployment. Documentation helps me learn by getting my thoughts down into writing - a task that usually then makes flaws or ideas clearer. I'll be posting the documentation here as it comes. A complex environment means lots of documentation!

Timeline right now is end of November will be the deployment. Why November? Black Friday sales, that's why. Getting an SSD for the desktop to help with running VMs locally.

More to come.

A tale of migration and miscommunication

2011-09-27T14:34:00.000-07:00

Before this last week, I considered myself reasonably competent and able when it came to handling myself in a professional manner. As the migration I've been working on wrapped up, I was sharply reminded that I have much to learn.

Planning and research can only take you so far - communication and experience are strong factors in successfully pulling off important (any) projects.

First of all, this is not a doom and gloom scenario, a lot went right. We're just going to focus on what went wrong as that's where the good lessons are.

Did not thoroughly check documentation of LOB software to ensure all the pieces were there to migrate it successfully.
Did not communicate with LOB developer/support (same person in this case) to ensure they would be available for emergency support during the migration.
When an issue did arise, used email as first line of communication.

So, let's deal with #1. The documentation was asked for when the software was written a year or so ago. When I initially went through it, I only noticed the client install, but figured the server part was at the bottom - it was just a database hosted on a SQL express instance running on the server, so wasn't much to it. Had I actually been looking for specific server-side installation instructions, I would have realized there was no guidance at all for that. That may have prompted me to try and set up a test instance in my lab - where I would have realized the problem.

Next up, #2. Assuming I'd done #1 properly, I would have contacted the developer right away, and not dropped a bombshell in his lap. To be fair, it was a pretty simple problem with an easy resolution, but an emergency none-the-less. By not contacting him prior to the migration I opened myself up for a potential show-stopper - what if he had been on vacation for three weeks? (I would have finagled the old server's SQL instance into life, but that's not the point)

Finally, I dropped the ball by again assuming he checked his email regularly (with an unhealthy dose of avoiding speaking to people). According to office staff, he did, so I went there first. My general instinct is to email first, as I've tended to get better response from technical people via email rather than phone. However, you can always get much more prompt attention when you're on the phone being a hassle. He responded right away on the following Tuesday morning when one of the office folks called him. We're still not sure why he did not respond to emails, but either way - email should never be the first line of offence when trying to resolve emergency situations.

Lessons learned:

Take time to run through each step of the migration. Do not assume documentation will give you all the answers. For mission-critical applications, KNOW that they will work, don't guess so.
Notify any parties with a stake in the operation about the event in question. Not via email, but by calling them so they KNOW something is happening.
In an emergency situation (should one arise) - USE THE PHONE. Email should only be used in emergency situations to send helpful information, not convey the initial problem.

There is a final lesson in all this. When estimating your time for anything with 'migration' in the title, double your estimate. If it involves tools you've never used before, quadruple it to cover learning time. Even if you end up far below your estimate, your client will appreciate it. Further, do not attempt to rush a migration - you will end up sorry you tried to beat the clock for convenience sake. Always wiser to just back away until you have enough time.

Hope this helps someone avoid my mistakes.

If you're curious how the rest went, I'll be doing another post on the merits of migration helper kits.

Making a case for a purchase to management

2011-03-30T12:33:00.000-07:00

An issue I've come across many times now is how to properly present a purchase request to your manager/his boss. There are occasional managers who will just take your word for it, but some (hopefully most) will question the need for it. As a technical person, I find it really easy to get caught up in what something can do rather than why we need it in the first place.

Case in point.

An upgrade was required for some KVM servers - more RAM required - and I put through the request for purchase to my superior. He sent it over to the director, who immediately shot back an email requiring a really nice reason to approve it. After some futzing about via email (I was out of the office that day), it became clear that he was fine approving it, but we needed to make the case as to WHY he should approve it. Frankly we needed to prove it to ourselves first, but he was technically-competent and was not just going to stamp everything 'a-ok'.

There was a clear reason to me and my co-workers why the RAM upgrade was needed - existing VMs were using 10GB, and only the primary KVM host had enough RAM to run them. If it needed to fail over to the secondary KVM host, bad news bears. If we needed to use the backup host, bad news bears. So two servers needed upgrading from 4GB to 16GB.

Wait, you say - 16GB? I thought you said 10GB.

*Blank stares from us.*

A nice stereotype we've set up for ourselves, eh?

What we should have said was:

CentOS requires 1GB overhead.
VM1/2 require 3GB each, thus 6GB.
VM3 requires 1GB.
VM4 requires 2GB.

Total of 10GB just to function at present levels. Now, to be prudent, we should never run at more than 80% capacity - this means we need a minimum of 12GB (note these are all 2GB sticks). Furthermore, to allow for future expansion, we should set ourselves up at 60% capacity, thus 16GB total.

Can you make a case for future expansion?

Yessir, we want to virtualize Physical1 and Physical2 in the near future, each requiring 1GB, bringing the total up to 12GB to function, 14GB as a minimum, and 16GB is a nice medium in-between.

Instead of spending a morning emailing back and forth, one tidily summarized email of the above could have resulted in a single 'approved' email, and less wasted time all around. I will strive to put this method into practice.

As a side note, the more money you want, the more research/proof you need. This was a relatively inexpensive venture, so one proper email would have sufficed. If we're talking a SAN expansion...better make sure you have your ducks in a row and every reason accounted for.

Lab setup 101

2011-03-16T19:53:00.000-07:00

Lab setup
I finally finished the cabling and BIOS finagling on the main ESXi box. Trying to run 20 SATA cables neatly in a small space is tricky business, I tell you, but in the end it turned out clean enough that the wiring should not restrict air flow too much.

The motherboard is a Supermicro X8SAX. RAM is Kingston ValueRAM (6x4GB). CPU is an i7-950. For storage I have two PERC 6i cards and one PERC 5i (actually an LSI8408E). The 6i cards each have eight 2.5" 15k 73GB SAS drives in RAID10 and the 5i has four 1TB 7.2k SATA drives in RAID10. I will need to revisit the cooling inside, as the RAID cards get quite hot. Right now I've slung a fan behind them blowing out of the case.

ESXi is running off of some Verbatim USB drives I got from newegg ('clip-it' model). I have just finished 'Maximum vSphere' by Eric Siebert and he gives a really cool method of creating the OS on these drives. Instead of installing manually, you use Workstation, connect the USB drive plugged into your PC into a VM, attach the ESXi ISO and install onto the USB drive. Once done, power the VM off, unplug the USB drive, rinse and repeat. Brilliant! I must send Eric a note letting him know how excellent that trick is.

Also...the rest of the book is just as good - I highly suggest picking it up. A very different read from books like Scott Lowe and Mike Laverick have written for vSphere. Those books are also excellent...but I digress.

Anyways, I set my BIOS to use VT-d and 'SRIO' or something like that. Why not, right? Cool features I might want to try out.

I booted up the system (after an hour of sorting out why I could not get video), configured the ESXi management network, and connected from my PC using the vSphere client. Right off the bat I noticed that the PERC 5i did not have a datastore automatically configured (the 6i datastores were present...although one was 5GB larger than the other... :S ).

I checked the hardware health status, saw the card present there. I even checked the HCL just to be sure - the 5i/8408 is still compatible to 4.1.1 - and then tried to add storage. The drive was showing correctly (1.82TB)...but something was amiss.

I ran across this doozie of an error when I tried to add a datastore using that controller:
"Call "HostDatastoreSystem.CreateVmfsDatastore" for object "ha-datastoresystem" on ESXi "192.168.21.30" failed."

Once again, Google came to the rescue. I searched for that string but sans-IP, and came up with a number of dead ends (for my instance). One last item remained - a link in German. I figure, hey, Google translate is at least functional, why not?

http://armin.langhofer.at/?p=110

Go ahead, translate it!

The gist of it says that turning off VT something resolves the issue (Adaptec for him). I figured that VT-d is the only option I enabled that was out of the ordinary (and that SRIO thing), so I disabled both. After rebooting, I could now see why the drive had previously looked amiss. On the 'add storage' page I could now see more volume details (I should take screenshots!), two more lines below the naa and such. Adding storage now worked perfectly.

Lesson: VT-d can mess up some cards! Thanks Armin!!

Perhaps this will be patched at some point. It will be interesting to see if the other hosts have the same issue, albeit different hardware.

Lab update

2011-03-11T10:06:00.000-08:00

Criminy, I've never had so many shipping boxes arrive in the course of a week. Cables are out for delivery today, I'm hoping we can get home in time to catch the UPS driver on his last round, as my wife did yesterday for the SAS drives.

Speaking of which...the 2.5" SAS drives are SOLID little things. Lovely. I got them all mounted in the IcyDock enclosures and only broke one LED indicator...the manufacturing on the drive sleds is not perfect - but for the price, I'm not complaining. I will send them a note to see if I can order a spare sled (I have two spare drives after all).

Support chat was very helpful - I can do an online RMA for the unit. Also, they provided me with the link for an individual tray: http://www.icydock.com/goods.php?id=125 Unfortunately the only place to offer it is amazon.com (does not ship to Canada) and one tray is $40. I'll pass!! Five or ten bucks would be ok...but $40 is nuts. Have to see about the RMA then.

I need to check and see if the PERC can do staggered spinup...pretty sure it can, but if not, I'll need a different PSU. When I had the PERC 5i running the 8x1TB array, I could hear them doing staggered spin-up, so pretty sure we're ok. Google indicates the same. I also checked the PSU I'll be using (of course AFTER purchase) - PPCMK2S500 - and the rails will be plenty, even if there is no staggered spin-up option.

This weekend I will spend a bit of time mounting things in the rack, and getting the hosts installed in their cases.

Lab - Bright news! FC addition

2011-03-08T15:19:00.000-08:00

Update: More Arstechnica goodness! Brocade provides free e-learning: http://www.brocade.com/education/course-schedule/index.page

See Server Room thread here: http://arstechnica.com/civis/viewtopic.php?f=21&t=1138681&p=21409635#p21409635

Thanks to the good people in TheServerRoom@arstechnica, the lab has a wonderful new addition - FC! I have managed to acquire the following for a very low price:

Brocade 3800 FC switch (2Gb, 16-port w. SFPs, all licensing)
QLogic QLA2344/2342 HBAs (1 quad/4 dual, PCI-X)

Very exciting! I have very little hands-on experience with FC, and seeing as how it's an industry standard for serious storage, I should probably fix that. Thus, extra connectivity!

I will be using the trial version of the Open-E VSA to start, as it can simulate an FC SAN. I think the FalconStor VSA does this as well. The quad FC HBA will go in the i7 ESXi host, and each of the i5 hosts will get a dual FC HBA. Of course, this brings me down from seven ethernet ports per host to five, but the added option of FC is worth it. Besides, configurations are not set in stone.

Cables will be chunked in to my monoprice order - which I need to plan out/order already.

One bummer I discovered today - the PERC5i will not be going in the i7 host - the x8-sized slot is actually only x4 electrically. Pretty sure it won't work in that config. Hopefully the ICH10 RAID will be seen by ESXi for datastore usage.

Lab - Datastore & iSCSI config

2011-03-07T09:38:00.000-08:00

Physical layout
FastArray01 & FastArray02

Disks: 8x 2.5" 73GB 15k SAS (IBM)
Card: PERC6i w. BBU kit (PCIe x8)
RAID level: RAID10
FastArray01 size: 292GB (raw)
FastArray02 size: 200GB (raw)
FastArray03 size: 92GB (raw)

LargeArray01 & LargeArray02

Disks: 4x 3.5" 1TB 7.2k SATA (WD)
Card: PERC5i (PCIe x8) (will try to get BBU working)
RAID level: RAID10
LargeArray01 size: 1500GB (raw)
LargeArray02 size: 500GB (raw)

Logical layout
Storage service

iSCSI provided over two Intel PRO1000MT quad NICs
Local storage (FastArray03 & LargeArray02) for home VMs

VLANs

VLAN50 - for LabSite1 iSCSI storage
VLAN60 - for LabSite2 iSCSI storage
VLAN 200 - for 'home' iSCSI storage

iSCSI notes

iSCSI target for each array
CHAP security

VSAs to test

Open-E DSS v6 (iSCSI and FC)
HP Lefthand P4000 (iSCSI only)
FalconStor NSS (iSCSI and FC)
NetApp ONTAP8.0
EMC Celerra
NexentaStor (FC plugin available)
Starwind (via 2008 R2)
Openfiler
Solaris ZFS

Tests to run

SQLio
Random/sequential transfer speeds
?

Infrastructure

2011-03-07T05:21:00.000-08:00

Thank the Lord the back door was open at the office. The building manager had emailed me Saturday informing me that my card would not work for the back door on the weekends. Thankfully he was incorrect! What he did not mention (I guess he deemed it irrelevant) was that the door between the lobby and rear hallway was locked (physical, not electronic), so the elevator was no longer an option. This meant carrying everything to the back door from the basement. It actually was not that bad, but I was doing some fine huffing and puffing by the end of it.

Two-post rack
Two rackmount shelves
Two PDUs (1U and a 0U)
Two SUA3000RM2U
One SUA2200RM2U
Three RBC43 packs from the aforementioned UPSes

Not sure which of the UPSes I'll be putting to use. I have to tally up the equipment and see what kind of load I'm generating. Either way, I'm putting in a new circuit dedicated to the UPS - either 30A or 20A. Good thing the rest of our house is wired so awfully (our entire upper floor is on one 15A circuit - pretty sure code has something to say about that).

The rack has a 'shelf' on the bottom U...hard to describe, kind of a platform that forms the base of the rack, but also is level with the bottom U. Anyways...that will need to be extended about 12" to fit the UPS, unless I'm satisfied with the weight distribution - APC has kindly put all of the weight in the front 2/3 of the UPS.

When I rack it all, it'll look something like this (starting from the bottom):

UPS
Main ESX host (w. "SAN")
ESXi-03
ESXi-02
ESXi-01
3U space for another host
Linksys SRW2048
Firewall (1U)
Lots of U
1U PDU
Dell 2708
Shelf w. Synology, few other items.
Lots of U
Top few U will be the other shelf with the WLAN, etc.

Something like that anyways.

Lab - SAN results

2011-03-06T12:51:00.001-08:00

Well, decided I had wasted enough time hemming and hawing on what to do for lab storage, so it's done.

A VSA-style storage setup will offer more flexibility, but since I want this host to be somewhat permanent I've decided to go from just adding storage to two ESX hosts to a dedicated 'perma' ESX host. This host is spec'd with the following:

Core i7-950/24GB/Supermicro X8SAX-O (lots of PCIe and PCI-X slots)
iStarUSA D-400-6-ND case (6 external 5.25" slots)
PC Power & Cooling 500w PSU
4x1TB WD Caviar Black (RAID10 for my larger datastore)
4x IcyDock MB994SP-4S SAS/SATA hotswap bays
2x PERC 6i RAID cards w. fanout cables and BBU kits
16x IBM 2.5" 73GB 15k SAS drives (two RAID10 arrays for the 'fast' datastores)
2x PRO1000MT quad NICs (PCI-X)

This host will run my 'home' VM environment and will never be subject to wiping or messing about. Kind of a lab 'production' environment for VMware and Microsoft lab work. The lab 'lab' environment will be the other three i5-based hosts.

It's costing more than initially anticipated, but it should last a long while.

And a final note on SSD - I was not prepared to deal with the entire RAID set dropping and requiring replacement, especially at $180 a pop.

Tonight I pick up the rack, PDUs, and UPSes!

Lab - SAN options part 3

2011-03-05T12:15:00.000-08:00

I've realized that the last two entries on this don't really explain what I need and why. So here goes.

The next few years I will be heavily dependent on having a lab at home for study purposes, 'just messing about', and for my part-time work.

Thus, while I now have hosts for virtual machines, handling the RAM, CPU, NIC duties, I now need some back-end block-level storage (NFS can be done in a VM).

Requirements

Decent IOPS & throughput
iSCSI connectivity with MPIO being a 'nice-to-have'
Allowing for separation between 'sites/datacenters'
250/500/1000GB of space (min/ok/max)
Resilience of some manner to disk failure
Cost no more than $2500 all-in
Be reasonably power-efficient

A few notes on the above.

MPIO would be nice as it is a common thing requiring advanced configuration in ESX deployments, it would also speed things up quite a bit.
iSCSI connectivity is mandatory, as FC would cost way too much (yes I just spent the last 20 minutes researching 'or is it...?' and yes...it will*).
I am most likely asking for the impossible...or what I really want is a SAN.

*FC may be possible by purchasing HBAs and a switch, then attaching the HBA to a VSA. Definitely going to look into this.

Yup...next step is to find out if the VSA can see an FC HBA and put it to use. If that is possible, there is absolutely no reason why I should go with anything other than VSAs. However, I may decide to go with a dedicated VSA box - none of my boards have PCI-X, and all the HBAs I'd be looking at would require PCI-X.

Lab - SAN options part 2

2011-03-05T09:19:00.000-08:00

I was doing some work today using Workstation backing onto the Synology, and MAN is it slow. Six VMs are running, but only two or three are actually using any disk (updating the template, and installing SBS on another). Now, this is of course running off of the Microsoft iSCSI initiator - no VMFS, so we'll let it slide until the lab is functional (picking up rack stuff on Sunday, cases/PSU arrive from Newegg early next week).

Here is another option that I have been toying with:

Use VSAs exclusively
There are a number of VSAs available, and it would make some sense to use it just for vendor experience.

With regards to hardware, I have a few options here as well. I can have each ESXi box host a local datastore that the VSA would then serve from. No idea on performance, but I would imagine if the back-end was fast enough the overhead could be mitigated somewhat.

Hardware
PERC 6i w. BBU -> SATA breakout cables -> IcyDock 4x2.5" hotswap bay(s) -> disks

The disks are yet another option: 2.5" SATA, SAS, or SSD In the following examples, I'm assuming the SAS has an 8 disk RAID10 and the SSD has a 4 disk RAID0. Sounds unbalanced, but here is the reason: The SAS setup needs the RAID10 and spindle count to match the SSD for performance (even then, probably will not be as fast). The SSD is a four-disk RAID0 to match the SAS setup for space. Even then, we're only talking 250-300GB datastores...not a huge amount of space, but it should be enough. To further add a point to the RAID0, if a disk fails it will give me some nice experience in recovering a datacenter from backups. Ok, lame point.

DiskOption: 73GB 2.5" 15k SAS
SubTotal: ~$2000 (if I remove the two spare disks, prices are identical)
Price per GB: $6.85
Est. cost per annum: $120
Raw array size: 292GB
Pro: Resilient to disk failure. Lots of spindles means potentially better sequential performance (good for vMotion?). ~40GB more space than the SSD array. Two spare drives included in price.
Con: Disks are used/older. Less IOPS than the SSD array.

DiskOption: 80GB SSD
SubTotal: ~$1885
Price per GB: $7.37
Est. cost per annum: $41
Raw array size: 256GB
Pro: Super duper fast (probably will max the RAID card and bus, not to mention the iSCSI VLAN). Only need one dock per ESX host. Will use less power than the SAS array. New drives.
Con: Cost per drive is high. RAID config necessary to gain space is not resilient to failure. SSD is not proven to last long in RAID.

Another downside to this configuration is that the boards I have purchased will require a PCI video card when configuring the RAID array. This is due to the x16 slot forcing 'PEG is here', disabling the on-board graphics. A bummer, as the on-board graphics was part of why I chose that board. At this point I am assuming it will work in the x16 slot...I will obviously test before I go through with this - I have a PERC 5i to test with.

Huge Openfiler or ZFS box
This is a flight of fancy, it really doesn't do what I want, but it's fun anyways.

Norco 4U chassis - 24 hotswap bays
Appropriate PSU (depends on availability of staggered spinup)
Two dual-NICs (PCIe?)
24 WD 1TB 7200rpm
RAID can be RAID10, or broken up to two arrays of RAID10 (simulate multiple chassis)
HP SAS Expander & HP P400 card (actually allows up to 32 drives) & Mini-SAS cables
Core i7-950, 24GB RAM
Supermicro X8STE-O (lots of PCIe slots)

SubTotal: $4000
Price per GB: $0.16
Est. cost per annum: $245
Raw array size: 12TB
Pro: A lot of spindles, a lot of space. Very resilient to failures.
Con: A lot of power usage. More complicated methods of simulating multiple sites. Probably not as fast as a four-SSD RAID0. Dependence on Openfiler or ZFS to stay VMware-compatible.

The 4U Openfiler box is only really necessary if I were to be storing a LOT of data. And I'm not. But it was a fun mental exercise.

Conclusions

SSD will be faster, but has a much greater chance of failure.
SATA will provide a lot of space, but has a much higher initial investment and annual cost.

Well...SAS comes out on top of this one. But I'm still not convinced...it leaves me with several wildcards:

VSA performance overhead
VMware overhead
Being locked down to storage on a host (i.e. anything happens to the host, array is gone).

There is no point going to Openfiler I think, because I would need two, and that means two more computers that could otherwise be hosts.

Which of course leaves the real option: Waiting.

:P

*Note: The difference in cost between the SAS and SSD is pretty significant! My calculations used 1w per SSD and 7.2w per SAS disk (a figure I found online stating that was the wattage during seek). $70/year is not insignificant...but I suppose that is dependent on how long the SSDs last in RAID usage.

*Note: The kWh estimate based on heavy usage, so is most likely exaggerated in this case.

Lab - SAN options

2011-03-01T10:31:00.000-08:00

Update: Going to wait and see what the next few months brings for new products. Hesitant to purchase the Iomega units as they are several years old now.

I had a sobering thought this afternoon. My entire VMware lab is basing iSCSI SAN usage off of our Synology DS410j. The same NAS that houses all of our pictures, data, etc. If my lab kills it, my wife kills me. (and frankly I'd be super-upset as well) We do have backups (that I need to test) done to an external, but it would be a giant pain if it broke.

Thus, I consider my options.

Dedicated Openfiler box (~$1800)
This would be a stand alone machine running the latest version of Openfiler. Dual RAID cards with 8 spindles each, two dual NICs for MPIO to two locations, thus mimicking two SANs.

Pro: Does MPIO, high spindle count.
Con: Cannot test SAN replication. Power hippo.

Two NAS that support iSCSI (~$2200)
Basically buy two DS411+ and fill with 1TB or 2TB drives.

Pro: Simple to configure/maintain. Can do synchronization between units.
Con: Only four spindles. Only one NIC (no MPIO). Relatively high cost per unit ($690 per bare chassis).

ESX host running Openfiler VM (~$2200)
Basically an ESX host with a LOT of local storage (dual RAID arrays, 16 spindles).

Pro: VM appliance. Able to run other VMs on same machine.
Con: More complex to maintain. Less ports usable for ESX purposes as they are tied to Openfiler VM. More layers so more latency.

All in all, pricy, and none of them really give me what I'm looking for.

However, there is one more option that I've been avoiding:

NAS Option #2 (~$1300)
IOmega IX4-200d (4x1TB). A lot of the VM bloggers use these, so maybe it's wise to just jump on board. VMware HCL is no small thing either. I would have liked some sort of MPIO option. According to the manual, it does a limited version of MPIO, so maybe this is the best way.

Pro: On the VMware HCL. Dual NICs. 4x1TB is plenty of space.
Con: Bad support (many people not happy). Cannot buy diskless.

Groups not resolving - final solution

2011-03-01T05:34:00.000-08:00

Let's go through this without any assumptions (or try to, anyways).

Problem
Users in the parent domain send an email to a group (whose users are all in the parent domain). Email does not come through, no errors, no indication to user that the email has failed. If the email is also addressed to individuals (internal or external), those individuals receive the email.

On the Exchange side, the message log for an email sent only to a group stops at the 'Categorizer' step. For an email sent to a group and individuals the log completes for the individuals, but the group is not logged past the Categorizer stage.

Environment
Exchange 2003, AD at the 2008 functional level. Exchange 2010 domain preparation has been performed. Clients using WindowsXP/7 and Outlook 2007/2010. Exchange server is hosted in the parent domain. Child domain connects to that server.

Exchange 2003 machine has Directory Access set to automatically discover DCs, and sees all the DCs in the domain (5 child domain, 6 parent domain). No stale entries.

The groups in question (thus far) only contain parent domain users, and no stale user entries in these groups. At the same time that userA will try and fail sending an email to groupA, userB can successfully send an email to groupA.

Experiment
Using the 'reconnect' button in Outlook Connection Status, I will attempt to send to a group when connected to each separate DC. The object is to see if a single DC connection is causing issues.

Tested sending to DistList-Universal, SecurityList-Universal, DistList-Global, SecurityList-Global.

Test1: 8:55am
This test was just sending to DistList-Universal, DistList-Global, SecList-Universal, SecList-Global. Below are the message log results:

DC1: Only DistList-Global was stopped at Categorizer, other three successfully sent
At this point I went on a hunch and disregarded the other three, focused only on DistList-Global.
DC4: Success.
DC5: Success.
DC6: Success.
DC1: Success.

Test2: 9:24am
This test was just sending to DistList-Global. Below are the message log results:
DC3 – Stopped at Categorizer
DC4 – Success
DC5 – Success
DC1 – Success
DC6 – Success
DC2 – Success
DC3 – Success
DC4 – Success

Test3: 9:35am
This test was just sending to DistList-Global. Below are the message log results:
DC1 – Stopped at Categorizer
DC2 – Success
DC1 - Success

Test4: 9:45am
Tested sending only to DistList-Universal.
Success for all DCs twice around.

After setting logging for the Categorizer to 7 in the registry, discovered the following:

(missing the screencap...here's the text)
MSExchangeTransport
Categorizer
Event ID 6009
Categorizer encountered a hard error while processing a message. While processing user 'SMTP:groupemailaddress@domain.com', the function 'CCatExpandableRecpi::HrExpandAttribute' called 'plUTF8->BeginUTF8AttributeEnumeration' which returned the error code '0xc0040550' (). ( f:\tisp2\transmt\src\phatq\cat\src\ccatrecip.cpp@3048 )

Possibilities
I Googled the error specifics and came up with two options:

Errors could be due to Symantec Mail Security – Heuristic Spam. Other suggestions turn up similar recommendations against various AV/Anti-Spam.

Another suggestion is that it could be related only to Global groups, and to convert the groups to Universal. “…issue can occur if mails are sent to Global Distribution Groups in a multiple domain environment…” This was confirmed by me testing sending only to a Universal group - I could not reproduce the issue.

I conferred with our team, and they confirmed that the GFI Mail Essentials and Mail Security software we are using is out of date and can be removed. We have an external check for spam/AV done by MXlogic. They also agreed that the proposed root cause is most likely correct.

I will post in the comments as to whether or not that actually resolved it. We do have the Universal group thing to fall back on. We are attempting the GFI option first because it needs removing anyways, and moving from Global to Universal groups is a much more drastic change.

What I learned

Never base an investigation on assumption. Ever.
We do not have AD sites configured in our environment.
We have out-dated GFI software on the mail server.
AV/Anti-Spam can cause weird random stuff to happen to Exchange.
A lot about how Outlook works (Connection Status, Directory Access round-robin).
A lot about how Exchange works (Categorizer, Directory Access tab).
That logging can go to '7' via the registry (and that Exchange has a lot of logging options).

Background Notes
Categorizer
The Categorizer is responsible looks up the proxyAddresses attribute in AD for any recipients of the message. If that list includes a distribution group, it expands the group (unless an expansion server is configured). Currently the group in question has ‘any server in the organization’ set (Exchange 2003 machine plus the two new Exchange 2010 boxes).

Sources
SMTP Categorizer: http://technet.microsoft.com/en-us/library/aa998977%28EXCHG.65%29.aspx
Categorizer logging: http://msexchangeteam.com/archive/2006/06/23/428114.aspx
Group conversion solution: http://social.technet.microsoft.com/Forums/en-US/exchangesvrtransport/thread/6a453b9c-7b77-4f10-8567-87d6118fdec3
AD integration Ex2003: http://blogs.msdn.com/b/douggowans/archive/2007/01/04/ad-integration-from-2003-to-2007.aspx

Course/exam/certification links & books & skill goals, oh my

2011-02-25T15:50:00.000-08:00

The following are training options that I'd like to pursue. Just noting links for future reference.

VMware courses/certifications

VTSP4 (completed, Q1, 2011)
VSP4 (when time allows, 2011)
VCP4 (booking shortly...May, 2011)
VCAP4-DCAdministration (Q3, 2011?)
VCAP4-DCDesign (Q1, 2012?)
vSphere Troubleshooting (VMware link) (confirmed, April 18-21, 2011)
vSphere Manage for Performance (VMware link) (as funds allow)
vSphere Manage and Design for Security (VMware link) (as funds allow)
vSphere Design Workshop (VMware link) (as funds allow)
VCDX4 (by the time I'm ready for this, it'll be VCDX5 or 6)

Microsoft exams/certifications

MCM: Windows Server 2008 R2: Directory (by the time I do this, it'll be Server 2050 R3 or something)
70-297 - Designing a W2k3 Active Directory and Network Infrastructure (definite, Q4, 2011)
MCITP: Server 2008R2 Virtualization Administrator

The VirtAdmin cert consists of:

70-699 - TS: Windows Server 2008 R2, Desktop Virtualization
70-693 - PRO: Windows Server 2008 R2, Virtualization Administrator
Elective: 70-659 - TS: Windows Server 2008 R2, Server Virtualization
Elective: 70-652 - TS: Windows Server Virtualization, Configuring

This is a new certification and does not have any books listed on their site yet. I'd go for this one just to round out my virtualization skillset, or if it was required.

And a list of the books I intend to go through to aid my training this year:

Mastering VMware vSphere 4 (Scott Lowe)
vSphere 4 Implementation (Mike Laverick)
Maximum vSphere (Siebert & Seagrave)
VCP4 Study Guide (Robert Schmidt)
VMware ESXi Planning, Implementation, and Security (Dave Mishchenko)
vSphere 4.1 HA and DRS technical deepdive (Epping & Denneman)
vSphere 4.0 Quick Start Guide (Epping et. al.)
vSphere and Virtual Infrastructure Security (Edward Haletky)
VMware Cookbook (Troy & Helmke)
vSphere 4 Administration Instant Reference (Lowe, McCarty, & Johnson)
Managing VMware Infrastructure w. Windows PowerShell (Hal Rottenburg)
70-297 Microsoft Press book

Yup! It'll be busy...good thing I have 2 hours/day on the train!

Skill goals

PowerCLI/vCLI/PowerShell
Necessary items for VCP/VCAP
Better understanding of AD roles

How can I demonstrate them? Well first of all passing the VCP (and the VCAP-DCA would be very nice). The AD stuff is coming naturally just from my job - a lot of AD digging lately. PowerCLI/vCLI is going to be essential for ESXi implementation, and PowerShell is just something I've never gotten around to. We'll call that a tertiary goal...probably alongside IPv6.

And if time allows...

REALLY understand IPv6 already

Honestly. I know enough to get my MCITP:EA, but I still feel like a lost boy in the v6-woods.

New lab setup

2011-02-25T08:58:00.001-08:00

Update:
My employer has kindly allowed me to take some APC UPSes off their hands, a two-post rack with 1U PDU and some shelves, and a zero-U PDU (which is on loan). I can take two of either a 3U 3000VA or 2U 3000VA. Just need fresh batteries. I wonder what they utilize for normal operation... The rack is also perfect...it has a shelf-type bottom so I can mount heavy long stuff from the bottom up.

We've decided to invest in a lab for certs and training. Immediately it will be my VCP lab, and afterwards for VCAP and other MS certifications (not to mention test environment/learning lab for new tech).

Key items:

Firewall (Astaro) with multiple NICs for DMZ, etc.
Switch upgrade (from 8 port to 24 port Gb)
Three ESX hosts (i5, 16GB)
The NICs and switches I'm sourcing from Ebay/local.

Infrastructure
My employer has also graciously offered to donate a two post rack for the lab, just need to get it home and mounted to the floor. All will be mounted on said rack in the basement. Currently I have all my network gear on a 1200VA APC UPS down there...will keep it all through that until I can justify a racked UPS.

Firewall
The firewall is a Supermicro SYS-5015A-EHF-D525, putting 2x2GB RAM, a dual-port PCIe PRO1000PT NIC, and a spare laptop drive into it (also got two sub-20db fans for cooling). Going to run Astaro v8. Total of four NICs (all Intel-based).

Astaro is free for home use. I've used it in a production environment before, and it's a very nice mix of price/performance. Astaro home/free are the same thing, just business use requires licensing.

Networking
I will keep the 8-port switch as it's rack-mounted already, and you can never have enough ports! Going with a Linksys 48-port (SRW2048). The 8-port may be retained if necessary.

Since each host has 7 NICs, I'll need 21 NICs just for the hosts, not to mention other network items like the Synology. Going with the 48-port will give me greater flexibility and room for expansion.

ESX hosts
The ESX hosts are spec'd something like this:

3U iStarUSA D-300-PFS case (thanks to Chad Sakac!)
Upgraded fans (Vantec variable-speed, 2x60mm, 1x80mm)
Core i5-760 (quad-core w. VT)
Crucial 4x4GB RAM (CT2KIT51264BA1339)
USB flash drive (ESXi)
Intel motherboard w. on-board DVI and Intel chipset NIC (BOXDH55HC)
3x PRO1000MT dual-port NICs (total of 7 NICs per host, all Intel)
Seasonic S12II 380w PSU

Storage
My trusty Synology DS410j will do iSCSI duties until it can't keep up, then I'll look at options.

Alternatives include a lightweight server with a many-spindle RAID10 running Starwind or the Synology DS411+ (approx 4x the iSCSI performance of the DS410j). I think if I upgrade down the line I'll want to move to MPIO (so at least two ports). Not for performance requirements, but for having that option to configure in my lab.

VMware backup for ESXi free

2011-02-19T14:56:00.000-08:00

I've been doing some digging for a client. I've got him started on the ESXi free path for his small VoIP company, and he loves it. I had always figured that for backup we would just set him up with VCB (even though I've had awful past experience with it, it's a free option), or more to the point, GhettoVCB.

I checked out the FAQs for GhettoVCB more closely today, and it seems you must have the licensed version in order to access the APIs that VCB uses. Fair enough, an Essentials pack is cheap enough, and he and I have discussed it in the past. He is amenable to buying the base pack - he would never need more than two host machines anyways.

Another issue crops up: VCB is EOL as of 4.1. Going forward, it is all vStorage APIs for Data Protection (VADP). This is a nicer solution than VCB, but it seems that right now the only options in the VADP area are 3rd party paid software. For someone who wants to do everything cheap and open-source, this will not be a good option.

Looking into what other options we have, I think it's down to something like this:

Option 1

Have a really good backup system inside the guest OSes.
Have a really good host rollout solution (unfortunately BSD does not have a kickstart solution)

This means that if a guest VM goes down (irrecoverable), your data is backed up, and it's easy to roll out a replacement.

Pros: No software costs at all.

Cons: Built-in resiliency is a must. This means all critical systems must have a primary and secondary. Any VM that crashes must be rebuilt by hand from backup data.

Option 2

GhettoVCB

GhettoVCB configured to suit environment.

Pros: Pre-configured backup solution that is widely used by the VMware community. Allows VM backup/restore, so if a VM crashes, you do not have to rebuild by hand.

Cons: In order for VCB to function, the Essentials pack must be purchased to unlock the VCB APIs.

Option 3

VADP 3rd party

Solution provided by 3rd party.

Pros: Ease of use, reliable, supported by large companies.

Cons: High cost. Requires Essentials pack, plus 3rd party software (unknown cost).

I think we'll have to re-visit things. If he's not ready to purchase Essentials yet, then we will have to make do with system backups.

Certification path update

2011-02-18T18:12:00.001-08:00

I've asked my employer for support with the VCP4 training/exam, so we'll see how that turns out. Lord willing, I'll be passing this Q4 2011. One other item is the 70-297 exam (2003 AD design).

To get me to that point, I'm setting myself goals:

Read at least one VMware book per month. I have a lot of good VMware books - I need to read them! I am starting (rather, finishing) 'Mastering VMware vSphere 4', by Scott Lowe.
Get serious about my home lab. Sounds like a silly thing, but I've bounced around here and there about how I want to do things - all Workstation, dual hosts, etc. I think I'm going to bite the bullet and get two identical PCs (probably whiteboxen, as I've had before).
Set aside study time and ONLY study during those set hours. Outside of that time, no studying. I have a (tendency is too light a word) compulsion to do things in intense chunks, usually separated by long periods of time. I think 3-4 hours each weekend, and 2-3 hours twice a week. This is what my wife and I agreed upon while I was doing my MCITP:EA, anyways (actually more...but this will only be one exam, not seven).

Beyond the above, I also intend on documenting all of my troubles/trials/successes in this blog as a kind of journal.

Q1 2012 (or perhaps sooner, I will be reading this book in between VMware studies) I'd like to get the AD design cert under my belt as well (70-297). Something they still haven't upgraded in the MS line of certs, so might as well get it done - I already have the book, and my experience is certainly leading in this direction.

Beyond those two, down the road I may do certs just for the learning challenge - like the Hyper-V cert, perhaps some Messenging stuff like Lync and Exchange. Those are secondary to my VMware and Active Directory pursuits, however.

Looking forward, I have another item to set my long-term sights on: Microsoft Master (AD) certification and VCDX (and surrounding certs) I recently corresponded with Mark Parris regarding the MCM:AD cert, and he sent me a laundry list of reading to do. Seriously, the list is epic, and we are talking years, not months, of study. Besides, I need significant deployment experience before I can even think about going that route. However, once I get the 70-297 done, I'll meet the scholastic requirements for the MCM:AD, so that's something, anyways.

The VCDX is something I've felt is in my future for some time now, so getting the VCP down and done is Part 1 - and it's usually the first step that's the hardest to take in a journey.

To recap:

VCP4 (Q4 2011)
70-297 (Q5 2012 or sooner)
VCDX (?)
MCM:AD (?)

Weird issue with Exchange Global groups

2011-02-18T10:59:00.001-08:00

Update: Ok, the below post contains erroneous conclusions. Clients are NOT using child domain GCs. I am becoming more convinced this is local to the Exchange server, as the child domain users are seeing their GCs (and can cycle with 'reconnect') with no issues. I observed a user here having the issue, and I noted that their Outlook 2007 Con.Status was not showing any directory items. However, when it started working for them after a while (15-45 minutes...?) they still did not see any directory items. Going to start from scratch in a new post.

Lesson: The danger of assuming something is happening when you don't know conclusively is that you send your entire investigation off in the wrong direction.

Sort of resolved. So we fix the issue of parent domain users having groups not resolve. What about child domain users trying to resolve child domain groups? If I remove the child domain GCs from the list, does that prevent child domain group lookups? Will investigate.

So Outlook definitely uses RR to get a GC. You can see this by CTRL+left click on the Outlook icon in the system tray, then choose 'connection status'. The server next to 'directory' is the GC in use. If you close Outlook, re-open, you'll see this change as it RRs among the available GCs. In my testing though, I could not get it to pick a child domain GC. ACTUALLY...if you just click the 'reconnect' button, it cycles through the GCs available to you. Nice!

Option1: Use AD Sites. Since the parent and child domains are geographically separate, issue could be mitigated to a degree. However, we'd have to figure out how the datacentre locations would come into play - I'll need to research more into how they actually work in this situation. In theory this should prevent Outlook's RR from picking GCs in the wrong site.

Option2: Remove child domain GCs from the 'Directory Access' tab. May impact child domain Outlook users - must verify this first. (Can parent domain GCs do lookups into the child domain? - Special trust required?)

Option3: There is a registry option to force Outlook to use a local GC for lookups. Not a good option.

---------------------

RESOLVED:

Ok, here's what's happened:

We had GC issues with Exchange a few weeks ago – this was because they were manually configured, and even then only one GC available to Exchange.
I changed that to ‘auto discover’ DCs. Resolved issue at the time.
List of available DCs/GCs now includes child domain's DCs.
When clients load Outlook, it round-robins for a GC to use.
Occasionally it RRs and gets a child domain DC.
Child domain DCs have no knowledge of groups in globalive.local.
Email to group fails at ‘Categorizer’ stage. (Categorizer resolves group names into individual email addresses.)

Resolution: Change 'Directory Access' tabs back to manual and add all parent domain DCs to the list.

Lesson? Do NOT set 'Directory Access' to automatically discover when you have child domains.

Boomshakalaka. Feels good to figure it out!

-------------------------------
Or at least, that's what I think is happening here.

Environment has a parent domain and child domain.
Exchange server is for both domains.
Almost all DCs across both domains are GCs.
Group in question is a global security group with members only from the parent domain (and no visible stale membership).
'Directory Access' tab in Exchange server properties lists all GCs in all domains (set to automatically discover GCs).

We occasionally have this issue:

User sends an email to a group from their outlook.
Email never arrives to the group.
Mail tracking indicates the mail has arrived at the server.
Mail tracking last event: SMTP: Message Submitted to Categorizer

Mail never leaves the server.

Some googling revealed that the 'Categorizer' is responsible for talking to a GC and getting group membership information.

What I think is happening stems from the list of GCs...the Categorizer goes to look up group membership and happens to choose one of the child domain GCs. That GC would not have knowledge of the group membership of the parent domain, and so the mail gets stuck and never sent.

There is one alternative: Orphaned user objects. However, the group in question is clear of old/disabled users, and there are no 'mystery GUIDs'.

At least that's all I can come up with. I've enabled the LDAP/directory logging, so we'll see what that turns up.

More info on the categorizer: http://searchexchange.techtarget.com/tutorial/Part-3-How-the-SMTP-categorizer-works-in-Exchange-Server-and-Active-Directory

And this nice tidbit:

"If your Exchange Server organization is having mail flow issues, use message tracking to see where the process is breaking down. If messages are stopping at the categorizer, you should begin troubleshooting the problem by looking at directory-access issues."

And for the random issues, the messages in question are being stopped at the categorizer.

With logging turned on for MSExchangeTransport, I am seeing categorizer 6020 events (...using LDAP server Host: "DCname") - indicating that the DSAccess service is polling the AD topology successfully. There are events for each DC in the 'Directory Access' tab. This indicates that there are no issues with discovering DCs. (this can also be tested by just viewing the 'Directory Access' tab - it should not take more than a few seconds to resolve the list. If it sits there for more than 30-40 seconds, it's probably timing out on one or more DCs.)

Since the error is difficult to troubleshoot (very rare, spread out), the only logical next step is to try and prove the theory - what GC does the Categorizer use? Any order?

Nice technet article on how Exchange 2003 uses LDAP connections: http://technet.microsoft.com/en-us/library/aa996247%28EXCHG.65%29.aspx#LDAPConnectionLoadBalancingAnd

Ok, we have an answer: http://technet.microsoft.com/en-us/library/cc751317.aspx

This doc explains things in detail. DSProxy is what actually talks to the GCs when doing lookups, and it uses load balancing. (emphasis mine)

"Although NSPI is a very efficient process, the DSProxy process uses a load balancing mechanism to ensure that client requests are divided equally among all available global catalog servers.

When a MAPI client contacts NSPI Proxy, the IP address of the requesting client is hashed against the number of available global catalog servers. DSProxy uses the result to either proxy or refer the client to one of the global catalog servers. This load balancing method enables the client to contact the same global catalog server, thus ensuring consistency. The Directory Service Referral interface (RFRI) uses a different load balancing mechanism; when a client connects to RFRI, global catalog servers are returned in round robin fashion."

It appears that Outlook 2000 and up use RFR to connect (aka use Round Robin for GC access). GC server refresh is done at startup of the client.

Summary: Ok, so that's good enough for me - I can test this next time - if a client is finding that emails to a certain group are failing, I can ask them to close Outlook, re-open, and try re-sending the mail. I will update when this comes up again (if it does...we're upgrading to 2010 shortly).

Update: Yup! 99.9% sure this was the issue. I spoke to the two people who were having the issue, and got one to send a test (it failed), and then close Outlook and test again. It worked this time! He got the other guy to test (failed), then close/re-open Outlook, and voila, worked. Cool!

SYSVOL replication issues

2011-02-11T07:03:00.000-08:00

Recently had a DC (we'll call it DC1) acting very strangely.

There were some errors in the event logs:

--------------------------
EventID 1058, The system cannot find the path specified. (then lists a GPO GUID path in SYSVOL)
--------------------------

Rebooting would just give me the error again. Tried gpupdate and got this:

--------------------------
C:\>gpupdate
Updating Policy...

User policy could not be updated successfully. The following errors were encount
ered:

The processing of Group Policy failed. Windows attempted to read the file \\domain.local\SysVol\domain.local\Policies\{9EA16EAF-3A76-4972-88CE-1BA2435CAE8
E}\gpt.ini from a domain controller and was not successful. Group Policy setting
s may not be applied until this event is resolved. This issue may be transient a
nd could be caused by one or more of the following:
a) Name Resolution/Network Connectivity to the current domain controller.
b) File Replication Service Latency (a file created on another domain controller
has not replicated to the current domain controller).
c) The Distributed File System (DFS) client has been disabled.
Computer policy could not be updated successfully. The following errors were enc
ountered:

The processing of Group Policy failed. Windows attempted to read the file \\domain.local\SysVol\domain.local\Policies\{9EA16EAF-3A76-4972-88CE-1BA2435CAE8
E}\gpt.ini from a domain controller and was not successful. Group Policy setting
s may not be applied until this event is resolved. This issue may be transient a
nd could be caused by one or more of the following:
a) Name Resolution/Network Connectivity to the current domain controller.
b) File Replication Service Latency (a file created on another domain controller
has not replicated to the current domain controller).
c) The Distributed File System (DFS) client has been disabled.

To diagnose the failure, review the event log or invoke gpmc.msc to access infor
mation about Group Policy results.

C:\>
--------------------------

To explain what exactly was going wrong, I must sketch out SYSVOL and GPO. SYSVOL is replicated among the DCs and contains things like logon scripts, GPOs, etc. When a GPO is applied, it's pulled from SYSVOL (simplifying things) which resides on the nearest DC. When a DC accesses it, it uses the local SYSVOL cache.

You can browse to SYSVOL's GPOs: \\domain.local\sysvol\domain.local\Policies

You'll see folders labeled with the GUIDs of all your GPOs. In this case, the GUID referenced in the error simply was not there. I checked all the other DCs and the GPO was present on them. Very odd.

I also began to notice other events, these on all other DCs except the one I started looking at: NtFrs EventID 13508, warning, trouble enabling replication between DC1 and THISDC.

Every other DC showed the same warning, so that pointed directly at DC1 being the issue. I checked into DC1's logs a little deeper this time, and came up with this: NtFrs EventID 13568

--------------------------
The File Replication Service has detected that the replica set "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)" is in JRNL_WRAP_ERROR.

Replica set name is : "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"
Replica root path is : "c:\windows\sysvol\domain"
Replica root volume is : "\\.\C:"
A Replica set hits JRNL_WRAP_ERROR when the record that it is trying to read from the NTFS USN journal is not found. This can occur because of one of the following reasons.

[1] Volume "\\.\C:" has been formatted.
[2] The NTFS USN journal on volume "\\.\C:" has been deleted.
[3] The NTFS USN journal on volume "\\.\C:" has been truncated. Chkdsk can truncate the journal if it finds corrupt entries at the end of the journal.
[4] File Replication Service was not running on this computer for a long time.
[5] File Replication Service could not keep up with the rate of Disk IO activity on "\\.\C:".
Setting the "Enable Journal Wrap Automatic Restore" registry parameter to 1 will cause the following recovery steps to be taken to automatically recover from this error state.
[1] At the first poll, which will occur in 5 minutes, this computer will be deleted from the replica set. If you do not want to wait 5 minutes, then run "net stop ntfrs" followed by "net start ntfrs" to restart the File Replication Service.
[2] At the poll following the deletion this computer will be re-added to the replica set. The re-addition will trigger a full tree sync for the replica set.

WARNING: During the recovery process data in the replica tree may be unavailable. You should reset the registry parameter described above to 0 to prevent automatic recovery from making the data unexpectedly unavailable if this error condition occurs again.

To change this registry parameter, run regedit.

Click on Start, Run and type regedit.

Expand HKEY_LOCAL_MACHINE.
Click down the key path:
"System\CurrentControlSet\Services\NtFrs\Parameters"
Double click on the value name
"Enable Journal Wrap Automatic Restore"
and update the value.

If the value name is not present you may add it with the New->DWORD Value function under the Edit Menu item. Type the value name exactly as shown above.
--------------------------

Essentially what has happened is the SYSVOL share has become corrupt and you need to pull a new copy from the other DCs.

1. Browse to the noted location in the registry on DC1.
2. Create the new key.
3. Change the value to ‘1’.
4. Stop/start the ntfrs service (net stop/start ntfrs).
5. Note the entries in the FRS log. (13560, 13520)
6. Change the value back to ‘0’.
7. Wait for replication to complete. (13553, 13554)
8. Note success. (13516 – no longer preventing from becoming DC)
9. Note other servers are now seeing event ID 13509 (NtFrs once again has connection).

You can actually replicate the 13508/9 events yourself, just disconnect a DC for about an hour, you'll see them pop up. Reconnect and you'll see the 13509 event.

Kerberos ticket etypes (eventID 27 on 2003 DCs)

2011-02-11T06:39:00.000-08:00

Our 2003 DCs are seeing these errors (not the 2008 DCs):
KDC EventID 27
While processing a TGS request for the target server krbtgt/DOMAIN.LOCAL, the account username@DOMAIN.LOCAL did not have a suitable key for generating a Kerberos ticket (the missing key has an ID of 8). The requested etypes were 18. The accounts available etypes were 23 -133 -128 3 1.

The reason behind this error is that the client is trying to authenticate with an unknown etype - unknown to the 2003 DC. Once the client finds a 2008 DC, all is well.

Solution is to replace the 2003 DCs with 2008 DCs, it is really just a question of compatibility.

Extended explanation:
Client is asking for a Kerberos ticket with which it will authenticate against domain resources. These tickets are encrypted. XP/2003 has a set list (DES) of supported encryption types (etypes), and Win7/2008/R2 support different etypes (AES/RC4). Since an XP/2003 client does not know about the new etypes, you'll see these errors when a Win7/2008/R2 client tries to request a ticket on a 2003 DC.

Sources:
http://social.technet.microsoft.com/Forums/en/winservergen/thread/8999cc8e-53b6-4b74-a310-167c8adba257
http://support.microsoft.com/kb/977321
http://technet.microsoft.com/en-us/library/cc733974%28WS.10%29.aspx

WINS errors

2011-02-10T08:56:00.001-08:00

I've spent a bit of time the last few days struggling with WINS.

My topology consists of three WINS servers, one for each location, with one functioning as a hub, the other two as spokes (as per Technet's recommendation). The hub is configured with the two spoke servers as push/pull replication partners. The spokes each have the hub as their push/pull replication partner. (as per Technet's recommendation)

So everything is set how Microsoft wants it to be. There are no firewall restrictions between the WINS hosts (who are also DCs). They are running Server 2008.

All my configuration seems correct, yet I get these errors in the System event log: EventID 4102 'connection aborted by remote WINS'. When you turn on advanced logging, you see more detailed errors: EventID 4149 'Winsock Send could not send all the bytes.'

Another server returned the following: EventID 4343 'WINS server noticed chance of duplicate name registration...' These are just warnings and are now showing because I turned on the advanced logging.

On the hub server, I restart the WINS service and get this: EventID 4224 'WINS encountered a database error. This may or may not be a serious error. WINS will try to recover from it.' It mentions checking the application log for more details about database errors.
NOTE: According to this, the error is deprecated.

The ESENT eventID 103 (wins (2752) The database engine stopped the instance (0).) in the application log also has a DWORD for the error code '2752' and an EventRecordID of '5932'. There is another event immediately following: EventID 102 (wins (6956) The database engine (6.00.6002.0000) started a new instance (0).)

I restarted WINS on all servers.

Now I see this: EventID 4141 (WINS pulled records from a WINS while doing Pull replication. The partner's address and the owner's address whose records were pulled are given in the data section (second and third DWORD respectively). The number of records pulled is the fourth DWORD.) DWORD info referenced is not clear, just hex codes.

On Spoke1, I see this: EventID 4283 (WINS encountered an error while processing a push trigger or update notification. The error code is given in the data section. If it indicates a communication failure, check to see if the remote WINS that sent the trigger went down. If the remote WINS is on a different subnet, then maybe the router is down.) It is immediately followed by EventID 4149.

On Spoke1, I force a 'pull replication' and see this: EventID 4141
On Spoke2, I force a 'pull replication' and see this: EventID 4141

On Hub, I see this twice: EventID 4121 (WINS's Replicator could not find any records in the WINS database. This means there are no active or tombstone records in the database. It could be that the records being requested by a remote WINS server have either been released or do not exist.)

On Hub, I force a 'push replication' to each spoke: EventID 4121
On Hub, I check 'Active Registrations'. All three servers are listed as 'owners' for active records.

On Spoke1, I force a 'push' to Hub: EventID 4141 on Hub.
On Spoke2, I force a 'push' to Hub: EventID 4141 on Hub.

Back to basics...if they are replicating properly, the DB should be the same across all servers. Checked that, and they are not. The two spoke servers look similar, but the hub is very different.

Deleted old owners off the hub, then repeated on each spoke.

Restarted WINS on each spoke, then the hub. Each server seemed to start ok, barring the 4224 error on all servers.

Checked the 'Active Registrations' on each server again. Spokes are nearly identical, hub appears to be missing a huge amount of new records. I sorted them by expiry date (in this case the latest is 3/6/2011). The spokes have many pages of 3/6/2011 records, the hub has only a handful.

At this point I've wasted enough time on this...it should work, but doesn't, and nobody is complaining. Squeaky wheel gets the grease and all that.

Fixing duplicate SPNs (service principal name)

2011-02-09T08:13:00.000-08:00

This is a pretty handy thing to know:

SPNs are used when a specific service/daemon uses Kerberos to authenticate against AD. They map a specific service, port, and object together with this convention: class/host:port/name

If you use a computer object to auth (such as local service):
MSSQLSVC/tor-sql-01.domain.local:1433

If you use a user object to auth (such as a service account, or admin account):
MSSQLSVC/username:1433

Why do we care about duplicate SPNs? If you have two entries trying to auth using the same Kerberos ticket (I think that's right...), they will conflict, and cause errors and service failures.

To check for duplicate SPNs:
The command "setspn.exe -X

C:\Windows\system32>setspn -X
Processing entry 7
MSSQLSvc/server1.company.local:1433 is registered on these accounts:
CN=SERVER1,OU=servers,OU=resources,DC=company,DC=local
CN=SQL Admin,OU=service accounts,OU=resources,DC=company,DC=local

found 1 groups of duplicate SPNs. (truncated/sanitized)

Note that you see the SPN (MSSQLSvc/server1.company.local:1433) and two objects (CN=SERVER1 & CN=SQL Admin). Use ADSIedit.msc to view the 'serviceprincipalname' attribute for each object, and you will see the same SPN.

To figure out which one to delete, log on to the server where the service/daemon is hosted. Find out what it uses to log on to AD. In this case, it is a Microsoft SQL Server, and I can easily see what credentials it uses by checking services.msc, scrolling down to the service in question and viewing the 'log on as' column. It says 'Local Service', so therefore it uses the computer object. I can delete the SPN entry under the SQL Admin user object's SPN attribute.

Pretty simple when you get down to it.

Sources:
http://alt.pluralsight.com/wiki/default.aspx/Keith.GuideBook/WhatIsAServicePrincipalNameSPN.html
http://msmvps.com/blogs/vandooren/archive/2008/03/11/getting-rid-of-the-duplicate-spn-in-active-directory.aspx
http://alt.pluralsight.com/wiki/default.aspx/Keith.GuideBook/HowToUseServicePrincipalNames.html

Cannot join domain - 'network name no longer available' & 'access denied' DHCP authorizing

2010-12-16T07:33:00.000-08:00

(sorry for the long title)

We ran into an interesting problem I wanted to document here, hopefully it will help someone else out.

At head office: Old call center is being moved to a new building, but cannot have any downtime. We are getting infrastructure set up (DC/proxy/etc) ahead of time. We had a test network set up at head office but it was having issues (ADSL flapping), so while on the new IP subnet we were not able to join the domain (seeing 'network name no longer available' error). We figured this was due to the flapping, and we decided to leave the join/promote to when we were local.

Catch to the setup: We cannot see two of the four DCs, as our hub/spoke network design does not allow the spokes (remote offices) to see each other, only to see the hub (head office). This is fine, as we have two DCs at the spoke office, and two DCs at the head office (for this child domain...not in total).

At the remote office: Same issue persists, but we no longer have the flapping.

1. We could ping the FQDN of the domain: child.domain.local
2. We could ping the DCs involved (the two at the head office).

We brainstormed, checked security settings, disabled firewalls, telnetted to ports, and eventually we ticked on there being multiple roles, and some are only set on specific DCs (naming master, ops master, infrastructure master, etc). Last year they moved several key roles down to the remote site to speed up logon times. It turns out that you need these roles to join the domain.

The roles were moved back to the two DCs at the head office, and we were able to join the domain immediately, and promote it to DC without any issues as well.

One last issue cropped up - the new DHCP scope could not be authorized. It could be created, services started, but not authorized. A little research turned up that for CHILD domains, only ENTERPRISE admins could authorize scopes. Small point, but pretty key!! Gave the correct permissions, and authorized the scope.

More career update stuff

2010-10-20T05:21:00.000-07:00

Well, I think I've gotten a clearer picture over the week. I've run across a few articles/blogs in the virtualization community and it has recalled to me why I enjoy that field so much.

So, as of right now (valid for the next 5 minutes), I'm going to 'major/minor' in virtualization/storage.

Cert-wise, the VCP is first in my sights, then I'll work on getting the NetApp training material done. I think I'll also get the VTSP while I'm at it, it should help me get a better business perspective on the VMware technical side of things.

After those (so...mid-end next year), I may still consider the RHCT or CCNA, but I need to learn more about what it means to be a storage expert first. The CCNA, if I'm not focusing squarely on storage, is probably not necessary, as I already have a decent grasp of the material (a bit stale...but still there).

CertPath 2010/2011
1. VCP
2. VTSP
3. NetApp exam
4. ?

Lab environment 2010/2011
1. Start from scratch!!
2. Diagram out what it will look like.
3. Document the reasoning behind the design.
4. Decide what hardware is needed.
5. Learn! (whee)

I think it makes more sense at this point to upgrade my existing hardware rather than add more physical boxen to the mix. I can go up to 24GB on my desktop, and it's cheap to do so (sub-$500), so that makes sense. I may go with another NAS device (i.e. Synology DS410), but I want to see how the VM NFS/iSCSI appliances work first.

Windows environment: DC, Exchange, SQL cluster, Sharepoint/IIS/vCenter
Linux environment: Samba/Apache/MySQL cluster, gateway
ESXi environment: 4x ESXi, 2x Storage Appliances

This will be my VCP/Windows/Linux lab environment, with the Windows portion functioning for a home network/Microsoft professional development environment. Gotta keep current, after all!

The hardware behind this:

Asus X58 motherboard (can't recall model), 12GB (to be 24GB) RAM, Core i7 920
Intel GbE nic, cheapie dual graphics cards for 2x2007FP & 1x2405FPW
Local storage: RAID5 4x1TB 7.2k, 500GB for ISO/etc
NAS: Synology DS410j RAID5 4x1TB 7.2k

Running Win7x64 and Workstation 7. Only other stuff I use the computer for is Sketchup, Office...and Minecraft.

The other options for the hardware was to get two HP ML110 G6 servers, but they would cost ~$650 each after tax/shipping, and only come with 2GB RAM, allowing a max of 8GB per server. I think the 24GB on my desktop is a better option! This is just a lab, after all. It will also save me a significant amount on my power bill. (Rather...prevent increases to my power bills.)

Career path update

2010-10-13T18:54:00.000-07:00

After a good bit of prayer, advice from friends, and a lot of thought, I've laid out a career path and set of goals.

Long-term goal: Senior Engineer/consultant in storage or virtualization or both.

I had also considered architect-type roles (not to be confused with archetypical roles), but decided that I'm not super enthused about going into a role that deals primarily with relationships. Not that I have a big problem with that sort of thing, just that people and relationships are not my passion, and long-term goals require passion. Learning and mastering technology, now there's something I have passion for. And in my relatively short IT career thus far (5-6 years), storage and virtualization have been head and shoulders above most everything else that has interested me..

So, to get on that route, what must I do? Why, certifications of course! Yep, I really enjoyed getting my MCITP:EA cert, and can't wait to start on the next round.

Here's my path:

Part 1: Short-term goals (present +2 years)
The reasoning behind this section is to gather a broad base of certifications and get exposure to as much of the areas I'm interested as possible.

~~MCSE (MCITP:EA)~~
This is done (7 exams over 5-6 months).
Optional extra under this is the Hyper-V certification (I think only one, and not R2 yet).

RHCT (Red Hat Certified Technician)
One exam.
This will be the most difficult of the four due to lack of exposure/experience.

VCP (VMware Certified Professional)
One exam, one 5-day course.
Have the books (and how!), just need to do the labs and the course.

CCNA (Cisco Certified Network Associate)
One exam. Reasoning behind this is a) I've already done the training for CCNA/CCNP, b) SAN = storage area network, and c) the CCNA is a great base with which to refresh my network knowledge.
Need new books/simulator.

Other
NetApp ONTAP (I have a friend who's helping me with materials, it's a good intro to vendor-specific certs relating to SANs/storage appliances)
Employer-specific courses?
This looked interesting, though I've never heard of it: http://www.snia.org/education/certification/

Once all these are done I'd take a good look at my options and decide if it was time to start focusing on one area or the other.

Part 2: Mid-term goals (+2-4 years)
This section is to further focus and develop what I have learned in the previous part.

CCNP (Cisco Certified Network Professional)
Three or four exams.
Only take this if storage networking is a definite path I want to go down. Or maybe just take it because the troubleshooting course is worth its weight in gold. Hands down the best course from my college program.

VCAP
Datacenter operations or design, or both.
This is iffy, but would be good to have for the deeper knowledge of datacenter VMware stuff.

RHCE
Written exam + practical exam.
Advanced Linux cert, would need at least two years' worth of solid Linux experience, but a great cert to have on my resume, and I'd learn a ton besides.

Part 3: Long-term goals (5+ years onwards)
By this point a focus should be in mind, so a choice for one or both of these certs is clear. At this point it's impossible and slightly foolish to say this is DEFINITELY where I'll be, because who knows. Good to know there are certs out there to shoot for however.

CCIE StorageNetworking?
Two exams, written and practical.
Learn more about this cert. Maybe there are others to look at first? This is loooong term...

VCDX?
This is more of an architect cert...but is, at the moment, the be-all end-all of VMware certs.

Anyways...long post, but I have felt such clear direction from the Lord on this - gotten me quite excited about the next few steps!

vmnic0: Link is Down

2010-03-25T17:24:00.001-07:00

Honest to goodness.

Mar 25 07:12:26 osh-esx-1 vmkernel: 9:12:00:32.092 cpu1:4210)<6>0000:01:00.0: vmnic0: Link is Down

I'm at a loss with this system. At the same time my iSCSI box powered itself off, and it gives me the most informative:

The operating system started at system time ‎2010‎-‎03‎-‎25T11:15:02.484375000Z.

Lovely.

Once I'm back from my vacation (starting tomorrow morning for a week), I'm nuking everything and starting fresh - it is a lab, after all. If I continue to get errors, I'm replacing the RAID card - my faux PERC5i - with a PERC6i.

At that point I'm also going to reconsider my storage infrastructure. I may give Windows another shot, and hope it indeed was the RAID card. Or, if I can find a storage filer that does iSCSI and MPIO, I'll give that a shot. I'm truly constrained by the gigabit network - when doing VM deployments it maxes out at 50% on the line, either one or two VM deployments.

Of course, this is all silly...I don't need MPIO. For standard lab stuff what I have is perfectly fast, but if I could use MPIO that would definitely be something to learn with, and a lab is all about expanding your horizons, after all.

ESX vmkernel errors - solved!

2010-03-10T20:24:00.001-08:00

Let's start with the errors I was seeing in: /var/log/vmkernel

Everything was fine until this showed up:

Mar 10 22:09:13 osh-esx-1 vmkernel: 0:00:06:47.624 cpu0:4096)<3>e1000: vmnic2: e1000_clean_tx_irq: Detected Tx Unit Hang
Mar 10 22:09:13 osh-esx-1 vmkernel: Tx Queue <0>
Mar 10 22:09:13 osh-esx-1 vmkernel: TDH
Mar 10 22:09:13 osh-esx-1 vmkernel: TDT
Mar 10 22:09:13 osh-esx-1 vmkernel: next_to_use
Mar 10 22:09:13 osh-esx-1 vmkernel: next_to_clean <13>
Mar 10 22:09:13 osh-esx-1 vmkernel: buffer_info[next_to_clean]
Mar 10 22:09:13 osh-esx-1 vmkernel: t
Mar 10 22:09:15 osh-esx-1 vmkernel: 0:00:06:49.377 cpu7:4256)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba34:CH:0 T:1 CN:0: Failed to receive data: Connection reset by peer
Mar 10 22:09:15 osh-esx-1 vmkernel: 0:00:06:49.377 cpu7:4256)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000001 TARGET: workstation-disks TPGT: 1 TSIH: 0]
Mar 10 22:09:15 osh-esx-1 vmkernel: 0:00:06:49.377 cpu7:4256)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 10.10.10.5:56632 R: 10.10.10.1:3260]
Mar 10 22:09:15 osh-esx-1 vmkernel: 0:00:06:49.377 cpu7:4256)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba34:CH:0 T:1 CN:0: Connection rx notifying failure: Failed to Receive. State=Online
Mar 10 22:09:15 osh-esx-1 vmkernel: 0:00:06:49.377 cpu7:4256)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: 00023d000001 TARGET: workstation-disks TPGT: 1 TSIH: 0]
Mar 10 22:09:15 osh-esx-1 vmkernel: 0:00:06:49.377 cpu7:4256)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 10.10.10.5:56632 R: 10.10.10.1:3260]
Mar 10 22:09:15 osh-esx-1 vmkernel: 0:00:06:49.377 cpu7:4256)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba34:CH:0 T:1 CN:0: Processing CLEANUP event
Mar 10 22:09:15 osh-esx-1 vmkernel: 0:00:06:49.377 cpu7:4256)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: workstation-disks TPGT: 1 TSIH: 0]
Mar 10 22:09:15 osh-esx-1 vmkernel: 0:00:06:49.377 cpu7:4256)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.10.10.5:56632 R: 10.10.10.1:3260]

At which point I would then get things like this repeated a lot:

I am going to post the whole log at the end of this post because I found searching for specific chunks of log helped on Google.

I discovered that I could accelerate the issue by starting a second VM. When running one it was just fine. After starting the second and seeing this error log output, I noticed that I could no longer vmkping my iSCSI Target, and I could not ping the iSCSI initiator from the target.

I played around with physical stuff, and even changed the VMnic from E1000 to VMxnet2, but the error persisted. I came back to the logs. It took me a while to get to the 'you turn on the 2nd and it breaks' thing because it wasn't instantaneous. I was trying to install 2008 R2 onto the 2nd VM I was starting up, and it kept failing right around 42%. That clued me into the idea that it was a time thing.

Anyways, since I'm using StarWind iSCSI, I decided to check the settings. Sure enough, deleting the target and re-creating with MULTIPATHING checked off has resolved the issue. I have no clue why that was not checked in the first place.

So there you have it...if you're seeing these log errors, you've not set MultiPathing up for StarWind, or if that's not the case, I'd wager it's got something to do with cache/bandwidth running out.

Edit...I'm not really sure how to re-size the blog width...the vmkernel log looks ridiculous, and excessively long, posted plaintext. I'll have to link a TXT file or something.

Few notes on the home ESX box

2010-03-02T18:45:00.000-08:00

I'm now running ESX within Workstation. It connects via a separate Gb NIC to the iSCSI switch on to Starwind 2TB imagefile. On the other Gb NIC is the LAN/Service console.

Note if you want to use NFS to host your ISOs on the iSCSI server: Completely doable, but if you find yourself getting 'NFS Error: Unable to Mount filesystem', it's probably because you either don't have the NFS Client firewall rule enabled, or you haven't added a VMkernel to your LAN vSwitch. I just added one with an IP address on the LAN's subnet. Did that, works like a charm now!

Oh yeah, you need root access enabled in the NFS properties on the server that is hosting your NFS share.

I also had the NUMA CPU error on booting up ESX4 inside Workstation 7. The following link fixed it: http://communities.vmware.com/thread/244537 (the post by dmadden)

I've since installed ESXi since this will be a permanent install (also got the NUMA CPU error). Oddly enough, NFS works fine, but iSCSI refuses to see the iSCSI targets. I can vmkping the target IP.

Decided to reboot, found out that there is no shutdown command! See here: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013193

Thank goodness for the fast reboot times of ESXi.

Some ESXi commands related to iSCSI:

esxcli nmp device list
esxcfg-swiscsi -e
esxcfg-swiscsi -s

Update: I'm just retarded. My iSCSI issues lay in the fact that I'm going through another adapter, courtesy of VMware Workstation. Another adapter, that I configured with an IP address. You know...the same IP address I set on the ESX virtual machine.

At any rate, I've run into more issues...whereas before ESX would boot my old VMs, I now cannot create a x64 VM at all, due to VT somehow not working correctly.

Update 2: Actually...if I'd just read the release notes...it says explicitly that you can only run 32-bit guests inside ESX inside Workstation. Sheesh. Well don't I feel silly. First time release notes have been worth something! I'll probably just run everything as VMs in Workstation at this point.

Update 3: Yeah, this won't work. Can't connect to the iSCSI network now! Dagnabbit. I am now officially out of luck. Can't install the 8TB array in this machine even if I wanted to...I'd need a board with 1 x8 and 2 x16 slots. Honest to goodness.

On servers and racks and stuff...

2010-01-01T09:23:00.001-08:00

For goodness sake order a DVD-ROM in the least. We have several 2950s and 2850s that have CD-ROM drives and are utterly useless for ESX installs. I'm now praying the DRAC card's ISO mount works. If not, big trouble!!

Also, a few other points off the top of my head:

Order less DIMMs, but higher capacity, even if you have no plans to expand. It will happen, and you will throw out a ton of 512MB/1GB/2GB sticks of FB-DIMMs that nobody else wants for this very reason.
Order a ILO/DRAC/etc card. Just do it. It will save your bottom later.
Only use an x64 OS if you're doing a native install, otherwise use ESX or something similar. Just doesn't make sense to have single-purpose servers in this day and age unless you're talking about an absolute monster of a SQL server or something similar.
Strongly consider an expansion NIC card (dual port at least) if you do not have Intel NICs on-board. Just a good redundancy strategy.
Don't skimp on PDUs in your server rack - get a good quality network accessible zero-U PDU that has plenty of outlets. 1U servers with dual-PSUs can add up to a lot of needed outlets.
For goodness sake use a cable colour scheme. We just started doing this and it is fantastic. We use purple for POE/phone, blue for data, white for switch/router interconnect, orange for iSCSI, and red for internet-facing cables. Yellow would be DMZ if we had any.
Monoprice.com for CAT6/power cables. Buy extras as some come DOA. The price can't be beat.
Plan ahead with your cabling methods. Do a dry run if you have the opportunity - it could change your mind for the actual installation.
1 foot patch cables - just try it. Patch panel rack looks like this: 2U 48 port patch panel, 2U horizontal cable management, 2U (1 48 port GbE switch, with room for a second). Alternatively you can do 2U patch, 2U mgmt, 2x48prt switch, 2U mgmt, 2U patch, pattern repeats. See www.neatpatch.com for inspiration. It looks fantastic, leaves your vertical management spaces nice and clean, and makes tracing cables super easy. It also means you don't need to label every cable, and makes the longer cables clearly defined in the rack.
Honest to goodness, have a wiring diagram of the building posted in the server room. It should label every ethernet port, power outlet, and cable trunks. Relevant power information (max circuit sizes, UPS capacity, etc.) should also be included.
Power drops should be labeled and load monitored for each circuit. Also, if you have a dual PSU system, each PSU should go to a separate PDU which in turn are on separate circuits, and if you're able, separate UPS and power sources. But for the average admin, separate PDUs on separate circuits is standard. Keep a tally of what's on what PDU to get an idea of load on each circuit.
No more than 80% load on a circuit.
Switches - when you get to 80% used ports, buy another one already. We noticed that we were using a lot of iSCSI ports. What we didn't think about was how many ports each server really needed. Dual quad-port NICs on each server really starts to add up.
Network monitoring - have some mechanism for checking network stability. We recently had some DHCP servers freak out on us with massive broadcast storms of DHCP server packets. It took down the VOIP phones really fast, and took hours to isolate.
VOIP phones - give them a separate VLAN in the least. Just keep the packets isolated from the data network as best possible.
Velcro - buy it in large rolls. I found some in .75" x 25 yard lengths, UL rated, for around $20CAD each.
Cable labeling - when needed, do it right. Get self-laminating labels, be it laser printed, or a handheld (just got a Brady Idxpert, and it's very nice). Label the cable with two pieces of data, the port, and a brief description of the function. For example, our fax line cable label says: V2-047 FAX-3010 (the pipe denotes a new line). FAX-3010 tells me it's a fax line, and for the number ending in 3010. V2-047 indicates it's a voice port, 2nd floor, port 47. The port label system was already there from the company we used to run the cables. Again, note that only long runs of cable need labels if you use the 1-foot patch panel cable method.
VMware VCenter. It may look like a nice way to save a few grand, but going with multiple Essentials packs (limited to 3 hosts per VC) versus getting one VC license is NOT the way to go. Extremely annoying, and hard to manage when moving VMs around.
Think long and hard about using ESXi in the corporate environment. It lacks a lot of features that make simple things like changing datastores turn into acts of glacier-like time.
Document any and all procedures into a knowledge base...even though you think it won't be needed again, it will.
When cabling, do a neat, professional job - show some pride in your work! Label appropriately, and with the right types of labels. Nothing looks more awkward than labels peeling off. If possible, comb bundles of cables and velcro every 18" or so. Only use zip ties for permanent spots like trunks attaching to vertical management, even then, try to avoid them.

ESXi storage migration

2009-12-31T22:50:00.001-08:00

Figured I'd post something useful about this experience.

Really, it's all about planning. Not just having a plan, or even having a plan that covers the basics, but having a plan that allows for practical things, like time off, shipping errors, configuration oversights, and just plain bad luck.

We've had all of the above and then some on this latest project. No matter how hard we tried to make our deadlines, we either forgot a key point (ESXi doesn't do storage migration on its own), or were given misinformation or not enough information (ET RS-16 IP-4 doesn't handle jumbo frames including/past 8000), or just plain bad luck abounded (the crux of the project lay across a perfect storm of not one but three long weekends (Christmas, Boxing day, New Years) compounded by different locations having different ideas about which days to take off.

I've spent my New Years Eve (well, 95% of it) on one task: Migrate the storage of all the VMs on our NY site's ESXi box from local to an iSCSI server we just shipped down and arrived in the nick of time. Simple enough, but man we messed up on some basics.

Now granted, we're a bit overworked lately, but dood, we both were in agreement that I just had to storage migrate from one datastore to the other. Simple! Except ESXi cannot do storage migration, as that is a VCenter function (could be ESX as well). So I tried FastSCP. Nothing fast about it, and it also destroys any thin-provisioned disks. Big bada boom.

The fix? Well, we have a great candidate for a VCenter host - that physical server hosting the iSCSI targets! Except it doesn't have the VIM ISO. FTP that over, 1.2 hours. Install Daemon tools, reboot. Try adding host to VCenter after install, fails. ESXi needs patching. Patch, it reboots. Added host. Works! Started moving over storage, working like a charm. So we're okay at this point...

The last piece that's weighing on my mind is that one disk has failed in the iSCSI array (an MD1000). Local people can't pick one up, so I have to get one when I arrive in NY on Sunday, Lord willing.

Ok time to go enjoy the last bits of New Years and go to bed.

Last thought of the evening: Backlit keyboards should be mandatory. Thank you Apple for forcing Dell on the bandwagon.

Edit: The VPN died shortly after posting this, around 1am. Had to give up after spending 2 hours messing about trying to get back in. D was able to get in via logmein and reboot the VPN server, but it's still acting up. I managed to get back in via VPN this morning and set up logmein straightaway.

MOSS on a fresh 2008 R2 sever - error on install

2009-11-03T17:18:00.000-08:00

Trying to install MOSS on an 2008 R2 server, fresh installs each.

Got a "This program is blocked due to compatibility issues" with references to KB article 962935.

Here's the fix, courtesy of Thibault B.

Slipstreaming SharePoint SP2 into SharePoint 2007 Server with SP1 does the job when trying to install on a fresh Windows Server 2008 R2. Basically all you have to do is the following:
Extract OfficeServerwithSP1.exe by executing "OfficeServerwithSP1.exe /extract:C:\yourpathsp1\"
Extract officeserver2007sp2-kb953334-x64-fullfile-en-us by executing "officeserver2007sp2-kb953334-x64-fullfile-en-us /extract:C:\yourpathsp2\"
Copying all files from "C:\yourpathsp2\" to "C:\yourpathsp1\Updates\"It worked like a charm for me :)

From here: http://social.technet.microsoft.com/Forums/en/sharepointgeneral/thread/c7091bda-867e-49a1-9bc8-c2ef847c92e7

Neat trick when making outside IP changes

2009-10-30T19:28:00.000-07:00

The guy I'm working with for our router changes has a neat trick for insurance against a bad IP change.

Run this command prior to making the IP change on your internet-side interface:

reload -n XX (where XX is the amount of time to hold the reload)

Then run your IP change commands.

If you make a mistake, or have bad info, the router reboots itself after XX time, clearing the bad commands!

Clever use of the reload command - there's something you wouldn't learn in school!

Clean-up follow-up

2009-10-27T10:11:00.000-07:00

...too many ups...

Did a massive amount of clean-up yesterday, and the office looks great now!

I've ended up moving the monitors further down the desk, and setting the PC up so that it's easy to un-plug/tidy cables. The UPS is now on the desk next to it as well, leaving nothing on the floor. The fourth monitor has also been removed, for now, as it was just used for a second PC.

I'm going to P2V the BSD box, if possible, so that we don't have another PC sitting around NOT running ESX. There will be two spare PCs if we can get the BSD box virtualized.

Tonight I'm going to clean up all the spare parts, organize them, and label the boxes, as well as take a quick inventory so that I can look into getting rid of some of it. Of course I also need to finish up the basement wall...going to be another late night!

Oh yeah...I've now also got to build a shelf for the computers...or pile them all up together.

Update: I've got most of the stuff cleared out of the office now, and almost all of that sorted/catalogued. I just need to find a good spot in the basement for it.

My next big project is to virtualize my freeBSD mySQL/Asterisk test/dev box. So far Mondo Rescue is looking like the best option (http://www.mondorescue.org/), as it has a native capability for BSD. My only catch is that I'm really not familiar/good with installing/using Linux-ey apps. Hmm.

Where to put it all?

2009-10-25T21:07:00.001-07:00

I was telling my wife the other day how back in college, or earlier, I would have killed for the setup I'm using now - only now, I'm finding it extremely cumbersome and prone to making my desk very unusable.

The desk measures 6x3' and is crammed to the gills with stuff - NICs, RAM, the odd HSF, headphones, CD spindles, screws & drivers, papers, four monitors, speakers, two 8-port switches, wireless router, two keyboards w. mice, and cables upon cables. No matter how often I clean it off, it gets stacked with stuff again in no time. I think the real issue is that I'm just not organized, and have no place to put things even if I was organized. The other issue is that there's a lot of junk! It spills over onto the floor, under and around the desk (where the two APC 1200VA UPSes live). This drives my wife crazy. Actually...I think all of it drives her crazy. I should do something about that...if for nothing else than to make her life a little more sane!

It doesn't help that new computer 'stuff' just keeps appearing...I may want to keep the two additional PE4600 chassis out in the garage for now... :)

Oh yeah...there are also two Antec P180's and one PowerEdge 4600 converted for ATX usage sitting on the desk, with two more standard-size cases looking for homes currently sitting on my wife's desk (it's okay, she's out of town - when she's back...not so much okay).

I've taken a 'before' picture, so hopefully before she's back I'll have an acceptable 'after' picture.

New home server environment

2009-10-25T19:59:00.001-07:00

Well, with the days off I took, plus a few extra bits of hardware, things are shaping up nicely.

Hyper-V server
Motherboard: Asus P5E-VM HDMI LGA775
RAM: 2x2GB & 2x1GB Mushkin PC2
CPU: Intel E6400 2.13GHz
Disks: 160GB Seagate SATA for boot, 8x1TB WD Black in RAID5
Cards: Dell-branded LSI 8408 Dual SAS w. SAS-SATA breakout cables, Intel PRO/1000 CT nic

Running 2008 R2 x64 with Hyper-V and the Starwind iSCSI target. The RAID5 array is broken down into three 1.8TB and one 1TB logical disks, the former being used as VMFS iSCSI datastores and the latter using NFS and SMB shares for ISOs, files, etc. The VHDs for the Hyper-V part of the server are also stored on this disk.

The Starwind was super-easy to set up, only hitch I ran into was that I needed to restart the Starwind/iSCSI initiator services to get ESX seeing the LUNs. Performance is 'just fine' so far - nothing empirical to report, yet.

The on-board nic is used for LAN access, while the CT card is used for iSCSI, each on their own separate Gb switches (a Dell 2708 and D-Link Green, both 8-port). I had to disable the Hyper-V vSwitch to get LAN access working properly...so may need to tinker with that yet. Also, had to disable the firewall to get iSCSI working, even with the proper in/outbound rules enabled. I'm guessing there are some ports needed that are not documented or somesuch. Both ESX and Hyper-V have 3260 open, so not sure what else is needed.

Still trying to decide what to do with the Hyper-V box...for sure a VC is going on there, not sure what else yet. 6GB doesn't give me a huge amount of room, especially with a VC in the works that'll need 2-3GB of RAM just for itself.

ESX vSphere server
Motherboard: Asus P6T SE X58 LGA1366
RAM: 6x2GB G.Skill DDR3-1600
CPU: Intel i7 920 2.66GHz
Disks: 36GB SATA Raptor
Cards: Two Intel PRO/1000 CT nics

Running ESX 4.0 (or vSphere, as it's being called). I've not separated the service console from the VM Network, as I needed the second nic for iSCSI, and due to the strict HCL, I cannot use the on-board nic. Still need to get the VC operational.

Plans are to run a 2008 R2 domain for MCITP purposes, as well as a few BSD/Asterisk machines for some other work I'm doing.

I'll try and keep more posted here in future, I've been lax about it as of late.

2008 R2 troubles continued, but solved

2009-10-25T19:51:00.000-07:00

Once I got the disk portion working, it failed at the 'Copying files' part, got to between 15 and 40%, then gave an error (0x80070750 - files missing). I burned another DVD at 4x speed, same error. I re-downloaded the ISO, checked the SHA-1 hash, and burned another DVD at 4x speed. Same error. I made one of my USB sticks an install disc (http://myitforum.com/cs2/blogs/bbird/archive/2008/10/01/making-a-usb-stick-bootable-for-vista-or-server-2008.aspx , an excellent post, btw), and STILL the same error.

At this point I was pretty frustrated. There was no way it was install media-related...so back to the basics. I pulled all the expansion cards and did more googling. I came up with someone mentioning that over 4GB of RAM caused their Win7 install to fail. Win7 and 2008 R2 are similar, so I pulled 3 of the 6GB RAM I had installed (2x2,2x1). Bingo!! Installed 100% from there.

Pretty dumb problem...especially since it's an x64-only OS.

The hardware I'm using is most likely the culprit: Asus P5E-VM HDMI with the 0604 BIOS

At any rate...all good now.

Dumb Win7/Server 2008 bug - BIOS settings

2009-10-24T21:29:00.000-07:00

I was trying to get a new frankenstein server up and running - boot would be a 320GB SATA drive off an ICH9R chipset, pretty standard stuff. The OS I wanted to install was Server 2008 R2 x64. I kept getting 'setup was unable to create a new system partition or locate an existing system partition' errors at the disk stage of things, even when using the advanced tools to delete/create new partitions. I even tried another SATA disk.

The solution?

Some googling turned up this: http://social.technet.microsoft.com/Forums/en-US/w7itproinstall/thread/8c2c1ac9-dc18-40af-a2b1-88dfaf9c2e70

The part of that thread that worked was BIOS settings. I am not joking. I had set the BIOS settings to only boot from the CD-ROM, nothing else. Changing the BIOS settings to have the 320GB disk as primary boot with the CD-ROM as secondary gave no issues at the disk portion of the installation.

Retarded, and made extra frustrating by the long waits caused (presumably) by the LSI SAS RAID card doing scans. Argh. I hope this helps one of you get by this problem faster than I did!

More than 6TB for ESX4.0

2009-10-12T06:35:00.000-07:00

Note: I was doing some cleanup of old posts, and noticed this was never published. Figured I'd throw the draft up anyways.

A bit of an issue with ESX4.0, but a little intro first.

I'm trying to run ESX vSphere on a home setup. An LSI 8408E (PERC 5i) SAS/SATA card with two SAS-SATA breakout cables connects to 8 WD 1TB 7.2k SATA disks. I also had one drive DOA initially, and have now had it replaced, but I'm still hearing sounds of head access that is too consistent for my liking.

They are configured as RAID5 with four logical disks, because the initial 6.4TB logical disk could not be used by ESX due to the limitation of 2TB per disk. After changing this configuration, ESX locked up at the 'checking filesystems' step of the boot process twice (only a few hard resets and mucking with the BIOS got it to clear finally). Now that I'm finally back into ESX with my new 2TB partitions, I cannot add them as disks in my storage configuration!

I get errors pertaining to partitioning when trying to add disks.

When I try to ssh in to the console, su - to root, and then run fdisk -l, it locks up. I waited a good 5+ minutes, but no change. I can only assume the disk controller is having issues. As an additional hardware note, I've re-flashed the card to the latest available ROM from LSI - upgraded from v5 to v7!

I rebooted the host in the hopes that the ssh commands would now work. It seemed to get stuck on the store-logs step of the boot process. After five minutes of this, I ctrl-alt-deleted for another try. Got past it this time, and I noticed briefly that it listed the names of the disks after the 'checking filesystems' step - sdf something in this case.

Ok, so I gave up on ssh and just used the physical console, since it was right next to me and all...fdisk -l gave a huge list of stuff instantly this time.

/dev/sda = 2199.0 GB - LSI 2TB partition

/dev/sdb = 400.5 GB - LSI leftovers partition

/dev/sdc = 37.0 GB - This is the boot 36GB Raptor used for the ESX install.

/dev/sdd = 2199.0 GB - LSI 2TB partition

/dev/sde = 2199.0 GB - LSI 2TB partition

/dev/sdf = 8095 MB - Not sure about this one...

At any rate, I've done the fdisk partition creation and rebooted - since it was not the console drive, it did not ask me to reboot, but another one could not hurt...right?

Upon reboot, the 'starting udev' step of the boot process took some time, as did the 'checking filesystems' step - presumably because of the 6.4TB-worth of new partitions.

I have to step out at this point, however. More to come.

...

Well, came back to the console, so must have just taken a bit longer than usual. I'm pretty sure something is up with my array at this point, because it was not this slow prior to the array being set up. Anyways...I digress.

The second link in my sources shows a correction for the error in the first link. It's not vmkfstool, but vmkfstools, as well as /devices, not /device. At any rate, using another command from the first page's comments (esxcfg-mpath -l), we can see what disks we're dealing with. For a clearer version of what this command essentially lists, under the MUI->configuration->storage->add storage->add disk, you can see a list of what disks/LUNs you have available. If you click on one and hit CTRL+C, you get this:

Local LSI Disk (naa.600605b000175dd01264d94cc2a2d019)

vmhba0:C0:T0:L0

2.00 TB

Hiyo! Something to put in our vmkfstools command!

[root@localhost disks]# vmkfstools -C vmfs7 /vmfs/devices/disks/naa.600605b000175dd01264d94cc2a2d019

Creating vmfs7 file system on "naa.600605b000175dd01264d94cc2a2d019" with blockSize 1048576 and volume label "none".

Usage: vmkfstools -C vmfs3 /vmfs/devices/disks/vml... or,

vmkfstools -C vmfs3 /vmfs/devices/disks/naa... or,

vmkfstools -C vmfs3 /vmfs/devices/disks/mpx.vmhbaA:T:L:P

Error: Permission denied

(I tried the vmhba:C0:T0:L0 in place of naa, but it errors out)

Okay...permission denied even though we're logged in under a valid user and we have elevated to su -. Weird. Maybe it just doesn't like vmfs7. Tried again - nope, still denied (turns out vmfs3 and 7 are really differentiated when creating vmdks, not the bare filesystem). Alright, man vmkfstools. I'll add in -v for verbose.

I'm still getting the 'tpm ... failed to load tpm' error on booting ESX as well... something else to troubleshoot and fix at some point. ( http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011452 )

Sources:

http://communities.vmware.com/thread/98477

http://blogs.egroup-us.com/?p=33

Offline files for Windows 7

2009-08-25T12:03:00.000-07:00

Random weird problem day - Tuesday.

We're trying to get offline files and folder redirection working through Group Policy (GP henceforth) for our Windows7 and/or Vista clients (ok, just 7, but we did have a random Vista desktop to test with). Our issues are that a) the settings that work for XP do not work for 7, and b) some settings show up in RSOP, but don't actually work.

Symptoms: Client refuses to synchronize, or even see that it should be synchronizing certain files that we've redirected from the file server. In GP we had the same settings used during the XP reign, located under the 'Computer Configuration' side of things.

Fix: In GP - User Configuration, set the 'Administratively assigned offline files' to Enabled and put in the Value Name field: \\server\usershare\%USERNAME%\

Ensure all your other settings make sense. For our part, we set this up under the user policy, and under the laptop or desktop policy, we enabled (laptop) or disabled (desktop) offline files.

In hindsight, it makes total sense: Your user specific settings, such as which folders to have offline, must be set in the User Configuration side of things, while other settings such as whether or not the computer itself uses offline files are set under the Computer Configuration. So simple!

We have the following settings in place:
Desktop policy
CompConfig\AdminTemp\Network\Offline Files\Allow/disallow use of offline files (disabled)

Laptop policy
CompConfig\AdminTemp\Network\Offline Files\Allow/disallow use of offline files (enabled)

User policy
UserConfig\AdminTemp\Network\Offline Files\Admin assigned offline files (enabled w. the above settings)

There are others, of course, but these are the ones that allow laptop users to have offline files and desktop users to not (awkward wording...yeesh), while still allowing us to only use one GP for our users, and not have to differentiate between the two. This is super-great, because users that have a desktop AND a laptop now don't have weirdness when moving between the two. Nice!

weird ESX networking issues

2009-07-16T08:31:00.000-07:00

Problem: Three VMs are dropping network connectivity

SQL4
SRV_GP
YYZ-MAIL-1

(200 packets sent over an hour or so)
SQL4->gateway : 100%
SQL4->service console5: 100%
SQL4->10.1.1.2: 100%
SQL4->YYZ-MAIL-1: 93%

(75 packets sent over 20 minutes or so)
10.1.1.2->gateway: 100%
10.1.1.2->service console5: 100%
10.1.1.2->SQL4: 73% and dropping
10.1.1.2->YYZ-MAIL-1: 100% so far

(after testing like crazy)

On the advice of the VMware rep (well, something he said might fix it), we tried separating the service console vSwitch from the LAN vSwitch, giving the LAN one nic and the service console one nic, and then rebooting. My understanding was that having the service console and LAN on the same nic was supported in 4.0. Guess not.

So far, all the servers we fixed are 100% fine, but that could be due to the reboots. We'll have to wait for 2-3 weeks for verification on that - it was about two weeks after we got set up that the issue appeared, and after some changes, it re-appeared after two weeks again. So...second round of changes have been done, meaning another two weeks of things working. If there have been no re-occurrences after a month, I'll be satisfied.

New ESXi server

2009-07-13T12:47:00.001-07:00

Well, the parts arrived today! Got it all set up and kitted out as well.

Specs:
- Ci7 920 - 2.67GHz - 12GB RAM - Scythe Ninja.B HSF
- Intel Pro 1000 CT
- 36GB Raptor for the ISO datastore & ESXi installation
- 1x1TB, 2x160GB, and 2x320GB SATA for VMFS
- ESXi 4.0

I would have added more hard disks, but I ran out of ports and drive locations! I'm going with the multi-disk setup like this because I can't do RAID with that many un-like disks, and at this time could not afford the RAID card even if I had the disks. By spreading out the VMs to say two per spindle, I should be able to get acceptable performance.

My goals for this ESXi installation are:
- Personal web server for WWW, FTP, media, etc. (hopefully host the blog here one day)
- Domain for home PCs
- Server 2008 sandbox environment (furthering MSCE goals and general knowledge)
- Development VM for asterisk training

The specs should be up to the job, and it'll be fun setting all this up.

edit: Also, I really want an easier way to include photos/movies with the blog. That'll be another project.

Word Test

2009-07-08T15:34:00.001-07:00

This is a test of publishing in word.

Edit: Interesting HTML tags on it, and it seems I need my own server to set up pictures. Well, good thing we have a static IP! Just need to get the DNS records moved...

ESXi box for certificate/study purposes

2009-07-06T08:06:00.000-07:00

Well, the AOpen P4 box just can't cut it anymore. I booted it up after network connectivity was lost, and suddenly ESX doesn't see any hard disks. Lovely.

Time for something reliable - won't be able to learn much on something that is made by AOpen. Or run more than 3-4 VMs on 2GB of RAM.

If I want to use thin provisioning, I need to upgrade the NIC to a proper Intel pcie gig nic. While I may be able to get away with that, I'm still stuck with 1 core and 2GB of ram...with a max of 8GB! Not that much anymore, not when you can have 25 VMs running on a single quad core with 16GB....

I'd like to be able to run at least a dozen Windows Server VMs at a time, and that means a bunch of RAM is needed. In my experience with ESX, it's the RAM that gets sucked up first, and the CPU:RAM ratio is usually 1:3, or more.

So, it makes sense to me that the Core i7 architecture is a good starting point.
- has RAM capabilities up to 24GB
- LGA1366 socket, relatively new, could have 8 core CPUs compatible in future
- LGA1366 is the same socket for the new Xeon 5500s, which are excellent for VMs
- Bloomfield CPUs are supposedly very good on energy bills
- lots of PCIe slots, should I want to set up storage controllers or nics

Well, that pretty much settles it, unless I can get two Core2Quad systems for the same price. Even then, at that point we're into double the amount of PSUs, cases, HDDs, etc. I'll post more on this later.

Later:

Here's pretty much what I've decided on:

# Asus P6T SE
# Core i7 920 2.66/8MB
# Scythe Mugen 2 HSF
# Corsair XMS3 (6x2GB)
# Seasonic S12II 500w
# Intel PCIe gig nic (Here's the one I'll need)

Seems the Gig nic is going to be the hardest to source (after that Supermicro X8SAX board, of course). ESXi 4.0 only supports a bare few nics at this time, and the Intel CT series is one of them.

Same issue we had with the quad nics we got - first ones in Canada, and VMWare didn't support them! Dan, the genius that he is, recompiled the VMWare drivers with the new driver info and got that working - a week or two later they released drivers for the card.

Update:

Purchased!
CPU: i7 920 2.66/8MB Bloomfield
RAM: G.Skill 6x2GB
MB: Asus P6T SE
HSF: Scythe Ninja rev.B
HDD: WD Caviar Black 1TB
NIC: Intel Pro CT gigabit

And some AS5 for the HSF. I'm running out after doing the ESX servers at work!

I'm also picking up a Dell 2708 - $50US+ shipping...not bad! Cheaper than one off NCIX, that's for sure.

migration update

2009-07-03T07:14:00.000-07:00

Well, got lots of sleep!

After all that hassle, they really didn't point out too much. RPC was timing out due to a few packets not making it to the destination - okay, interesting, but how come everything else was working 100%? Why only RPC for replication between 2008 DCs? What about the fact that 2003 DC replication works 100%?

Clearly either there is a bug with 2008 or the Cisco routers are filtering certain packets during replication, i.e. a filter template just needs updating by Cisco.

However, we have no time to figure out those problems, not now. We're moving all our DCs back to 2003.

Dan dug up this link: http://www.experts-exchange.com/Software/Server_Software/File_Servers/Active_Directory/Q_24423610.html

Indicates that 2008 replication to multiple sites has issues, while replication with no sites configured works just fine. We haven't had a chance to test this yet.

Exchange migration

2009-07-01T05:25:00.001-07:00

4:00am (11:30pm start)
Not going well so far.

Problem: The US exchange server refuses to route emails to the CAN exchange server.

Here's what we know:
- They are in the same domain.
- They can ping each other using IP and DNS.
- They can telnet to each other on port 25.
- They can send a message via SMTP commands across the VPN connection that arrives in a recipient's mailbox successfully, verified from both sides.
- Active directory replication has been troublesome, but seems to work, for the most part.
- Tracert functions correctly - three hops, one local, one *s, one destination - so the VPN is functioning correctly in that respect.
- A share can be mapped from one server to the other.
- AD domains and trusts becomes unresponsive when opened - RPC issues
- dcpromo fails at replication (times out) - RPC issue
- SCCM 2007 installation completes with some failures - timouts - RPC issue

11:30am
We call Microsoft and are assured contact soon.

2:30pm
Unfortunately Microsoft has been lax to get back to us with an appropriate contact. The first guy didn't really have a clue, and then he said it would be an additional 1-2 hours for someone else. After 2.5 hours, I called them back and asked for someone local. No time for communication issues today.

3:00pm
Yeesh. Said that someone should be back to me in 20 minutes...that was 30 minutes ago. It's now 2:50pm...I'd kinda like to enjoy some of Canada Day.

8:16PM
Tech support guy has gone on lunch for 15 minutes.
They made some registry changes, after hours of "'replicate now' ... failed". That got kinda boring. They were also using this a lot: repadmin /Syncall /e /P

They tried this a bunch as well: netdom /resetpwd /server:DC1 /userd:domain\admin /password:*

HKLM\System\CurrentControlSet\Services\TCPIP\Parameters
- new DWORD - DisableTaskOffload - Decimal - 1
- new DWORD - MaxUserPort - Decimal - 65535

HKLM\System\CurrentControlSet\Services\DNS\Parameters
- new DWORD - SocketPoolSize - Decimal - 2500

HKLM\System\CurrentControlSet\Services\LSA\Kerberos\Parameters
- new DWORD - MaxTokenSize - Decimal - 65535
- new DWORD - MaxPacketSize - Decimal - 1

They then had me restart each DC sequentially, allowing full boot-up before proceeding.

I launched dssite.msc on each DC, and things were looking better, but for the US DCs, the CAN DC1 (PDC) NTDS settings are blank. Not sure

12:00am
I finally ask why this is taking so long with no real results, and speak to a 2nd level technician, who is in fact a 1st level technician, and just scripts the front-line people to do his bidding. Essentially this is a known flaw with Server 2008, and they are working on a patch. He tells me that if we set up two 2003 DCs, one on each site, then the issue will be resolved. I tell him to call me back in 15 minutes.

12:40am
Still no word. Calling the tech line. Again.
Guy was very apologetic, but he had no control over where the tech support came from. Said it was all coming from India at this point. Lovely. He said he would escalate it as my case was priority one at the moment.

12:55am
No word yet. In the mean time, I've just started working on it again on my own, but looks like replication is failing anyways. Right off the bat I got an RPC error, so I changed the DNS servers to the YYZ-DCs, and I managed to get to the replication part of the dcpromo for the JFK 2003 DC. It is, however, taking a long time at this point, so pretty sure it's timing out. That normally takes about 5 minutes...so...we wait.

5:35am
So tired.

This guy I just spoke to actually knows what he's doing, and is methodical and takes notes.

He'll be emailing me the results of his analysis in a while, so I can at least get a few hours sleep before the day starts. He has come to the conclusion that is could be:
- Packet loss causing network timeouts, specifically RPC packets being lost
- Symantec Endpoint
- bad switchports
- corrupt vnic, said we could try adding a second one, giving it a new IP, disabling the old one, and updating any relevant DNS records.

Argh. We'll see.

SCCM 2007 install

2009-06-30T20:39:00.000-07:00

Seems to have a lot of dependencies!

Installing on a bare-bones 2008 x64 VM, and it requires:

- Schema extensions (warning)
- WSUS SDK on site server (warning) (needs WSUS, IIS (app/web), .net 3.0, SQL 2008 SP2, Report Viewer 2005
- MS Remote Differential Compression library registered (Features)
- SQL server sysadmin rights (needed to add the current user to the SQL installation with sysadmin permisson)
- SMS provider communication
- IIS not running (Features)
- BITS installed/enabled (Features)
- WebDAV installed/enabled (http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1621)

Update: Finally able to finish this. After the debacle with Microsoft and AD replication, it's working great so far. I'll be looking at building the Office2007 package this afternoon.
( Refs here )

Update: Turns out if you don't extend the schema (we were hoping to avoid that), then it just doesn't work as it should. So we've extended the schema, rebooted, and we're trying again. Unfortunately the ccm.log file is not very descriptive, so it's hard to pin down why the client installs are not going through. At this point the ccmservice is installed on the client computers, but it doesn't stay running more than a second or so (by design, I figure), and no client install completes - I'm judging that by the control panel, which doesn't have the config options yet.

Well, the errors we've been seeing are HTTP (404, 500, 403, 405), and a webDAV error as well.

!--[LOG[Failed to correctly receive a WEBDAV HTTP request.]-->time="23:15:30.138+240" date="07-03-2009" component="ccmsetup" context="" type="3" thread="2312" file="ccmsetup.cpp:6064">

!--[LOG[Failed to successfully complete HTTP request. (StatusCode at WinHttpQueryHeaders: 405)]-->

Dan opened up the IIS permissions, and we got as far as the aforementioned two errors, but that's it. The 405 error didn't seem to be in the IIS log.

We are at the point where installing this on a 2003 machine is looking tempting.

Hmmm..I installed WebDAV but didn't enable it. Now we're getting:

A new HTTP error. 401.

More errors, but different!

More IIS fiddling...

It was missing a virtual directory to point to, and even though IIS users were allowed, wasn't working, so we allowed everyone.

Dan also had to change the client.msi to an application...not 100% sure about that one.

Client install worked, but client/server cannot see each other. Turns out the site was not published (under site management), in either DNS, AD or anything. It has been published now, so we'll see how long this takes.

Update: Well, the pushing to clients failed on a bunch of them, so we ended up just setting it up via the package manually. Not a complete failure, but still not as nice as I'd have liked it to be.

The lesson is: SCCM 2007 is COMPLICATED, and not an afternoon installation kind of product!!! Read through the technet articles more than once...print them out if need be!

Sheesh.

Emergency migration!

2009-06-30T20:31:00.000-07:00

Got a few calls/emails missed while I was out at Transformers 2 - the Exchange 5.5 box filled up!! it reached its 16GB limit, and shut down all mail functions...sheesh! So now the migration is happening now...11:35pmEST, instead of Friday morning. Jeepers!

So instead of having 2-3 days to set up SCCM 2007 at their site, I have hours. Ah well, gotta have some excitement!

Exmerge and NT

2009-06-30T10:16:00.001-07:00

So, lesson for y'all!

When you are running exmerge on an NT box with Exchange 5.5, here's how it works:

You give it the list of boxes you want out, and it then exports each box to a PST file wherever you want it to go.

The hidden catch is: It removes the data from the Exchange store, and creates a PST file! So say you just want to test the utility, but not do anything, while it's functioning users are losing Outlook data left right and centre! Crazy fun times.

Also, when you try to exmerge a user's mailbox that is larger than 2GB, exmerge crashes. If you restart exmerge, everything you took out prior to that will be overwritten (since you say 'sure, why not start from scratch'), but since you didn't know that exmerge removes the data from the Exchange store, you are effectively deleting all their emails. It's awesome.

So then, after you've tried restarting the exmerge process a few times (deleting all users' emails earlier in line than the 2GB+ user), you give up and move on with a new exmerge starting after that user.

This is fine, except then you find out all the users are losing their mailbox content and decide to reverse the process, and discover the above. Fancy!

Anyways, in summary:

- Exmerge for 5.5 does not allow for duplicate copies of the mailbox, so deletes the source once exported.
- You cannot have a PST file larger than 2GB, even when exmerging.
- Users will continue to send/receive email even though their mailbox has been effectively deleted from the store, making your restore process much more complicated than it needed to be two days out from a major migration.

To note, the 2003 exmerge creates copies when it exports to PST, hence the confusion when this happened.

Network timeouts for VM guests

2009-06-30T05:41:00.000-07:00

This is SO weird. That SQL timeout issue was caused by network timeouts, and it disappeared when we moved the SQL server to another ESX host. However, the issue re-appeared, this time on the host with the vCenter guests...

Further analysis reveals this interesting tidbit:

29/06/2009 12:08:29 PM -- Reply from 10.1.0.41: bytes=32 time<1ms ttl="">
29/06/2009 12:08:34 PM -- Request timed out.
...
29/06/2009 12:09:24 PM -- Request timed out.
29/06/2009 12:09:26 PM -- Reply from 10.1.0.41: bytes=32 time=1028ms TTL=128

29/06/2009 12:38:29 PM -- Reply from 10.1.0.41: bytes=32 time<1ms ttl="">
29/06/2009 12:38:35 PM -- Request timed out.
...
29/06/2009 12:39:24 PM -- Request timed out.
29/06/2009 12:39:27 PM -- Reply from 10.1.0.41: bytes=32 time=1445ms TTL=128

29/06/2009 1:08:14 PM -- Reply from 10.1.0.41: bytes=32 time<1ms ttl="">
29/06/2009 1:08:20 PM -- Request timed out.
...
29/06/2009 1:09:09 PM -- Request timed out.
29/06/2009 1:09:12 PM -- Reply from 10.1.0.41: bytes=32 time=1772ms TTL=128

29/06/2009 1:38:27 PM -- Reply from 10.1.0.41: bytes=32 time<1ms ttl="">
29/06/2009 1:38:33 PM -- Request timed out.
...
29/06/2009 1:39:22 PM -- Request timed out.
29/06/2009 1:39:26 PM -- Reply from 10.1.0.41: bytes=32 time=2554ms TTL=128

29/06/2009 2:08:23 PM -- Reply from 10.1.0.41: bytes=32 time<1ms ttl="">
29/06/2009 2:08:28 PM -- Request timed out.
...
29/06/2009 2:09:19 PM -- Request timed out.
29/06/2009 2:09:22 PM -- Reply from 10.1.0.41: bytes=32 time=1921ms TTL=128

29/06/2009 2:38:29 PM -- Reply from 10.1.0.41: bytes=32 time<1ms ttl="">
29/06/2009 2:38:34 PM -- Request timed out.
...
29/06/2009 2:39:23 PM -- Request timed out.
29/06/2009 2:39:28 PM -- Reply from 10.1.0.41: bytes=32 time=3497ms TTL=128

(it continues)

So every 30 minutes there is a network outage of a minute or so on those guests. You'd say, 'hmm, hardware? cabling?', but NAY! Host pings reveal no timeouts, and other guests on the same host do not have the same timeouts. How weird is that???

Dan suggested that maybe the OS was coming into play, but when the problem initially occurred, there were a mix of 2008 and 2003 clients affected.

I'm going to run a Wireshark session on the LAN switch they are using, see if that comes up with something.

Update: The wireshark session hasn't really added too much to the proverbial pot here, especially since I can't figure out how to get a time range filtered out. Frame.time "" to frame.time "" doesn't seem to work.

Oh well, other issues to deal with for now.

More to report...

2009-06-27T10:58:00.000-07:00

While troubleshooting, I've had the RDP connection drop out on me a few times, while the VM console is still relatively active. After this happening the last time I tried pinging it, and discovered that it was unreachable - timed out. I'm not sure how to get a ping log done with a timestamp without scripting, so for now we're going to try another VMnic on a different adapter, but same network. We'll give it the same IP and disable the old one.

Ok, that's done - new vmnic on a different adapter, but same IP setup. I'm running another ping log. The last one had 5 instances of 10-20 request timed outs over three hours...should know more in an hour or two.

Well it's still dropping the connection. So, we need to narrow the problem down to: VMware or the guest OS. I now have pinglogs going to all the guests on that ESX server, so if I see timeouts on all the VMs, then we know it's VMware, and if no timeouts occur, then we know it's SQL4 being silly.

More to come later...

Well, weird answer. All the guests on that host are getting timeouts, but the host itself doesn't get timeouts. I did a few other guests/hosts on other machines to verify this, and ONLY the guests on that host are having issues. Very strange.

Another oddity was an error message saying the SQL4 server was getting a DOS attack from our integration server, I find it strange that an event like that would not cause physical problems, but I guess in a virtual environment things behave a bit differently.

So, Dan is coming over and we're going to reinstall ESX - shouldn't take too long - and then get the four VMs on that ESX host back up and running, and I'll set up the ping logs once again.

I found this on technet: http://www.microsoft.com/technet/scriptcenter/resources/qanda/aug06/hey0817.mspx

I'm using it for the ping logs now - clunky, until I can figure out how to ping multiple targets to separate log files.

UPDATE!!!!

The issue completely disappeared, which is great! Except that I don't know WHY it disappeared. Very very odd. No more timed outs on any of the afflicted servers. Our SQL box is back to 100%. MESSED UP, MAN. Messed.

More netlogon.log digging

2009-06-26T19:19:00.000-07:00

We have been having GP (on the mentioned SQL4 box) heck over the last week or so, probably caused by all the network changes all at once, but who knows, so I'm digging! Clients are fed up with massive lag, errors, and general unusability. Up until a week ago or so, things were just peachy.

While troubleshooting SQL connection issues, I've come across about 1000 entries in the netlogon.log from the past three days like this:

06/23 23:30:02 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\sqlsvc from EXCHANGE (via EXCHANGE) Returns 0xC000006A

6A, recall, being bad password, and the account shows as locked out in the adlockout tool.

So a service is trying to access something on Exchange from Exchange...weird.

I took a look at one of the SQL boxes, and it uses this sqlsvc account for its SQL services. The security log in SQL shows no failed logon attempts. The event log is showing W32time warnings however - might not be that important - the clock is only off by 30 seconds or so compared to my workstation.
Interestingly enough, there is a database mail section in the log viewer, and there are 'DatabaseMail process is started' and 'DatabaseMail process is shutting down' entries every two hours or so, with varying times between. Does this correlate to the netlogon.log events?

At 14:21:01 there is this:
06/26 14:21:01 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of (null)\orthotic\sqlsvc from SQL4 (via EXCHANGE) Returns 0xC0000064

At 14:20:02 there is a 'DatabaseMail process is started' event on SQL4 using sqlsvc.

Oooh! Very interesting!

At 13:51:05 in netlogon.log I see this over the space of 1.5 seconds for about a page's worth (at 1600x1200):
06/26 13:21:08 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\sqlsvc from EXCHANGE (via YYZ-DC-2) Returns 0xC0000234
06/26 13:51:05 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of (null)\orthotic\sqlsvc from SQL4 (via EXCHANGE) Entered
06/26 13:51:05 [LOGON] ORTHOTIC: NlPickDomainWithAccount: orthotic\sqlsvc: Algorithm entered. UPN:0 Sam:0 Exp:0 Cross: 0 Root:1 DC:0
06/26 13:51:05 [CRITICAL] ORTHOTIC: NlPickDomainWithAccount: orthotic\sqlsvc: Must be either UPN or SAM account. UPN:0 Sam:0
06/26 13:51:05 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of (null)\orthotic\sqlsvc from SQL4 (via EXCHANGE) Returns 0xC0000064
06/26 13:51:05 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of (null)\orthotic\sqlsvc from SQL4 (via EXCHANGE) Entered
06/26 13:51:05 [LOGON] ORTHOTIC: NlPickDomainWithAccount: orthotic\sqlsvc: Algorithm entered. UPN:0 Sam:0 Exp:0 Cross: 0 Root:1 DC:0
06/26 13:51:05 [CRITICAL] ORTHOTIC: NlPickDomainWithAccount: orthotic\sqlsvc: Must be either UPN or SAM account. UPN:0 Sam:0
06/26 13:51:05 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of (null)\orthotic\sqlsvc from SQL4 (via EXCHANGE) Returns 0xC0000064
06/26 13:51:05 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\sqlsvc from EXCHANGE (via EXCHANGE) Entered
06/26 13:51:05 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\sqlsvc from EXCHANGE (via EXCHANGE) Returns 0xC000006A

Horrific. An interesting line:

06/26 13:51:05 [CRITICAL] ORTHOTIC: NlPickDomainWithAccount: orthotic\sqlsvc: Must be either UPN or SAM account. UPN:0 Sam:0

Hmm. At 13:50:04 there is a 'DatabaseMail Process is started' event on SQL4.

13:59:25, hmm...less entries, but from the other SQL server this time...I wonder what runs on SQL3 at this time...

06/26 13:59:25 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of (null)\orthotic\sqlsvc from SQL3 (via EXCHANGE) Entered
06/26 13:59:25 [LOGON] ORTHOTIC: NlPickDomainWithAccount: orthotic\sqlsvc: Algorithm entered. UPN:0 Sam:0 Exp:0 Cross: 0 Root:1 DC:0
06/26 13:59:25 [CRITICAL] ORTHOTIC: NlPickDomainWithAccount: orthotic\sqlsvc: Must be either UPN or SAM account. UPN:0 Sam:0
06/26 13:59:25 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of (null)\orthotic\sqlsvc from SQL3 (via EXCHANGE) Returns 0xC0000064
06/26 13:59:25 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\sqlsvc from EXCHANGE (via EXCHANGE) Entered
06/26 13:59:25 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\sqlsvc from EXCHANGE (via EXCHANGE) Returns 0xC0000234

At 14:00:20, more SQL4:
0 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of (null)\orthotic\sqlsvc from SQL4 (via EXCHANGE) Entered
06/26 14:00:20 [LOGON] ORTHOTIC: NlPickDomainWithAccount: orthotic\sqlsvc: Algorithm entered. UPN:0 Sam:0 Exp:0 Cross: 0 Root:1 DC:0
06/26 14:00:20 [CRITICAL] ORTHOTIC: NlPickDomainWithAccount: orthotic\sqlsvc: Must be either UPN or SAM account. UPN:0 Sam:0
06/26 14:00:20 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of (null)\orthotic\sqlsvc from SQL4 (via EXCHANGE) Returns 0xC0000064
06/26 14:00:20 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\sqlsvc from EXCHANGE (via EXCHANGE) Entered
06/26 14:00:20 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\sqlsvc from EXCHANGE (via EXCHANGE) Returns 0xC0000234

14:21:01, a whole rash in netlogon, and lookee...14:20:02 has DatabaseMail process is started. 59 seconds off again, explaining the w32time warnings I've been seeing.

Probably safe to say at this point that the rest of them are caused by the DatabaseMail process starting. So!! What jobs run at those times? Somewhat frustratingly, there are no records of failed jobs at all of those times, and since every job is run by that user...doesn't make sense that some would work, and others not....unless only some jobs use the DatabaseMail process...

I checked the setup of the DatabaseMail under SQL4\Management, and updated the password. I unlocked the account using the adlockout tool on all DCs, so we'll see how long it takes to start erroring. I'm going to check other features to see if the sqlsvc password is out of date.

Doesn't seem to be set anywhere else...I'll try the services. They look correct - I don't want to change them yet, as they seem to be working fine. 30 minutes and no more lockouts, but I'll probably have to wait until tomorrow to see more.

Signing off for now....

IPv6 and dcpromo

2009-06-26T11:03:00.001-07:00

SO. LAME.

We've been having issues for the last week because dcpromo was failing across the VPN. We thought it was the firewall causing the problem, so we got a completely new firewall - no dice. We then tried a VPN within the site-to-site VPN, and it worked! Really weird. Dan checked the ports using portquery, and they matched up just fine, no errors there.

But when we did a dcpromo from a remote site in, the dcpromo would fail at the replication stage - it would time out and say 'RPC call ended'. Keep in mind this machine was able to join the domain just fine.

So while on hold with Microsoft, we check the error logs, and nothing shows up. Then Dan starts checking the network settings, and BOOM! Lo and behold the DCs in our site have IPv6 disabled, and the server we're trying to bring up to DC status HAS IPv6 enabled!!!

Disabled it, tried the dcpromo again, worked! It was promoted in 30 seconds...used to take 5 minutes to time out.

When the lady came back to inform me that someone would be calling us back, I told her we fixed it. I then mentioned that perhaps the tech would like to speak to us for details on how this happened. She said sure, but if it turned out to be our fault, they would charge us the $300 for a support call. I passed, and posted the info here.

Pretty fancy that a default option would cause something as important as dcpromo to fail!!

New firewall setup

2009-06-25T11:39:00.000-07:00

We've moved our firewall duties over to a Cisco router with security features from our old ISA2006 server. It was getting clunky, buggy, and ridiculous. While this new method of ACLs is slightly more complex, it definitely won't have the issues the ISA server had. We're still maintaining our use of ISA for internal stuff, like the web proxy, and we'll be setting it up for DMZ usage as well, eventually. Silly Cisco router can only have 3 FA ports...even though it has the physical capability to run 6 (2 onboard & 2 twin-port cards). Odd that Cisco would do that...I guess force people into the higher-end routers. (yeesh...base 2801 is $3500...then add on some WICs...up around $4500 or more!)

While it'd be great to run it all off the router, we've already gone over our budget with the SAN purchase, so it'd be best not to press our luck.

It's actually quite nice, this setup. I'm looking forward to getting some info from our Cisco guy on proper ACL setup. I learned it back in college, but that was over three years ago, and I haven't touched any of it since then...gets rusty after a while.

It's a shame Terago was awful about all this. They eventually helped us out, but their level 1 techs are expert stonewallers, and not-so-expert networking professionals. Thank goodness our Cisco guy has dealt with these situations before - he was a real champ! What should have been a 1-2 hour job turned into a 3-4-5 hour job, mostly waiting on Terago to call us back. And then, once they had performed the fix we needed, a typo set us back again! 248 mask is different than a 240 mask.... took us a few to figure out why our VPN on 26 wasn't working, but everything below 24 was working. Sheesh!

SANs and ESX guests

2009-06-25T07:10:00.000-07:00

So, learned a valuable lesson today: If you want to restart a SAN array, FOR GOODNESS SAKE power off any VM guests using the volumes it hosts.

To flesh out the details, we are moving all our volumes off the loaner PS5000 and onto our new PS6000. I'm only seeing one interface being used, so figured that was a config error. Ensured all the eth interfaces were up and had addresses, and spoke to tech support about it. They said a restart of the array might help things.

Well, they didn't mention shutting down attached guests first! I knew that you shouldn't, but it didn't click that our file server was using that volume, and should have been powered off first. I restart the array, and try to move a volume again, but it's still only using one eth interface, albeit a different one this time.

It turns out, from another tech support rep, that when moving volumes the PS doesn't see that as a priority, and therefore only uses one eth interface to do so. Argh!

So I spent the morning cleaning up the disaster that ensued. No file server = no My Docs, and no My Docs = hung logons, can't save files, etc etc etc. Since it was a VM, the ESX host needed to be rebooted, as that's the only way to clear something like this up. I tried to shut down the VM on it's own, but it timed out...after 20 minutes....yeah, a 20 minute timeout. That's a bug!

Anyways, things are up and running now, and lesson learned without too much of a cost. Still kinda feel stupid about it though.

Subnetting trick

2009-06-23T06:04:00.000-07:00

I've not been doing any true networking for quite some time, so the bits I picked up in college have been growing mental mold in my head. I've forgotten much of what I learned about subnetting, so when I had to quickly figure out what slash notation our mask was, it was Google to the rescue, or more accurately, mark-scott to the rescue! I've summed it up below. Click the link for a much better explanation.

Link to original content: http://www.lammle.com/discussion/showthread.php?t=737

So you know what the mask is, but you need to quickly figure out what the slash notation is?

Basically, remember these numbers:

1 128
2 192
3 224
4 240
5 248
6 252
7 254
8 255

Consider each octet of the example: 255.255.252.0

If you add 8 (255) + 8 (255) + 6 (252) = 22

So 255.255.252.0 = /22

How cool is that??? Thanks mark-scott, wherever you are!

Peter's account logout saga - Episode four

2009-06-22T07:30:00.000-07:00

Dear me. It has happened again.

06/22 10:14:31 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\peter from ISA2006 (via ISA2006) Entered
06/22 10:14:31 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\peter from ISA2006 (via ISA2006) Returns 0xC000006A

Thank goodness we are replacing the old ISA server in the next few days.

A few other notes: We have removed the web proxy and VPN functions from this ISA server...so that leaves only the firewall function to be the cause of these logon attempts!

Crickets.

VPN goes down after 2-3 hours - server 2008

2009-06-22T05:25:00.000-07:00

Weird VPN error - which is more likely linked to VMWare or Server2008.

We moved our VPN access over to a new Server 2008 x64 SP2 VM. It was working fine at first, but then after a few hours all VPN access would be lost.

First step was to reboot the server, and that fixed it. But again, VPN would be down after a few hours.

Some troubleshooting and checking things out shows that it's not VPN going down - it's network connectivity across the board on that VM. I can't even ping 127.0.0.1 anymore! So it's a failure of the entire network stack - pretty rare, in my experience.

Actually...I can't ping loopback when it's functioning...so I guess there's something deeper here. I just spoke to Dan about it, he said he'd added a second nic, then removed it when things weren't working correctly. He's just going to blow away this VM and start again.

Strange...well, the issue seems related to the missing nic, so not much can be done about that.

IP address scheme changeover

2009-06-20T05:18:00.001-07:00

Today we're (I'm) changing over the rest of our servers to the new IP addressing scheme. This is noteworthy because we've only really done one so far - our Sharepoint box - and it broke a few unexpected things.

Reason? We are going to change all the IPs to a new subnet range to clean things up. We're halfway there, and now we can't get past ISA blocking RPC due to us trying to access different subnets, rather, go across subnets. Just doesn't work properly. Kinda silly.

Anyways, we had two options, make ISA disappear, or finish up the IP address change - something I was sure would break a lot of things.

So far, things are going pretty well, but I've set it up so we do everything easy first!

Changing the IP for the Exchange and BES servers is a little unnerving...but I think that Exchange pretty much exclusively uses DNS - I don't ever recall seeing statically set IP addresses, except in the TCP/IP settings. I've updated the static DNS records, so everything should work. Update: Yep, it works. AFTER you change the ISA rules to accommodate new IPs!!!

The SQL machines, however, I'm concerned about. There's a lot of custom stuff in there, and who knows what person decided an IP was a good way to connect something.
Update: Things seem to be working just fine so far....we'll see when people actually start using the system.

Changing the printers is proving to be the most difficult of all the changes. So far at least one of them is on a JetDirect box. It required a reboot before it would take the new IP settings, and now the server refuses to see that it's online. I may need to restart the printer itself.
Update: Server reboot fixed it.

Another thing to take into account on ISA is the network setup. If you remove the network/gateway ip from the old subnet, then try to access stuff on the old network - no connection until you re-add the network range and gateway ip!! Confused me for a minute...

Nmap is still turning up some random hosts, including an IP I'm sure that I have already disabled...nic is made by Accton Technology...and I'd thought it was on one of our Cisco devices, but the Cisco device no longer has that IP enabled, so this is messed up.
Update: Haha! Figured out what they were, and it's another lesson: Keep track of your network devices, and their respective logins!!!

We're working on a comprehensive document to keep better track of devices. We'll eventually tie it into something like Nagios.

Well, everything is up and running, so that concludes this entry. I'll post up any other oddities I run across.

Peter's lockout saga - Episode three

2009-06-19T07:56:00.000-07:00

Well, it fixed itself. I unlocked his account at 8:30am today, and since then he is no longer getting locked out. So. Messed. Up.

We've been messing about with the ISA server, so maybe that's what caused it to stop. Hopefully this will be the last we'll hear about it.

Peter's lockout saga - Episode two

2009-06-17T13:55:00.000-07:00

Dan, our consultant, had a really good idea for temporarily helping me out with this.

Move the user to a new OU. Create a new GPO with one change: account lockout policy is set to 0 (never lock out). Set the policy to enforced, and 'block inheritance'.

Voila!

Actually this didn't work. Shame. It seemed like a good idea. The issue with it is that the GPO is only applied on either: the computer, or the user. Since this request is coming from neither a computer nor a user, the GPO does not apply, and the lockouts continue.

To continue that line of thought, what on earth is trying to use his account?

Dan checked it out a bit more, and discovered (using more auditing) that it was the NETWORK SERVICE account using the Firewall PID. Really weird.

As part of our network revamp process, we're going to be isolating the functions of ISA - namely just having it work as a web proxy, and move the firewall functions over to the Cisco router, with a few other bits in between.

More to come...

Technical version: EqualLogic PS series controller firmware update

2009-06-17T09:42:00.000-07:00

This documents the process for updating your EqualLogic PS5000* to a newer firmware (from 4.0.6 to 4.1.4 in our case). Should work for the PS6000* as well. Note that the material here is from the document 'PS Series Storage Arrays - Updating Storage Array Firmware' by Dell EqualLogic, the tech case sent over from Carl at Dell, and my experiences from all this. It's pretty unnerving to have a key piece of hardware die on the ONE piece of equipment that stores everything in the company, but now that I've run through this once, it's not really all that bad to do.

So if you have a controller update fail, it's probably just in need of the following!
-------------------------------------------------------------------------------------------------

Here's the process for getting the firmware kit from the internet to the controller.

1. On your SAN monitor box (XP or whatever), download the firmware from EqualLogic's tech site (logging in is another story!!). DO NOT rename the file to make it easier for you to FTP. You will be foiled. Place it in your C:\ (for ease of use).

2. 'open' ftp to an IP assigned to one of the controller's ethernet ports. Set 'binary' 'put' the kit file onto the controller ftp site.

2. You can only update online from 4.0.1 to 4.0.2, and the SAN must be offline from 4.0.1 to 4.1.1, for example. So...all your hosts, in our case the 10 ESX hosts, must be powered off to prevent any issues of LUNs losing connection, or a host powering on and trying access the SAN while it's updating.

3. Once everything is off, SSH onto the controller IP (grpadmin login). Run the 'update' command. Say 'yes' when prompted.

4. Wait for the update to complete. If it takes longer than a few minutes, there's a good chance that it could have failed. When ours failed, it took about 5 minutes of 'retrying' the operation.

5. If it completes successfully, 'restart' the array.

6. Once you're back in, run 'member select show controllers'. You should see both controllers with the version of firmware you updated to. If you are missing a controller, or don't see the correct version, you'll need to follow my other steps.

-------------------------------------------------------------------------------------------------
Get a serial connection (same setup as a Cisco connection - 9600/8/1/n/n) to the controller that is missing. You will need physical hands-on access to it, as the break sequence has a small window.

Restart the passive controller from the CFE> prompt with the command "reload -softreload" and 'yes'.

In between the SP and NP messages, type Ctrl-P then Space. Do that a few times. You should get a prompt like this:

Boot Ethernet Port: []

Hit backspace a few times to make sure you're not accidentally putting anything in that field. Then hit return until you see "Boot Path". Type in "backup", and hit return until you get to the CFE prompt. Enter "reload" there to reboot on the old image.

Boot Ethernet Port: []
CM Boot IP Address: []
NetWork Mask []
Boot Gateway IP Address: []
Boot Server IP Address: []
Boot Path (primary/backup): [primary] backup
CPU0 File Name: [eqlstor.gz]
CPU1 File Nmae: [eqlqrq.gz]
Core Dump Ethernet Port: []
Core Dump IP Address: []
Core Dump Network Mask: []
Core Dump Gateway IP Addressp: []
Core Dump Server IP Address: []
Core Dump Enable: []
Core Dump File Name: []
Pss Log Enable: []
CPU0 Memory Alloc: [75%]
CFE>reload

You should then boot properly to 4.0.6. Login as grpadmin, and do the following:

1. Get to bash
2. look at the primary directory in the CF card, and verify that it's missing a lot.
3. rm the files that remain in the primary directory.
4. cp the files from backup to primary.
5. restart the array.

CLI> support exec /bin/bash
cli-child-3.2# cd /pss
cli-child-3.2# ls -al
total 29294
dr-xr-xr-x 1 root wheel 16384 Dec 31 1979 .
dr-xr-xr-x 1 root wheel 14973061 Aug 7 22:25 ..
dr-xr-xr-x 1 root wheel 1024 Jan 24 2008 TOOLS
-r-xr-xr-x 1 root wheel 1107 Aug 21 07:54 agent.cnf
dr-xr-xr-x 1 root wheel 1024 Jan 24 2008 backup
-r-xr-xr-x 1 root wheel 218 Oct 7 05:07 config.txt
dr-xr-xr-x 1 root wheel 1024 Oct 7 04:11 dumps
-r-xr-xr-x 1 root wheel 0 Jan 24 2008 eco.a
-r-xr-xr-x 1 root wheel 0 Jan 24 2008 eqlcfe.gz
-r-xr-xr-x 1 root wheel 601 Oct 7 03:30 passwd
dr-xr-xr-x 1 root wheel 1024 Oct 7 05:07 primary
dr-xr-xr-x 1 root wheel 1024 Oct 7 03:57 update
cli-child-3.2# ls -al primary
total 760
dr-xr-xr-x 1 root wheel 1024 Oct 7 05:07 .
dr-xr-xr-x 1 root wheel 16384 Dec 31 1979 ..
-r-xr-xr-x 1 root wheel 1826 Oct 7 05:07 build.c
-r-xr-xr-x 1 root wheel 369664 Oct 7 05:18 ccom0102.bin
cli-child-3.2# ls -al backup
total 40934
dr-xr-xr-x 1 root wheel 1024 Jan 24 2008 .
dr-xr-xr-x 1 root wheel 16384 Dec 31 1979 ..
-r-xr-xr-x 1 root wheel 1813 Oct 7 04:01 build.c
-r-xr-xr-x 1 root wheel 393216 Oct 7 04:01 ccom0103.bin
-r-xr-xr-x 1 root wheel 393216 Oct 7 04:01 cemi0611.bin
-r-xr-xr-x 1 root wheel 203 Oct 7 04:01 cksums
-r-xr-xr-x 1 root wheel 636061 Oct 7 04:01 eqlqrq.gz
-r-xr-xr-x 1 root wheel 18069007 Oct 7 04:01 eqlstor.gz
-r-xr-xr-x 1 root wheel 929027 Oct 7 04:01 eqlx811.gz
-r-xr-xr-x 1 root wheel 393216 Oct 7 04:01 sumo0310.bin
-r-xr-xr-x 1 root wheel 120989 Oct 7 04:01 update.sh
cli-child-3.2# rm primary/*
cli-child-3.2# cp backup/* primary (this takes about 10 minutes, so be patient - it really does)
cli-child-3.2# exit
CLI> restart

Then catch the boot process again when it asks for Ctrl-P, and enter Ctrl-P Space again. Change the Boot Path back to primary from backup, and reload from CFE.

Once all that's done, you run the 'member select show controllers' command. You should see both controllers again! Yay!

EqualLogic controller firmware update gone wrong

2009-06-17T07:54:00.000-07:00

Yesterday evening we set aside three hours for:
- Installation/cabling/setup of DRAC5 cards into our six ESX production machines, which involves shutting down all the ESX machines and their guests.
- Firmware update of our EqualLogic PS5000X loaner SAN array to be done while all ESX machines were off.

We had set aside 7pm to 10pm for this, plenty of time - or so I thought.

At 7pm I shut everything down, and at 7:10pm I started pulling out servers for the DRAC install.

Cabling confusion - we have two quad-nics in each ESX box, so there are eight cables going to the SAN switch from each machine. Tack on the two for the redundant LAN cabling, and you have ten CAT6 cables from each 1U machine - four total, ten from one 2950, and twelve from the other 2950. Lot of cables in a small space!!

What I'm getting at is that the cables needed to be re&re'd into the same ports on the server, so they all had to be labeled. I know, we should be labling off the bat, but time and resources are against us...I'd say at the moment, but they always are. It's a big weekend project at some point to re-label and re-cable the server room. It was originally super-neat, but impossible to trace cables. So now it's a mess...and hard, but now not impossible to trace cables.

But I digress. Once the DRACs were installed and configured, I powered off all the ESX machines again. Oddly, ESX3 and 4 kept powering themselves back on, just for fun. In retrospect, I'll unplug power to all hosts to prevent issues.

I SSH'd into the SAN, and eventually got the kit file FTP'd up there. TIP: You cannot FTP across subnets when using ISA as your gateway, even if you specify all ports open.

I ran the update command, but the second standby controller timed out, and the update failed. I tried to run the update command again, but once it completes, it deletes the kit file. Had to go through a huge hassle to get it back on again. TIP: Have a physical or at least non-SAN XP client on your SAN network for monitoring and FTP/Serial functions.

Long story (2.5hours) short, we ended up having to serial console into the standby controller, force it into backup mode, delete files, copy files, restart, etc etc etc. But finally, it's working again. Dell tech suggests that the card lost communications long enough for it to time out, corrupting the firmware, causing the controller to fail/not be seen. Anyways, it's working now, both updated from 4.0.6 to 4.1.4. Unfortunately you cannot update each card and keep the SAN online - need to power down all hosts that are connected to the SAN before doing any firmware updates. They said that 'minor' f/w updates can be done while online, but 'minor' = 4.0.1 to 4.0.2.

Anyways...it took forever to do, what with me not having a proper client to access the SAN network by. Argh.

Tech doc to follow.

Exchange/Outlook synchronization errors

2009-06-17T07:53:00.000-07:00

I shut down our ESX hosts last night, and after the EqualLogic debacle, I powered everything back up. Exchange came up, but refused to send emails properly. I couldn't receive any emails at all, and sending them to my gmail account gave this:

------------------------
Please note that a critical failure occurred on the sender's messaging system at the virus scanning stage.

The sender should be notified and reference made to this tracking number: src53_failed_f7e789cd-9a1a-

439c-a08a-ed68b4bbef39

Regards,

GFI Content Security.

------------------------

While a very polite message, it failed to mention that GFI also does not provide technical support outside of business hours. They will not be renewed if that continues to be the case.

With that, I googled, and checked out their Kbase - nothing. I tried stopping all the GFI services, nothing. Tried restarting the Information Store - nothing. Tried deleting my Outlook profile - nothing. Tried logging in on another computer and setting up a new profile - nothing.

I downloaded the latest version of MailEssentials and MailSecurity (I was already on the latest for both), and uninstalled/reinstalled both.

Rebooted Exchange - and everything worked again!!! Stupid. Took up another two hours to troubleshoot and fix that issue - ended up leaving at 3:15am.

Tracking down Peter's lockout

2009-06-10T23:24:00.000-07:00

I've been driven bananas long enough by this problem. I have an entire evening with no interruptions (of course, also no sleep!) so I'm going to use the time to figure out WHERE on earth these lockouts are coming from.

We are gathering info from:
1. The ADLockout tool that monitors his logon status from both DCs. It is used to unlock his account as well.
2. A full netlogon log (all options selected) pulled from the ADLockout tool from DC-1.
3. Other methods I'm sure will be used.

By this point, we've deduced:
1. It's coming from ISA (Transitive Network logon of orthotic\peter from ISA2006 (via ISA2006))
2. It's not a bad password on a mapped share, or him typing them in incorrectly.
3. His saved cached of passwords in XP is clear.
4. He is having the problem regardless of the state of his laptop or desktop (e.g. if both are off, lockouts continue).
5. He has checked to see if his credentials were still in use for a VPN session he helped someone set up - the computer was off for days at a time and he was still getting locked out.
6. I have disabled his account and still the lockouts continue (although this may be normal behaviour).
7. This is now the second occurrance of this - it happened for a period of a week where I updated all anti-virus software, cleaned up the computer, etc etc etc, then promptly disappeared for a month.
8. It should also be noted that there are no entries at all from any computer other than ISA.
9. He is not attempting to VPN in with bad credentials - this is happening as I type this at 2:30am.
10. Not sure if this is relevant, but I seem to see all bad password counts from the main DC, and other times all from the secondary DC. Must just be a round-robin thing.
11. ISA has been rebooted several times, so that won't fix it.
10. It is completely capable of driving me insane. Him moreso.

Here's a sample of the netlogon log: (I used: findstr /I "peter" c:\netlogon.log > c:\peter.txt)

06/10 20:20:59 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\peter from ISA2006 (via ISA2006) Entered
06/10 20:20:59 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\peter from ISA2006 (via ISA2006) Returns 0xC000006A
06/10 20:21:00 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\peter from ISA2006 (via ISA2006) Entered
06/10 20:21:00 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\peter from ISA2006 (via ISA2006) Returns 0xC000006A
06/10 20:21:05 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\peter from ISA2006 (via ISA2006) Entered
06/10 20:21:05 [LOGON] ORTHOTIC: SamLogon: Transitive Network logon of orthotic\peter from ISA2006 (via ISA2006) Returns 0xC0000234

Thanks to here: (http://support.microsoft.com/kb/189541), we know that a 6A return is 'bad password', and a 234 return is 'account locked out'.

We can watch my fruitless attempts to keep him not-locked out all day. And week. Things have been too busy to track this down.

At any rate, I am going to check ISA's logs now, and see if I can turn anything up.

ISA has many logs, some useful, some not so useful.

To find out who is failing VPN login sessions (because RRAS logs do not show this), check for: "C:\Program Files\Microsoft ISA Server\ISASummaries", and edit the latest summary file. It's gibberish, but it gives you two great pieces of the puzzle: usernames, and IP addresses. Sadly, it's just that - gibberish, and if you use the reports function in ISA to decode them, you lose all username and IP info. Lame.

I then figured, hey, if the IP keeps showing up, must be the source, right? Not so much...the report is a garbled mess.

I'm now down to refreshing the ADLockout tool every 10 seconds, and watching a packet capture on the WAN port filtering on port 1723 (pptp). 5 lockouts in an instant...no packets.

Argh.

More searching has revealed something interesting - a whole ton of [TCP segment of a reassembled PDU] from this IP: 121.235.177.227, supposedly registered in Beijing, China. Seems that someone in China likes our shoes, or is downloading the whole website.

There doesn't seem to be any way to glean the info I need...I think I'll need to look into expanding the logging that's actually ON ISA2006, and get sysinternals into the mix. That's a Monday job though. Thankfully I have tomorrow off.

Monday report:
His account is not locked after 4 days! Wonders! It was still locking on Thursday when I left.
We'll have to wait until next time this happens.

VM project progress

2009-06-07T11:26:00.000-07:00

Well, so far this project has been a huge learning experience. We've been through subnet changes, shared file location changes, group policy changes, user My Docs location changes, hardware upgrades & installation, SAN configuration and best practices (still fuzzy on the latter), multipathing using the SAN, and the list goes on. It's been nice to actually use some of my Cisco training at last, although after three years of inactivity, my brain cells are fuzzy on that as well.

We're finally at the final stage of the SAN project - ISA. It's our last production server to be virtualized, and will be the most complex, as it's the most connected to the physical world. D has been great setting it all up, hopefully my routes will do the trick! If only the Digi PortServer TS1 would work...it worked flawlessly off the bat, but now that it's been installed/uninstalled a few times, doesn't want to connect anymore....must be something simple.

I'm really enjoying all of this, except for the long hours spent on evenings, weekends, and a few 36-hour shifts now (might be a few more of those). Missing the family time.

VMconvert speeds

2009-06-06T20:45:00.000-07:00

We've had some interesting luck with VMconverter. Seems we get good speeds off the bat, but eventually the speeds crawl.

I'd be happy with 20MB/s, but we start around 17MB/s, then it drops down to 4MB/s, then 2MB/s.

It's actually really funny...the timer went up to 54 minutes off the bat with 17MB/s, and two hours later it's up to 1:10 and 4MB/s. Awesome.

The sunny side to all this is that, barring the 'semaphore' errors, it's worked no problem.

VMconvert and error: 'semaphore timeout period'

2009-06-06T14:57:00.001-07:00

Well, discovered an interesting issue.

When running VMConverter as the server version and P2Ving remote clients, I've run into an error:

'Error 0x80070079: The semaphore timeout period has expired'

Can't really find any reasoning behind this (possibly ISA blocking RPC), but if you install the converter on the local machine it seems to work just fine.

The possible reason for it could be that our VMconverter server is running on one subnet, and the client is on another subnet. ISA has two IPs, one for each subnet, but that doesn't seem to matter. We have had RPC issues going from one subnet to the other already, so this could be a viable answer.

Remote users and their associated troubles

2009-06-03T13:30:00.000-07:00

So we have a few remote users - only ever connect via VPN. This is usually fine, unless - group policy updates need to be done!! Dun dun dunnnn.

Then it stinks.

Here's how we get around it for now: (I've been told that RPC over HTTPS is the cat's meow, but we'll have to wait for 2008 R2 first.)

1. Connect to VPN.
2. Run gpupdate /force.
3. Reboot.
4. Log on using dial up connection, use VPN connection.
5. It works!

Sounds easy, but took forever to figure out that's how it's supposed to be done. There is no real way to auto-run the VPN prior to logging on. You can set up a service to run, but there's no guarantees there.

Another catch is to ensure that the VPN you're using is set up so that anyone on the computer can use it - if you choose 'only for me' when installing your VPN, it won't work, as anyone can use the Windows logon screen!

SATA disk shelf research

2009-05-29T18:25:00.000-07:00

I'll compile it here for my reference really.

We use the Perc 5/e card in two servers - we'll use one to attach to our current MD1000 - I'm checking around for a better alternative to another MD1000 - the lowest price I've seen for an MD1000 is $1200 on Dell's Outlet site. Needless to say, it disappeared quite quickly. Now that we have a US address to ship to, that site can be used!

The current MD1000 will be filled with our existing 300GB 15k SAS disks, and probably set up for RAID10 or 50. It'll house our ESXi test environments in two arrays I would guess. We need another one for other purposes.

The purpose of the second unit: mass storage array for temp files, archives, miscelaneous files, and backup-to-disk. The more space and spindles, the better, however, price does factor into things. Another option would be to run openFiler alongside it. Have to explore that.

Here are the current options:
--------------------------------------------

Used Dell MD1000 (single EMM is fine, dual gives us more SAS ports to play with, and the option for two hosts hooked up properly) with 15 1TB ES.2 drives (quoted $235 per drive).

--------------------------------------------

Came across this: http://www.aicipc.com/ProductDetail.aspx?ref=RSC-2EG2-0

A nice SATAII enclosure, 12 disks, max of 12TB raw. It is compatible with the Seagate .250-1TB ES.2 series drives, and our Perc 5/e cards (which are actually re-flashed/branded LSI MegaRAID SAS 8480E cards). Nice. But 12TB isn't much in the grand scale of things...would be nice to have more.

For reference, here is the RSC5ED5R-SA1C-0: $1585.00 (RSC5ED5R-SS2 is for SATA, that price is for SAS)

So, $1500 or more for the chassis. We got quoted $235+tax for a 1TB ES.2 drive, so that gives us: $6000 for the drives, $1500 for the chassis, plus whatever we put into it. Sheesh!!

--------------------------------------------

Enhance-Tech also produces a slick looking unit, albeit running u320 SCSI: http://www.enhance-tech.com/products/diskarrays/index.html

The R16 unit may fit the bill, although lack of RAID10 or RAID50 is kind of a deal killer. They also look expensive (update: and how!).

--------------------------------------------

RAD-Direct has some fun names: SATAboy, SATAbeast, SASboy, SASbeast. Someone creative in marketing. Unfortunately, 'from $980/TB' is a deal killer for us.
SATAboy link: http://www.rad-direct.com/Product-SATAboy-SATA-Storage.htm

--------------------------------------------

More to come...

Here's a fun technical article on SATA arrays: http://www.eetimes.com/in_focus/communications/OEG20030414S0072

iSCSI and MPIO and VMs, oh my.

2009-05-27T17:59:00.000-07:00

Connecting a 2008 ESX VM guest to a raw LUN on the SAN.

Acronyms. Sheesh.

Set up the VM.
Install MPIO (requires a reboot).
Open up the iSCSI initiator in cpanel.
Change the firewall settings to allow pinging back and forth - required for MPIO.
Install DSM from the EqualLogic disk.
Ensure SAN nics are installed to the VM and configure with IP and subnet mask.
Under the iSCSI initiator, go to Discovery, and add your group IP under portals.
Choose Targets, then log on to your lun of choice.
When logging on, choose 'use multipathing' and 'restore connection'. Click advanced.
Select 'iSCSI initiator', the first nic IP, and the destination (group ip), no other options necessary.
Do the same for the second nic IP.
If you click 'details' on the lun you've targeted, you should see two connections.
Ensure they are set to round robin.
Check the network interface monitor on the SAN console - should see two interfaces active while transferring data.

SANs are different Part 2

2009-05-27T13:35:00.000-07:00

Well, we bought one. The EqualLogic PS6000XV, the 450GB flavour.

We're on our demo unit (a 5000-series) until the new one arrives, at which point we'll join the new one to our group and 'evacuate' the data over. Pretty nifty. That feature alone saved EQL the sale over the EMC AX-4 a company was trying to get us into. Sadly, they could not get us a demo unit in time, so we could not test in time. Ah well. We all really like the EQL solution, and features. Since we're close to Dell we got a pretty nice price as well.

Amusingly enough, the only complicated part of the SAN is the rack mounts - they are terrible! Or rather, the diagrams are...photocopied CAD diagrams at that.

We're using these specs to judge because the critical loads are SQL I/O, and the average request is around 50kb.

Our performance numbers looked something like this, keeping in mind this is the 10k PS5000. All are random reads, 64 threads to a depth of 128. Shown is the block size and max IOPS.

MD1000
- 64kb - 1456IOPS
- 8kb - 3058IOPS

PS5000E
- 64kb - 2640IOPS
- 8kb - 3393IOPS

PS5000X
- 64kb - 3601IOPS
- 8kb - 7389IOPS

According to what data we can find, the PS600XV should be at least 25% faster than the PS5000X due to 15k disks versus 10k, double the cache (2GB to 4GB), and an extra 1Gb port (for a total of 4Gb when using MPIO). I would guess the newer architecture would help as well, but I have no real basis for that.

I should stress that these numbers are positioned to make the MD1000 look like a dog compared to the 5000X, because in these instances it is. However, in the sequential department, the MD1000 was faster than the 5000X by a surprising amount. We discounted that because all our access would be VM, SQL, and Exchange - mainly random reads.

DCPromo

2009-05-27T07:16:00.000-07:00

We're virtualizing our DCs today.

Process:

Setup new 2k8 vm template.
Copy.
Get VM set up and newsid.exe it, rename.

For the domain:

Need to transfer the 5 FSMO roles.
Use ntdsutil and the GUIs to make sure.
ntdsutil - connect to the server you want to make master
- transfer the roles, only seize roles if you've messed up and forgotten something, or you've had the server crash.
- see here for details: http://support.microsoft.com/kb/324801

Once DHCP and DNS are setup and running on the two VMs, take one physical DC offline by doing DCPromo, then unjoining (is there a proper term for that?) from the domain, then unplugging the network cable.
Take the IP address of the decomissioned DC and set up the VM DC to use it. Reboot the VM DC.
Should be working now - check that DHCP leases are being created, and that DNS reports the correct IPs.

Exchange will need to be rebooted to take into account the new DCs for 'configuration manager'. You can try and change this manually, but it's recommended that you just reboot - it'll automatically pick up the new DCs if the old ones are offline.

Been up all afternoon with no issues. Great!

Going to get a backup physical DC running again on our old PE750, just in case. Not entirely sure how we'll do the DHCP for it yet.

SANs are different

2009-05-15T09:45:00.000-07:00

Simple title...but to me, it makes sense, especially when we're talking about disk performance.

My understanding of disk performance has changed dramatically these last few days. I went into it believing that it was all about disk speed and max throughput, so an array of 6 15k SAS disks in RAID5 was fast to me, and there was no way anything SATA could equal it.

However, when you speak of IOPS (I/O per second) - a term thrown around like jellybeans by SAN sales people - it really comes down to spindles, as in, the number of disks you have.

A good metaphor someone told me was to think of a library. It's a small thing for one person (one disk) to get six books that are on the same shelf. It's even easier for six people (the six SAS disks) to look for 6 books, especially if they are all together. However, in an SQL environment, requests are asking for data all over the place.

So, if you then think of one person trying to get six books from all corners of the library, it makes sense for you to have one person for each book - in the time it takes one person to get one book, you have actually gotten six.

But...then we move from our RAID5/6 disk array to a 16 disk array...sure, the 'people' getting the books are a little slower, but you have an extra 10 people! With the EqualLogic option we're looking at, we'll be doubling the number of disks next year, meaning we would have gone from 6 spindles on our heaviest of duty SQL DB disk array to 32 spindles... Yeesh!

The SATA option we're looking at is probably 25% slower than the 15k SAS unit we're testing this week, but I'll have concrete numbers next week when we test out the 16x250GB disk SATA array. The 500GB option may even come into play, as the price difference is not that big.

Let's now look at throughput.

You can apply the analogy to the size of the door you're trying to fit the books through. So, you've retrieved 100 books, and you can just fit them through the door. But what if you need more than 100 at a time? Throughput allows you to carry a certain amount of books through the door at one time.

It's all well and good to boast 150MB/s max throughput for this RAID5 array, specifically for sequential writes - they monster the SAN speeds. However, it still comes down to what the SAN does best - allow fast access for everything that needs disk.

More to come in Part 2.

Jetta front suspension almost done

2009-05-15T09:14:00.000-07:00

I upgraded, as mentioned below, some of the Jetta's front suspension the other day. After realizing that my Princess Auto spring compressors were universal, and needed some customization to work properly in the Jetta wheel wells (cut off about 4" on each one), the struts went in nice and easy! The front sway-bar bushings were super-easy to replace as well.

You can see in the one picture how I got the strut mounts off and then re-torqued.

While I was waiting for H to come by with a bolt-out kit (that proved fruitless against the might of the dogbone mount corrosion), I decided to get rid of the rest of the skunk smell (see short story). It was awful, and permeated the garage until I found it. No picture...cuz it's gross.

Found the passenger wheel well filled with dirt, behind the plastic guard! There's a write-up on the TDIclub forums about that...I'll link to it when I have a minute.

To get the passenger strut pushed in, we used a dowel and a small floor-jack - worked like a charm!

To get at the skunk smell I had to remove the entire front bumper cover - it did give me a chance to replace a few bulbs while I was at it.

A few more bits I didn't do because of time and patience - rear a-arm TT bushing replacement. I have the parts...but I'm terrified of breaking loose that captive nut inside the subframe, and I don't have any easy access to welding equipment.

The intake is another looming job...gross!

The skunk.

2009-05-15T09:12:00.000-07:00

Wrote this after an interesting morning. I was going to post up a picture of some of the carnage...but decided against it.

This Friday morn, please sit down and join me for a short story that will amuse:

As I left the house at my usual time today, and headed up Ritson Road for my journey to the north (away from the traffic of the city), I could have hardly realized what a tragedy the morning commute would be. The little black Jetta cruised lazily up the back roads, unexpectedly obeying the speed limits, and my thoughts were on the lovely surroundings.

Now, as commuters have been known to do, my attention wandered from the open road in front of me to things that tended to stand out. I entered a longer straightaway and saw a bump on the road, but could not make out what it was. At any rate, it was in the other lane, so my attention was more of a curious nature. As I drew closer, my nostrils betrayed what the decrepit bump on the road was, and my attention, now much more alert, was drawn to the large yellow object converging with the bump on the road.

It was a full-size school bus, trundling down the road making the morning stops, oblivious to anything in its path, because after all, it was a very large bus! A small bump on the road was no match for the sheer mass of the chassis, and could not cause the large diesel engine to cease its limitless output of torque. This was unfortunate. As I drew up to the deceased animal, the bus, somewhat tauntingly, sped up (or so I perceived), and my mind breathed a sigh of relief when the front tire of the bus only just missed the very smelly bump.

What was all the more unfortunate was the unpleasant fact, for myself, and the dead mammal, that school buses have not one but two axles. This would normally not cause me any worry, except that as the bus and I passed each other, and the skunk lay there in a fetid pile, my brain told me in a pleasant manner that not only do school buses have two axles, but the rear axle, the one about to pass me, had two wheels on either side.

So while the front wheel had missed the bump, one of the two rear wheels most certainly would be hitting the skunk, and this is a point I shall expound upon.

If you have ever blown up a plastic bag, and then popped it, you know that the air inside rushes out from the first weak point in the bag it finds. Let's reassign the properties of that analogy so that the plastic bag is the body of a skunk, and the person popping it is the dualie rear wheels of a school bus, and the air inside...well, you get the idea. Thanks to the significant mass of the school bus, and the rather poor qualities of holding in skunk innards under great pressure a skunk's body posseses, you begin to see what was about to happen.

Now, there are all sorts of places for innards to leave an animal in such a condition. The faux pas of the skunk was to die with his bottom facing my lane. My faux pas was to drive the speed limit, and simply stare in unbelieving astonishment at the aforementioned unfolding events. As we all know, skunks spray rather horrific smelling liquid from their bottoms at predators. When the majority of the inside of the skunk exited via that most potent area, it did so at an impressive velocity. This was not lost on me, as I experienced what could be likened to someone throwing strips of uncooked and marinaded beef at the side of my car. The audible nature of such an event was forever impressed upon me at that point, and even now, as I write this, I can hear it.

And now begins the tragedy of my morning commute. You see, it takes me approximately an hour to drive to the office. I had been on the road at this point for five minutes.

I had also, rather unfortunately, chosen to tilt open my moonroof as I left that morning for some fresh air. And such fresh air I received. I felt as if I could taste death and skunk potency in my very throat, and the urge to regurgitate my 'Oatey O's' was rising, no pun intended. I quickly rolled down all the windows, only to be smashed in the face with the full brunt of surely ethereal skunk's fury. My eyes began to water and I choked as though drowning in the smell. It is the single most horrifying smell to ever have crossed my nose, and I pray to never encounter it again.

At this point I left the speed limit far behind me, in an attempt to do the same to the fury of the skunk. It seemed to dog my every move for several kilometers, and then, suddenly, I was on the highway, and the smell diminished, and I no longer felt the shadow of the skunk tailing me. I luxuriated in the rich smells of a burning fire as I passed a farmer at work, and then the smell of freshly sprayed manure crossed my senses, and my spirits leapt in joy! What a smell...it was glorious!

And then I came to a stoplight. And the evil I had left behind hit my consciousness like a speeding train carrying dead skunks.

I could go on, but every story must come to an end.

There is no need to say that every car behind and around me must have been gagging as well. I noticed, at stop lights, the driver of the car behind me gesticulating wildly. Whoever you are, I am deeply sorry for the emotional scarring you received today.

I drove as quickly as possible to the nearest gas station and inspected the damage. I would have taken a picture, but the smell....it was horrifying. There were two strips of, I shall call it 'meat', now fused to the front bumper, approximately 18 inches long. Several other 'chunks' around 1/2 inch in size were splattered elsewhere, and there were droplets of skunk blood all over the hood.

One Ultimate Car Wash later, the detritus was gone, the smell was gone, but the shock remained. I believe that the skunk strips also remained, attached to the brushes in the car wash. This amuses me, in a rather awful way. I expect to see the car wash closed when I drive home from work.

In closing...the mess is gone, but the smell remains. While opening the windows gave me respite from the smell...it also ingrained the smell into my clothing.

And that is the story of Trotter, the skunk, and the school bus.

Performance Counters - Analysis

2009-05-12T14:18:00.000-07:00

I've struggled to find a good source on how to properly interpret performance counters. In our case, it was determining IOPS for a potential SAN we were looking at. I had really been trying to convince the management that we could better utilize our current server-base if we set up a SAN and went ESX for production (we had ESX for testing already). They were not convinced that the SQL servers would fare well under ESX, so I started researching.

I spoke to a number of vendors (Xiotech, Lefthand, EMC, Equalogic), and all of them wanted to know how many IOPS I needed out of the solution. They then of course went on about how many their particular ware allowed, but I digress... I had no idea how to calculate IOPS (I/O per second) requirements, so it was off to the internet to figure things out. One of the vendors offered to decode perfmon logs for me, but I really wanted to know how to do that myself. Besides being a handy skillset to have, it meant much less time spent waiting for others.

I'll have a list of links at the end of this post, as a lot of the info I gleaned was from multiple sources, all quite good in their own respect.

Here goes, and please note that this spec is for an SMB with around 100 users.

First of all - perfmon. It is not advisable to use readings on the fly - set a capture to run with all your desired counters and let it go for a business week (depending on the number of counters, you may need to allow perfmon to use the maximum size of the logs - you can also set a custom size - 4095MB is the maximum).

Exchange calculations
Pulled off Technet, the basics were this:
Based on a heavy user - 60sent/150received,500+MB mailbox), you get approximately 1.5IOPS. Take the number of mailboxes you have (I used a high number of 250 - we'll have around 200 mailboxes when all is said and done) and multiply by the IOPS, so 375IOPS is what we need to have Exchange running correctly with room to grow. Again, my calcs are pretty generous.

SQL calculations
This is a deep, deep pool to wade into...but I really had no other choice.

Basically, I needed to know what our current usage was, because if it is more than the SAN will be able to handle, then it'll clearly be a fruitless effort. (I will also use SQLIO to get our max throughput for a benchmark - thanks D!)

The essentials of a properly running SQL server are:

Hardware setup

Dual quads, 8-16GB RAM, 64bit OS (keep in mind that even if you have 16GB of RAM, and you allow SQL to use 15GB of it, then wonder why you have less than 1GB left over...it's because SQL will use as much RAM as you allow it to have - the basis being that it's faster to page to RAM than disk.)
Your DBs (databases) are on one array, preferably a RAID1 or RAID10 array with disk caching (pretty sure you can't get a RAID controller these days without battery backed-up caching). This array also has as many spindles as you can spare (for our CRM4.0 rollout the consultants specced 8x146GB 15k).
Your transaction logs are on another array, again preferably RAID1 or RAID10 with four disks as above specced.
There is a temp array for DB copying, backups, etc, same spec as the logs array.

With that number of disks (8DB,4LOGS,4TEMP,2OS), you're really into a shelf at that point, or some sort of SAN. Also, the reasoning behind RAID1 or RAID10 is with those RAID levels you get more reads, and since SQL is read-heavy you then have better performance.

Performance counters to note
This was all noted right off a series of videos hosted by techtarget.com. I'll note them Counter - Instance, if applicable -notes.

Paging - Total use % - Should be less than 70%.
Paging - Peak use % - Same as above, should also be fairly constant.

Processor - % Processor time - All - Should be 80% or less.
If you're troubleshooting the above:
Processor - % Privilged time - All - Should be less than 30% of total % processor time.
Processor - % User time - All - Should equal (100% - % Priv. time).

Memory - Pages input/sec - Should be less than 10.
Memory - Pages/sec - It's okay if this is around 500-600 or less.

SQLServer:Access Methods
- Index searches/sec
- Full scans/sec
If you divide (Index searches/sec) by (Full scans/sec), the number you get should be less than 1000.
- Batch requests/sec
- Forwarded requests/sec - Should be less than 10 per 100 batch requests.
- Page splits/sec - Should be less than 20 per 100 batch requests.

SQLServer:Buffer Manager
- Buffer cache hit ratio - Should be 95% or higher.
- Free list stalls/sec - Should be less than 2.
- Free pages - Should be at least 640, preferably more.
- Lazy writes/sec - Should be less than 20.
- Page life expectancy - Should be at least 300.
- Page lookups/sec - Divide (page lookups/sec) by (batch requests/sec) and it should be under 100.
- Page reads/sec & Page writes/sec - Should be less than 90. If not, could be your indexes or memory constraints.

You get the point. There's a lot to consider.

The consultant we have in has suggested that to get a clearer idea we should get max benchmarks using SQLIO. On the disk array with 6x300GB 15k SAS in RAID5 we got around 1400IOPS max when doing random read/write of different sizes.

File/print/other
This can be considered to be a fairly small part of the equation, unless you add VM access, but that's another category all-together.

I'll post more as I think of it.

Links
http://technet.microsoft.com/en-us/library/bb125019(EXCHG.65).aspx
http://searchsqlserver.techtarget.com/video/0,297151,sid87_gci1347899,00.html

Jetta dogbone mount

2009-05-11T10:20:00.000-07:00

After a weekend of replacing suspension bits (struts, mounts, bushings, etc), my last task was to remove the dogbone mount and replace it with a new one. I got four of the five bolts out, only to have the fifth snap off at the head. The bolt has completely corroded itself into the mount's bolt-hole, and no amount of Roost-off is helping get it out. Tried hammering, prying, etc, for an hour or so, but no luck.

H is going to bring over his torch - hopefully that'll work.

Update: I ended up jamming a cold chisel using a jack under the side that would move, then hammering some more on the stuck end, and prying up while hammering on the stuck end. Eventually it started to move and came off. The corrosion was fierce! I cleaned out the threaded holes with anti-rust stuff and brake cleaner, then blew them out with compressed air. I used anti-seize when putting it all back together, and torqued accordingly.

Sheesh!

The first two pictures are of the mount itself - the part that attaches to the subframe.
Second two are of the bolt after getting it out of the subframe/mount, and the collection of bolts/replacements.
Last one is of a Princess Auto allen key that failed spectacularly - managed to twist it 360 degrees with my bare hands and a pipe.

What to expect

2009-05-11T08:42:00.000-07:00

Good afternoon!

I'll be posting up lots of things here, partially as a reminder for myself, and partially for those random Google searches...I know some tech blogs have for sure helped me, so I'll try to give a little back.

Expect content to range from IT tech issues to engineering, automotive, photography, art, etc etc etc.

Until then...

CT