Mark Minasi's Reader Forum
Mark Minasi's Reader Forum
Home | Profile | Register | Active Topics | Active Polls | Members | Search | FAQ | Minasi Forum RSS Feed
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 HALP! Questions on Windows and Windows Server
 Active Directory
 AD / DC recovery document - comments?
 New Topic  Reply to Topic
 Printer Friendly
Author Previous Topic Topic Next Topic  

aval
Honorable But Hopeless Addict

USA
3412 Posts
Status: offline

Posted - 06/02/2008 :  11:23:00 AM  Show Profile  Reply with Quote
EDIT - document in question has been updated - please see second post.

I'm working on a step-by-step document on managing the loss of a domain controller in various situations.

I'm going to paste it below for comments (in an new post) once I've edited it for (hopefully) easier reading.

Having worked on this for some time, I have some questions myself:

Question 1

When performing post metadata cleanup operations, does anyone here use nltest.exe to deregister resources records for the removed domain controller in the various DNS zones and subzones?

I tried this the other day. It seemed to eliminate some references but not all.

Question2


After domain controller failure and seizure of FSMO roles, it seems to me that rebuilding a domain controller on the same material (once it's repaired, for example) would require a clean install:

- possibly without the help of a Symantec Ghost type image
- probably without restoring the C: drive
- certainly without a system state restore.


When I say "image". I mean a base image of the server AFTER installation of the OS and Windows updates as well as apps such as antivirus and UPS power management but BEFORE promotion to a domain controller.


At first, it might seem like the use of an image would quicken the recovery process (you wouldn't need to redo certain configurations, install the updates prior to image creation, reinstall apps mentioned above).

But which of the folowing elements would be a problem?

- The old image would contain the SID of the old OS installation? As long as it's unique...?

- Restoration of the C: drive files and folders would include the registry (everything in the C:\windows\system32\config folder) and thus references to the old domain controller?

- Restoration of the Program Files folder would do no good without the elements stored in the registry.

System State restore is out of the question.

- We could not simply opt to restore just system files or the registry (and see problem above). System State restore constitutes a whole.

- So you'd install a outdated replica of Active directory with references to former FSMO holders, old GUIDs and who knows what else. I imagine total chaos would result.

Edited by - aval on 06/08/2008 4:36:03 PM

aval
Honorable But Hopeless Addict

USA
3412 Posts
Status: offline

Posted - 06/02/2008 :  11:29:56 AM  Show Profile  Reply with Quote
UPDATE: First of all, many thanks to those who commented my first version of the document below. Taking their comments into account, I've incorporated a section on reinstalling DNS in the original body of the text as well as a reference on configuring a global catalog. I've added two sections in annex: DHCP and changing the DSRM password. I've also added a warning about USN rollback.

New sections are in blue, except for USN rollback warning in red.



Recovery operations after domain controller failure – on Windows 2003 Server


Manage FSMO roles and other key roles (DNS, GC).

- Seize FSMO roles (if applicable and if DC cannot be repaired and returned online)
- Configure another DNS server if needed.
- Configure another GC server if needed:

Configure a domain controller as a global catalog server:

http://technet2.microsoft.com/windowsserver/en/library/93ffc6d8-e4c9-4a5b-8b4c-7d426bcba5a11033.mspx?mfr=true




Notes:

- There should be more than 1 domain controller, 1 DNS server and 1 global catalog server to begin with.
- PDC role is probably most crucial.
- Schema and Domain Naming less important if not extending schema or adding/removing domains and domain controllers.
- Infrastructure important if there are multiple domains.
- Consider RID allocation overlap (options?).



NTDSUTIL – this is to seize FSMO roles - if applicable (failed DC may not hold them).

MS KB reference : http://support.microsoft.com/kb/255504

Perform on any remaining functional server (example: ns2.test.dom)

quote:
C:\>ntdsutil
ntdsutil: roles
fsmo maintenance: connections
server connections: connect to server ns2.test.dom
Binding to ns2.test.dom ...
Connected to ns2.test.dom using credentials of locally logged on user.
server connections: quit
fsmo maintenance: seize schema master
fsmo maintenance: seize domain naming master
fsmo maintenance: seize pdc
fsmo maintenance: seize rid master
fsmo maintenance: seize infrastructure master



NTDSUTIL – metadata cleanup

MS KB reference : http://support.microsoft.com/kb/216498/

Perform on any remaining functional server (example: ns2.test.dom)

quote:
C:\>ntdsutil
ntdsutil: metadata cleanup
metadata cleanup: connections
server connections: connect to server ns2.test.dom
Binding to ns2.test.dom ...
Connected to ns2.test.dom using credentials of locally logged on user.
server connections: q
metadata cleanup: select operation target
select operation target: list domains
Found 1 domain(s)
0 - DC=test,DC=dom
select operation target: select domain 0
No current site
Domain - DC=test,DC=dom
No current server
No current Naming Context
select operation target: list sites
Found 1 site(s)
0 - CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=test,DC=dom
select operation target: select site 0
Site - CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=test,DC=dom
Domain - DC=test,DC=dom
No current server
No current Naming Context
select operation target: list servers for domain in site
Found 2 server(s)
0 - CN=NS1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=te
st,DC=dom
1 - CN=NS2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=te
st,DC=dom
select operation target: select server 0
Site - CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=test,DC=dom
Domain - DC=test,DC=dom
Server - CN=NS1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,
DC=test,DC=dom
DSA object - CN=NTDS Settings,CN=NS1,CN=Servers,CN=Default-First-Site-Na
me,CN=Sites,CN=Configuration,DC=test,DC=dom
DNS host name - NS1.test.dom
Computer object - CN=NS1,OU=Domain Controllers,DC=test,DC=dom
No current Naming Context
select operation target: quit
metadata cleanup: remove selected server
[...]
metadata cleanup:


Post-metadata-cleanup operations

Eliminate references to removed domain controller in the following locations:

- Active Directory Users and Computers (ADUC)
- Active Directory Sites and Services
- DNS
- ADSIEdit

Notes:

- “ntdsutil metadata cleanup” may have already removed references in some of these locations.
- Even if the new domain controller will have the same name, references must be removed because of ownership (the new domain controller would not be the owner of the old references - DNS records for example).
- Use this command? : nltest /dsderegdns:FQDN-DC-Name /dom:FQDN-domain-name



Reinstall server OS

Resources

- OS CD or DVD
- IBM ServerGuide (or equivalent OS installation tool)
- Drivers (if not available on ServerGuide CD)
- Windows Support Tools W2K3 SP2, GPMC SP1

Notes:

- In a disaster recovery situation, reinstalling the OS "from scratch" may take too much time. Use image of configured server (before DCPROMO in the case of a domain controller). Options: Symantec Ghost, Symantec Backup Exec Server Recovery, Acronis True Image Echo.
- Besides System State, a full restore of the C:\ drive is preferred (when possible) – this restores the binaries of various apps: antivirus, UPS software, etc.



Warning: after DCPROMO is executed, do not use imaging software to create an images of a domain controller for backup. Restoring such an image will result in "USN rollback".

How to detect and recover from a USN rollback in Windows Server 2003

http://support.microsoft.com/kb/875495




DNS

Since Active Directory depends on DNS, the DNS service must be installed before executing DCPROMO (if the domain controller will also be a DNS server).

If other domain controllers present are running DNS, and if DNS is "Active Directory integrated", installing the DNS service will suffice. Replication will populate the restored DC with DNS data.

If no other domain controllers are present (for whatever reason), a DNS zone must be configured as explained below:

One can configure a DNS forward lookup zone manually or allow dcpromo to configure one.

Note: A reverse lookup zone is optional. Many networks function fine without it. It can be useful for troubleshooting and is necessary for using the NSLOOKUP tool.

If one configures the zone manually on a server named "dc1" for a domain that will be named "acme.com" (for example), the most common errors can be avoided by observing the following points:

1. In the TCP/IP settings of the future domain controller, the IP address of the preferred DNS server should be that of dc1 (it points to "self"). 127.0.0.1 can be used as well.
2. If the future domain will be called acme.com, the DNS suffix of the future domain controller should be "dc1.acme.com" rather than just "dc1". This can be adjusted in Computer properties | Computer Name | Change | More
3. Initially (and only initially), secure and non-secure DNS updates should be allowed.

After DCPROMO has finished and server has restarted, the DNS zone should be changed to "Active Directory Integrated" and only secure updates should be allowed.

Generally, DNS records are registered automatically and without a problem.

If necessary, manual registration can be performed as follows:

For A records:

ipconfig /registerdns

For SRV records:

net stop netlogon
(then)
net start netlogon

How To Install and Configure DNS Server in Windows Server 2003

http://support.microsoft.com/kb/814591



DCPROMO

Case 1: There are other DCs.

- Install OS, perform Windows Updates, install apps (AV, UPS).
- Perform dcpromo (additional DC option) and let replication take place.

Case 2: There was only one DC or all DCs are out of commission.

- Install OS, restore system state .
- If rebuilding from scratch, DSRM and “Ntdsutil auth restore” not possible or necessary.

Notes:

- Unnecessary to make future DC member of domain in which it will play this role: DCPROMO takes care of domain membership. If server is the (future) first domain controller this would be impossible anyway (cannot make it member of a domain that does not yet exist).
- Point DNS server to self for preferred DNS in TCP/IP properties – can be 127.0.0.1


Case 3 – SRV1 (for example) the FSMO role holder is down, but can be restored within several hours. NTDSUTIL metadata cleanup has NOT been performed on another DC.

1. Reboot SRV1 from UBCD4W (or equivalent tool). Note: RAID and SCSI drivers?
2. Reimage with member server image (state of server before DCPROMO) with UBCD4W and Symantec Ghost (or equivalent tools). Note: this erases all traces of Active Directory on the server in question – if the disaster had not done so already.

3. Perform backup of C:\ and System State.
4. Reboot.

Notes:

- Since server is rebuilt immediately or very quickly, there is no need to seize FSMO roles or clean metadata.
- If you seize FSMO roles, then you cannot bring the former FSMO holder server online!
- Since there is another DC present, there is no need to use ntdsutil restore database. Replication will intervene here.



Annex

DHCP

Note: for continuity, there should be a DHCP failover solution in place. The following does not ensure continuity while restore operations are being performed.

If DHCP was running on the failed server, the DHCP service will need to be reinstalled as well.

Restore can be performed with the GUI or at the command line.

GUI:

Administrative Tools | DHCP | Highlight DHCP server | Action | Restore | Locate backup file

Note: default location is: C:\windows\system32\dhcp\backup

Command line:

Note: command-line backup of DHCP is performed with this command:

netsh dhcp server dump > dhcpbackup.txt


By default, the backup file is saved to C:
For example:

C:\dhcpbackup.txt

To restore, locate the backup file and execute this command:

netsh exec dhcpbackup.txt


Change DSRM password

If you don't remember what password you selected for Directory Service Restore Mode (DSRM) when running DCPromo, you should change it to something you remember before having to restore Active Directory.

This is the procedure (using NTDSUTIL) on a server named "SERV3":

C:\>ntdsutil
ntdsutil: set dsrm password
Reset DSRM Administrator Password: reset password on server SERV3
Please type password for DS Restore Mode Administrator Account: *********
Please confirm new password: *********
Password has been set successfully.

Note: if changing password on the local server, this command can be used as well:

reset password on server null


How To Reset the Directory Services Restore Mode Administrator Account Password in Windows Server 2003

http://support.microsoft.com/kb/322672



Edited by - aval on 06/08/2008 5:08:54 PM
Go to Top of Page

NikolasE
Major Contributor

Cyprus
939 Posts
Status: offline

Posted - 06/03/2008 :  08:18:35 AM  Show Profile  Click to see NikolasE's MSN Messenger address  Reply with Quote
Hello Aval

I had 2 DC (DC1 as Fsmo Holder Ad , DNS , GC & DC2 as AD , DNS , GC)

What i did in a test enviroment so pls if any mistake correct me.

I shutdown Immediately the DC1 (Fsmo Holder) and deleted the image because i am using Virtual PC for Test.

1: I Seized the Roles using NTDSUTIL to DC2
2: DC2 was already set as Global Catalog or if it wasnt i would set it to be GC
3: I run NTDSUTIL to clean metadata about DC1
3: Check in AD , AD site and Services if DC1 Still exist if yes i delete it if no then NTDSUTIL did the job.
4: On DC2 AD integrated DNS i manual deleted carefully all the records associated with DC1 ip (I guess this could be done automatically if Aging/Scavenging is enable but if you consider build a DC immediately as Additional and you want to use the same name and Ip i guess RR for Failed DC1 will must be cleared.
5: If DC2 has Alternative DNS the DC1 Ip i make sure that is not pointing to it anymore.
6: I make sure all clients would point to DC2 as Preffered DNS.

This is the Article i followed for in case of DC failure.

http://www.petri.co.il/delete_failed_dcs_from_ad.htm


Case 1: There are other DCs.

- Install OS, perform Windows Updates, install apps (AV, UPS).
- Perform dcpromo (additional DC option) and let replication take place.

Question 1 : This means if i have more than one Domain Controller for a Domain in case of failure of one of the DC the best is to Install Fresh Windows on the Server and promote it again as Additional DC?

Or Install Windows Server Os and do restore of system state?

Question 2: If i have more than 2 DCs this means on each Live DC i must run the NTDSUTIL to clean metadata about the failed DC?
(Note : I know that if you delete RR in DNS, replication of the Deleted RR will occur on other DCS DNS for the same Domain.)








Edited by - NikolasE on 06/03/2008 09:05:33 AM
Go to Top of Page

aval
Honorable But Hopeless Addict

USA
3412 Posts
Status: offline

Posted - 06/03/2008 :  09:16:44 AM  Show Profile  Reply with Quote
First of all, thanks for your comments.

As someone else pointed out to me off the board, I need to consider DHCP:

1) If it's on the failed DC, one obviously has to deal with that.
2) If it's handing out information pertaining the the old DC that's not longer valid, that too must be addressed.

Your point 5 and 6 are good and I should address that in my document. If DC1 is restored quickly and uses the same IPs, DNS preferences for clients should be OK though. Even so, I think some mention of this should be made.

Overall, your steps look correct. If you're following Daniel Petri's directions you should be OK since he knows what he's doing.

As for your questions:

quote:
Question 1 : This means if i have more than one Domain Controller for a Domain in case of failure of one of the DC the best is to Install Fresh Windows on the Server and promote it again as Additional DC?

Or Install Windows Server Os and do restore of system state?


A fresh install is absolutely necessary if you removed the failed domain controller from Active Directory using ntdsutil metadata cleanup.

Restoring system state, and thus - in this case - and old copy of the Active Directory, would have negative consequences. Either the other domain controllers would simply not communicate with the restored DC (which would accomplish nothing) or worse, there would be two domain controllers that hold the FSMO roles (assuming the failed domain controller held these roles).

If the failed DC could be repaired and restored relatively quickly, and metacleanup was not necessary, then you could perform system state restore but just not make the database authoritative (just reboot after system state restore and let replication take care of the rest).

quote:
Question 2: If i have more than 2 DCs this means on each Live DC i must run the NTDSUTIL to clean metadata about the failed DC?
(Note : I know that if you delete RR in DNS, replication of the Deleted RR will occur on other DCS for the same Domain.)


Once the metadata concerning the failed DC is removed on one DC, it will no longer be part of active directory. You don't need to perform the operation on each domain controller. That's a good thing since some organizations have dozens if not hundreds of domain controllers

Edited by - aval on 06/03/2008 09:21:35 AM
Go to Top of Page

NikolasE
Major Contributor

Cyprus
939 Posts
Status: offline

Posted - 06/03/2008 :  09:42:39 AM  Show Profile  Click to see NikolasE's MSN Messenger address  Reply with Quote
A fresh install is absolutely necessary if you removed the failed domain controller from Active Directory using ntdsutil metadata cleanup.

= Thanks i needed to verify that.




Go to Top of Page
  Previous Topic Topic Next Topic  
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
Mark Minasi's Reader Forum © 2002-2011 Mark Minasi Go To Top Of Page
This page was generated in 0.06 seconds. Snitz Forums 2000