There has been a great deal of confusion surrounding obituaries stored in the directory, and as a result, some people have developed poor business practices to deal with then. Unlike some directory products, eDirectory ensures referential integrity between objects. For example, if Group A has a member, User B, and User B is later deleted, the directory automatically removes the reference to User B from Group A. Obituaries exist as operational attributes placed on objects by eDirectory as another way of ensuring referential integrity during delete, move, rename, restore, and other operations.
There are three general classifications for obituaries: Primary, Secondary and Tracking. Primary obituaries includes obituaries of type Dead (0001), Restored (0000), Moved (0002), New RDN (0005), and Tree New RDN (0008). Secondary obituaries are generally associated with a Primary obituary, and represent the agents and partitions that need to be notified of the operation specified in the Primary obituary. Secondary obituaries include the types Back Link (0006), Used By (000C), and Move Tree (000a). Tracking obituaries include Inhibit Move (0003), Old RDN (0004), and Tree Old RDN (0007).
Obituaries, with the exception of Tracking obituaries, must move through a set of synchronizing states:
The states are recorded in the Flags field in the obituary attribute. Before an obituary can move to the next state, the current state must have been synchronized to all replicas of the real object. In order to determine whether all replicas in the ring have seen a given obituary state, a vector is computed from the Transitive Vector. In eDirectory 8.6 and later, a non-stored Obituary Vector is used. In previous versions of eDirectory, the Purge Vector is used. If the Modification Timestamp (MTS) on the obituary is older than the corrupted vector, the server responsible for that obituary can advance it to the next state.
For a Secondary obituary of type Back Link, the agent that holds the master replica of the object with the obituary is responsible for advancing the states. For a Secondary obituary of type Used By, the replica agent that created it is responsible for advancing the obituary states as long as that replica still exists. If it does not still exist, the agent holding the master of that partition takes over advancing the obituary states for the Used By obituary. For a Move Tree obituary, the master of the root partition is responsible for advancing the states.
Primary obituaries can only be advanced in their states after all Secondary obituaries have advanced through all of their states. After the Primary obituary reaches its last state, and that state synchronizes to all servers in the ring, all that remains is the object husk, which is an object without attributes, one which can subsequently be purged from the system by the Purge Process. Tracking obituaries are removed after the Primary obituary is ready to be removed, or in the case of Inhibit_move, the Tracking obituary is removed after the Primary obituary has moved to the OBF_NOTIFIED state on the master replica.
The replica responsible for processing obituaries does so on a background process (the Obituary Process), which is scheduled on a per partition basis after a given partition finishes an inbound synchronization cycle. If there are no other replicas of the partition, the Outbound Replication Process is still scheduled on the heartbeat interval. The Outbound Replication Process then starts the Obituary Process. The Obituary Process cannot be manually scheduled, nor does it need to be. As synchronization occurs, the Transitive Vectors are updated, thus advancing the Purge Vector and Obit Vector. As these vectors move forward, the obituary states are allowed to move forward. This, together with the automatic scheduling done upon inbound synchronization, completes the obituary processing cycle. Therefore, the lifeblood of obituary processing is object synchronization.
For an object that is being removed, after all obituaries whose associated Primary obituary is of type DEAD have been advanced to the last state (Purgeable), and that state has been synchronized to all replicas, a new process is responsible for removing the remaining entry husk from the database. The Purge Process runs automatically to remove these husks. You can manually schedule the Purge Process and modify its automatic schedule interval by using
the Agent Configuration page in iMonitor.This section contains the following examples:
If the obituary is a Primary obituary, and there are no Secondary obituaries, and the attribute’s modification time (MTS) on the obituary is older than the Purge Vector, then all servers have seen the change and this obituary will be removed.
If the obituary is a Back Link obituary and this server is the master, then this server is responsible for processing this obituary.
Perform the required operation for this state if it has not been done. Most often, this is done by notifying an external reference.
If the obituary is a Used By obituary, and this server is the server where the delete occurred (determined by comparing the replica number in the obituary’s MTS to our replica number), this server is responsible for processing this obituary.
Move looks much like Delete, but with the following changes:
Objects with obituaries are considered every time an agent outbound synchronizes, and by the obituary process, which is scheduled to run at the end of an inbound synchronization cycle.
On a regular basis, run the iMonitor Server Information report. This report walks the entire tree, communicates with every NCP server it can find, and reports any errors it finds. You can use this report to diagnose time synchronization and limber problems, or to find out if the current server is able to communicate with all other servers from this server's perspective. If selected in the configuration page, this server can also generate NDS Agent Health information for every server in the tree. See Report Config for more information on running the Server Information report.
If you are using iMonitor 2.0 or later, make sure that the following report options are enabled: Errors and Health sub-report. The following items will be verified. You should browse the report and make sure that there are no errors.
If you are using iMonitor 1.5, select the Errors report option. The following items will be verified. You should browse the report and make sure that there are no errors.
Using the iMonitor Obituary Listing report or the iMonitor Object Statistics report, you can find any obituaries on your system. If you find any obituaries that you don’t believe are being processed, see Troubleshooting Tips.
There are two general reasons that obituaries don’t process: either the obit has been orphaned (that is, the obituary exists on some servers but not all servers), or the obituary is stuck (that is, it exists on all servers, but its states are not advancing for some reason).
Use the following items to troubleshoot orphaned or stuck obituaries:
–625, -622, -634, and -635 communication problems. See Server Information Report for more details.
–601, and -603, indicating servers that have been improperly removed, or that the server object might have a base class of Unknown.
Use the proper solution referred to in Troubleshooting Tips.
Before using any of these solutions, you must make sure that your data is safe. You might need to back up the directory database files, server configuration, and trustees. To increase the probability of success and to minimize future problems, upgrade to the latest eDirectory support packs.
- Preferred method: If eDirectory 8.6 or later is on any of the servers in the replica ring, browse to the object in iMonitor, then select Send Single Entry. This will perform a non-authoritative send to all other replicas.
- Less desirable method: Enable Advanced Mode in iMonitor, browse to the object and select Advanced Options, then Timestamp Entry. This will make the object as it exists on this server the authoritative copy. Novell does not recommend making objects authoritative as a matter of practice.
- Far less desirable method: If all servers in the replica ring that have a copy of the orphaned obituary are older than eDirectory 8.6, load DSBrowse with the –a option, browse to the object, then select Resend Selected Object to timestamp the entry. This will make the object as it exists on this server the authoritative copy. Novell does not recommend making objects authoritative as a matter of practice.
Resolving Orphaned Obituaries on Extrefs
- Preferred method: Run the most current DSRepair version with the option -XK3 selected. However, first check to see if all the operations performed by -XK3 are acceptable.
- Less desirable method: Move a real replica to the server, wait for it to turn on, then wait for the obituary to be processed. If the obituary is not processed, use the information in Troubleshooting Tips to resolve the issue now that the object is on a real replica. After the obituary has processed, the replica can be removed if desired.
Resolving OBT_INHIBIT_MOVE Obituaries
- Preferred method: If the destination object is complete, see if the source still exists. If the source object exists and still has the OBT_MOVED attribute, then troubleshoot and resolve what has prevented the OBT_MOVED from completing. If there is no source object, then use the Advanced mode iMonitor options to release the OBT_MOVE_INHIBIT on the destination objects master replica. If the destination object is incomplete, then release the OBT_MOVE_INHIBIT and remove the destination object.
In the past, several different strategies have been employed to clean up stuck obituaries. Some of these strategies involve expensive partitioning operations, or the use of undocumented features that might cause problems in the future. Many of the strategies listed below should not be used.
The first strategy was to switch which replica held the master. This would work in some cases because the master is the agent responsible for moving the Back Link obituaries through their various states. In the case where the replica was inconsistent and the master didn’t hold the deleted object, switching masters to an agent that held the deleted entry with its obituaries would give the new agent the license to push the obituaries through their states and eventually purge it out. Send Single Entry is a much cleaner and less dangerous way to resolve obituaries that are stuck because the replica is inconsistent.
The second strategy used is to run DSRepair with certain switches to delete all obituaries. (There is a third party application which cleans up stuck obituaries by launching DSRepair.) The eDirectory development team does not recommend this strategy. Using those switches will delete all obituaries on this agent, which means that obituaries that are not stuck might also be removed, creating further replica inconsistencies and more stuck obituaries. Since this is not a distributed operation, you must run DSRepair on all of the servers with stuck obituaries, which increases the odds that one of those servers has obituaries for another partition which will be prematurely deleted. The premature deletion of obituaries can cause additional orphaned obituaries, and in turn cause problems which can be found years later when you change replicas types, add new replicas, or perform other partitioning operations.
The third strategy used is to make objects authoritative, either using DSBrowse with the advanced mode operation and timestamping the entry, or running DSRepair with the –0T switch. This forces the entry to become authoritative and synchronize out to all other replicas. Timestamping and making object authoritative should be done with care because you might lose data changed on other servers. The eDirectory development team recommends that this be a rarely employed method of obituary cleanup.
For information about NetIQ trademarks, see http://www.netiq.com/company/legal/.