Obituaries

There has been a great deal of confusion surrounding obituaries stored in the directory, and as a result, some people have developed poor business practices to deal with then. Unlike some directory products, eDirectory ensures referential integrity between objects. For example, if Group A has a member, User B, and User B is later deleted, the directory automatically removes the reference to User B from Group A. Obituaries exist as operational attributes placed on objects by eDirectory as another way of ensuring referential integrity during delete, move, rename, restore, and other operations.

There are three general classifications for obituaries: Primary, Secondary and Tracking. Primary obituaries includes obituaries of type Dead (0001), Restored (0000), Moved (0002), New RDN (0005), and Tree New RDN (0008). Secondary obituaries are generally associated with a Primary obituary, and represent the agents and partitions that need to be notified of the operation specified in the Primary obituary. Secondary obituaries include the types Back Link (0006), Used By (000C), and Move Tree (000a). Tracking obituaries include Inhibit Move (0003), Old RDN (0004), and Tree Old RDN (0007).

Obituaries, with the exception of Tracking obituaries, must move through a set of synchronizing states:

The states are recorded in the Flags field in the obituary attribute. Before an obituary can move to the next state, the current state must have been synchronized to all replicas of the real object. In order to determine whether all replicas in the ring have seen a given obituary state, a vector is computed from the Transitive Vector. In eDirectory 8.6 and later, a non-stored Obituary Vector is used. In previous versions of eDirectory, the Purge Vector is used. If the Modification Timestamp (MTS) on the obituary is older than the corrupted vector, the server responsible for that obituary can advance it to the next state.

For a Secondary obituary of type Back Link, the agent that holds the master replica of the object with the obituary is responsible for advancing the states. For a Secondary obituary of type Used By, the replica agent that created it is responsible for advancing the obituary states as long as that replica still exists. If it does not still exist, the agent holding the master of that partition takes over advancing the obituary states for the Used By obituary. For a Move Tree obituary, the master of the root partition is responsible for advancing the states.

Primary obituaries can only be advanced in their states after all Secondary obituaries have advanced through all of their states. After the Primary obituary reaches its last state, and that state synchronizes to all servers in the ring, all that remains is the object husk, which is an object without attributes, one which can subsequently be purged from the system by the Purge Process. Tracking obituaries are removed after the Primary obituary is ready to be removed, or in the case of Inhibit_move, the Tracking obituary is removed after the Primary obituary has moved to the OBF_NOTIFIED state on the master replica.

The replica responsible for processing obituaries does so on a background process (the Obituary Process), which is scheduled on a per partition basis after a given partition finishes an inbound synchronization cycle. If there are no other replicas of the partition, the Outbound Replication Process is still scheduled on the heartbeat interval. The Outbound Replication Process then starts the Obituary Process. The Obituary Process cannot be manually scheduled, nor does it need to be. As synchronization occurs, the Transitive Vectors are updated, thus advancing the Purge Vector and Obit Vector. As these vectors move forward, the obituary states are allowed to move forward. This, together with the automatic scheduling done upon inbound synchronization, completes the obituary processing cycle. Therefore, the lifeblood of obituary processing is object synchronization.

For an object that is being removed, after all obituaries whose associated Primary obituary is of type DEAD have been advanced to the last state (Purgeable), and that state has been synchronized to all replicas, a new process is responsible for removing the remaining entry husk from the database. The Purge Process runs automatically to remove these husks. You can manually schedule the Purge Process and modify its automatic schedule interval by using the Agent Configuration page in iMonitor.

Examples

This section contains the following examples:

Deleting an Object

  1. Add the Primary obituary OBT_DEAD.
  2. The Back Link attribute contains a list of servers that have an interest in this object and need to be notified of changes to this entry. For every DN listed in the Back Link attribute and all servers listed in the entry’s partition replica attribute, eDirectory adds a Back Link obituary. The creation time of the Primary obituary, OBT_DEAD, is stored in the Secondary obituary.
  3. The Used By attribute contains a list of partitions that have an interest in this object and need to be notified of changes to this entry. For every DN listed in the Used By attribute, eDirectory adds a Used By obituary. The creation time of the Primary obituary, OBT_DEAD, is stored in the Secondary obituary.
  4. Remove all attributes but the obituaries.
  5. The Outbound Replication Process then synchronizes this change to all other servers in the replica ring.
  6. On the next inbound synchronization of this partition, the Obituary Process is started, which does the following:

    If the obituary is a Primary obituary, and there are no Secondary obituaries, and the attribute’s modification time (MTS) on the obituary is older than the Purge Vector, then all servers have seen the change and this obituary will be removed.

    If the obituary is a Back Link obituary and this server is the master, then this server is responsible for processing this obituary.

    Perform the required operation for this state if it has not been done. Most often, this is done by notifying an external reference.

    If the obituary is a Used By obituary, and this server is the server where the delete occurred (determined by comparing the replica number in the obituary’s MTS to our replica number), this server is responsible for processing this obituary.

Moving an Object

Move looks much like Delete, but with the following changes:

  1. Before the Primary obituary is placed on the move source, a partial entry is created in the destination container and a Tracking obituary (OBT_INHIBIT_MOVE) is placed on that partial entry. This Tracking obituary is placed to prevent the entry from being moved or taking part in a partition operation before the full entry is transferred from the source.
  2. On the source entry, the Primary obituary is OBT_MOVED.
  3. After the Primary obituary OBT_MOVED is moved to the Notified state (meaning that all replicas of the source know the entry is being moved), and all external references have been notified, the Tracking obituary (OBT_INHIBIT_MOVE) is removed from the destination entry.

Impact of Stuck and Orphaned Obituaries

Objects with obituaries are considered every time an agent outbound synchronizes, and by the obituary process, which is scheduled to run at the end of an inbound synchronization cycle.

Prevention

On a regular basis, run the iMonitor Server Information report. This report walks the entire tree, communicates with every NCP server it can find, and reports any errors it finds. You can use this report to diagnose time synchronization and limber problems, or to find out if the current server is able to communicate with all other servers from this server's perspective. If selected in the configuration page, this server can also generate NDS Agent Health information for every server in the tree. See Report Config for more information on running the Server Information report.

If you are using iMonitor 2.0 or later, make sure that the following report options are enabled: Errors and Health sub-report. The following items will be verified. You should browse the report and make sure that there are no errors.

If you are using iMonitor 1.5, select the Errors report option. The following items will be verified. You should browse the report and make sure that there are no errors.

Using the iMonitor Obituary Listing report or the iMonitor Object Statistics report, you can find any obituaries on your system. If you find any obituaries that you don’t believe are being processed, see Troubleshooting Tips.

Troubleshooting Tips

There are two general reasons that obituaries don’t process: either the obit has been orphaned (that is, the obituary exists on some servers but not all servers), or the obituary is stuck (that is, it exists on all servers, but its states are not advancing for some reason).

Use the following items to troubleshoot orphaned or stuck obituaries:

Solutions

Use the proper solution referred to in Troubleshooting Tips.

Before using any of these solutions, you must make sure that your data is safe. You might need to back up the directory database files, server configuration, and trustees. To increase the probability of success and to minimize future problems, upgrade to the latest eDirectory support packs.

Resolving Orphaned Obituaries

Resolving Orphaned Obituaries on Extrefs

Resolving OBT_INHIBIT_MOVE Obituaries

Previous Practices

In the past, several different strategies have been employed to clean up stuck obituaries. Some of these strategies involve expensive partitioning operations, or the use of undocumented features that might cause problems in the future. Many of the strategies listed below should not be used.

The first strategy was to switch which replica held the master. This would work in some cases because the master is the agent responsible for moving the Back Link obituaries through their various states. In the case where the replica was inconsistent and the master didn’t hold the deleted object, switching masters to an agent that held the deleted entry with its obituaries would give the new agent the license to push the obituaries through their states and eventually purge it out. Send Single Entry is a much cleaner and less dangerous way to resolve obituaries that are stuck because the replica is inconsistent.

The second strategy used is to run DSRepair with certain switches to delete all obituaries. (There is a third party application which cleans up stuck obituaries by launching DSRepair.) The eDirectory development team does not recommend this strategy. Using those switches will delete all obituaries on this agent, which means that obituaries that are not stuck might also be removed, creating further replica inconsistencies and more stuck obituaries. Since this is not a distributed operation, you must run DSRepair on all of the servers with stuck obituaries, which increases the odds that one of those servers has obituaries for another partition which will be prematurely deleted. The premature deletion of obituaries can cause additional orphaned obituaries, and in turn cause problems which can be found years later when you change replicas types, add new replicas, or perform other partitioning operations.

The third strategy used is to make objects authoritative, either using DSBrowse with the advanced mode operation and timestamping the entry, or running DSRepair with the –0T switch. This forces the entry to become authoritative and synchronize out to all other replicas. Timestamping and making object authoritative should be done with care because you might lose data changed on other servers. The eDirectory development team recommends that this be a rarely employed method of obituary cleanup.

For information about NetIQ trademarks, see http://www.netiq.com/company/legal/.