EMC Data Domain Advanced Command Troubleshoot

EMC Backup and Recovery with Data Domain

Differences between NetWorker 9.0 and previous releases Part 1

Differences between NetWorker 9.0 and previous releases Part 1

Note : Reference by EMC Networker

This Post given comparison of NetWorker 9.0 with the previous

Major release, NetWorker 8.2.

First we are going through NetWorker installation changes:

How to Delete a Networker License key

Open "Command Prompt" "Run As Administrator" or Unix "Root User"

To stop NetWorker services and dependent server services, "NetWorker Backup and 
Recover Server" and "EMC GSTService". on a Windows host:


Make sure you keep copy of res directory.

C:\>cd "C:\Program Files\EMC NetWorker\nsr\res"

C:\>nsradmin -d nsrdb

bash-$sudo nsradmin -d "/nsr/res/nsrdb"

NetWorker administration program.
Use the "help" command for help.
nsradmin>

Enter the following commands at the nsradmin prompt:

nsradmin> type: NSR license

nsradmin> print

nsradmin> delete

Delete? y

nsradmin> print

No resources found for query:

type: NSR license;

nsradmin> q

Start the NetWorker services and "EMC GSTService"

How to recover the Clone Save Set

How to test cloned save set recovery
How to test single file cloned recovery

We have two Networker Sites and Two Data Domain Sites

Networker1 : 

Data Domain 1 : Backup of Networker1 is stored on DD1 and Cloning to DD2

Data Domain 2 : Disaster Site DD 

We have data on both Sites.

But for testing we have to restore cloned save set (e.g from DD2 only single file daemon.raw file)

Backup Data Domain Volume: DD1Volume.AA
Clone Data Domain Volume:   DD2CloneVolume.BB

In this scenario  you should run first normal recovery like as below:

bash-4.1$ sudo recover -c backuptest.com -R backuptest.com -iR

Current working directory is /

recover> cd /nsr/logs  (Check your files path)

recover> add daemon.raw

1 file(s) marked for recovery

recover> volume

Volumes needed (all on-line):

        DD1Volume.AA at rd=storagenode01.backuptest.com:DD1DISK01

recover>

But above volume is belong to DD1 not Cloned DD2.

Go to following path:

Login NMC >>> Backup Server >>> Media  >>> Disk Voulumes >>> DD1Volume.AA

For single file you cannot find there SSID you can find SSID for whole save set.

Then run recover again with same procedure like above.

bash-4.1$ sudo recover -c backuptest.com -R backuptest.com -iR

Current working directory is /

recover> cd /nsr/logs  (Check your files path)

recover> add daemon.raw

1 file(s) marked for recovery

recover> volume

Volumes needed (all on-line):

        DD2CloneVolume.BB at rd=storagenode02.backuptest.com:DD2DISK01

And recovered on same server on E:\ drive

recover> relocate

6479:recover: <return> will recover files into their original location

New destination directory: E:\restored_11-11-2015

recover> recover

After Test recover is completed you can revert the changes:

 Login NMC >>> Backup Server >>> Media  >>> Disk Voulumes >>> DD1Volume.AA and mark save set from suspect to Normal.

If you want to recover whole save set like “/” you can find the SSID and Clone ID using mminfo command.

bash-4.1$ mminfo -q client=backuptest.com-ot -r "client,name,volume,sscreate,ssretent,ssid,cloneid"

client                   Save Set name         volume               created        ssid          clone id

backuptest.com      /                      DD1Volume.AA          12/02/2015 123456789  1111111111

backuptest.com     /                      DD2CloneVolume.BB   12/02/2015 123456789  2222222222

recover> recover -S 123456789/2222222222

For recovery more details check my below posts:

<http://learnbackupwithmoin.blogspot.com/2015/11/save-set-recover-from-command-prompt.html>

How to recover expired save set

• I am going to explain how to recover the expired save set or How to recover expired EMC Networker Data  or How to make Expired Save Sets to browsable for particular save set:

• Note:  If the browse and retention policy for the saveset has expired and the tape has been reused, you will be unable to recover the contents of the Saveset.

• Backup Software : EMC Networker

I will provide details as below scenario:

• Today Date : 12/05/2013

• Client want to restore data : 06/12/2013

• Browse/Retention Policy: Three month only

When I tried to browse the date 06/12/2013 with changetime command.

recover> changetime 06/12/2013

18360:recover: cannot browse /P/USER/ZZZZ55/docs/Outlook/ as of `06/12/2013 23:59:59' (mm/dd/yyyy)

recover>

Note: Need to follow below step to find the Index Tape related to that date,               ran mminfo to find the ssid of the index save set I'm looking for.

mminfo -r "name,savetime(17),ssid,cloneid,volume,location,level,ssretent" -q "client=Backup Server,name=index:client Name,savetime>06/01/13,savetime<06/30/13" -ot

name            savetime(17)      ssid     cloneid    level    volume      location    ssretent sumflags

index:BackupClient   6/12/2013   62433892   1371148564 full   11103L5      IBM3310      9/4/2013  cE

cE : E indicate Expired Save set

Make tape not recyclable as well as extend the expire time by Type -S and -o manully on command line:

nsrmm -S <ssid/CloneID> -e <new expiration date> ( DD/MM/YYYY )

nsrmm -S <ssid/CloneID> -o notrecyclable

nsrmm –S 62433892/1371148564  –e 12/12/2013

nsrmm –S 62433892/1371148564   –o notrecyclable

mminfo -r "name,savetime(17),ssid,cloneid,level,volume,location,ssretent,sumflags" -q "client=BackupServer,name=index:BackupClient,savetime>06/01/13,savetime<06/30/13" -ot

name   savetime(17) ssid cloneid    level  volume  location ssretent sumflags

index:BackupClient  6/12/2013  62433892 1371148564  full  11103L5  IBM3310  12/12/2013  cr

Cr : r indicate recoverable save set

State changed from cE to Cr Expired to recoverable and Expire date now changed from 09/04/2013 to 12/12/2013.

Then tried to rebuild index now.

nsrck -L7 -t MM/DD/YYYY <clientname> /  nsrck -v -L7 -t MM/DD/YYYY <clientname>

nsrck -t 06/12/13 -L7 BackupClient

recover> changetime 06/12/2013

18360:recover: cannot browse /P/USER/ZZZZ55/docs/Outlook/ as of `06/12/2013 23:59:59' (mm/dd/yyyy)

recover>

But we are getting same cannot browse error

Then search Data Tape and mark as recoverable and not recyclable and rebuild index.

mminfo -s BackupServer -r "client,name,savetime,ssid,cloneid,volume,location,ssretent,sumflags,level" -q "client=BackupClient,savetime>06/01/2013,savetime<06/30/2013" -ot | grep full

client             name    savetime      ssid           cloneid     volume         location   ssretent  sumflags level

BackupClient P:\user  06/12/13 414719219  1371108606 111023L5      IBM3310   09/04/13  hE        full

BackupClient P:\user  06/12/13 414719219  1371108606 111092L5      IBM3310   09/04/13  tE         full

hE : E indicate Expired Save set

tE : E indicate Expired Save set

nsrmm -S <ssid/CloneID> -e <new expiration date> ( DD/MM/YYYY )

nsrmm -S <ssid/CloneID> -o notrecyclable

nsrmm –S 414719219/1371108606  –e 12/12/2013

nsrmm –S 414719219/1371108606  –o notrecyclable

mminfo -s BackupServer -r "client,name,savetime,ssid,cloneid,volume,location,ssretent,sumflags,level" -q "client=BackupClient,savetime>06/01/2013,savetime<06/30/2013" -ot | grep full

client             name    savetime      ssid           cloneid        volume        location   ssretent    sumflags level

BackupClient P:\user  6/12/2013 414719219  1371108606 000023L5      IBM3310  12/12/2013   hr        full

BackupClient P:\user  6/12/2013 414719219  1371108606 000092L5      IBM3310  12/12/2013   tr         full

hr : r indicate recoverable save set

tr : r indicate recoverable save set

State changed from hE, tE to hr, tr Expired to recoverable and Expire date now changed from 09/04/2013 to 12/12/2013.

Then again try to rebuild indexes:

nsrck -L7 -t MM/DD/YYYY <clientname> /  nsrck -v -L7 -t MM/DD/YYYY <clientname>

nsrck -t 06/12/13 -L7 BackupClient

recover> changetime 06/12/2013

18361:recover: time changed to 06/12/2013 23:59:59 (mm/dd/yyyy)

Able to browse now 

Add all files and restored files as per my previous post which as below:

http://learnbackupwithmoin.blogspot.com/2015/11/directed-saveset-recovery-command-line.html

Also check Below Useful Commands:

• mminfo -avot (a for all, v for verbose, o for order and t for time)

• mminfo -q "volumename=<volname>" -r "name,ssid,savetime(26),ssbrowse(26),ssretent(26)" - this queries the media database for information relating to a specific volume

• mminfo -q"client=<clientname>" -r"name,ssid,ssbrowse(26),ssretent(26),ssflags,savetime(26)" - this queries the media database for information relating to a specific client

•  mminfo -avV -q"ssid=<ssid>" -r"ssbrowse(26),ssretent(26),savetime(26)"

• nsrinfo -t "US date and time for saveset" <clientname> - this is to list the files in the saveset

•  nsrmm -S <ssid/CloneID> -e <new expiration date> ( DD/MM/YYYY )

• nsrmm -S <ssid/CloneID> -o notrecyclable

•  nsrck -L7 -t MM/DD/YYYY <clientname> /  nsrck -v -L7 -t MM/DD/YYYY <clientname>

• mminfo -avV -q"ssid=<ssid>" -r"ssbrowse(26),ssretent(26)"

• scanner -i -S <ssid> <device path as configured in jukebox>

• scanner -S ssid -iv \\.\device.

• nsrjb -ln -S <slot number where tape is located> -f <device path as configured in jukebox>

Data Domain Initial Configuration using Enterprise Manager

Save set recover from the command prompt

Save set recover:

The save set selection recovery method, or save set recover enables you to recover data without browsing and selecting the files for recovery. Unlike a browsable recovery, a save set recover does not inspect the client file index for information about each selected file. 


When you perform a save set recovery, NetWorker recovers the last full backup first, then recovers incremental backups in the chronological backup order. 

Use a save set recovery in the following scenarios:


• To recover many files or all the data in a save set, for example, if there is a total disk failure. When you perform a save set recovery, you do not select   individual files or directories for recovery.

• To recover data from a recyclable save set. 

• To recover data on a host with limited memory resources. A save set recovery requires less memory than a browsable recovery.




1. Connect to the target host with the root account on UNIX or the Administrator on Windows.

2. Use the mminfo command to display information about the save set of the data that you want to recover.

For example, type: mminfo -av -r volume,savetime,client,ssid,cloneid,name

Output similar to the following appears:

volume            date                 client                 ssid              clone      id name
backup.001 05/03/2015  bu_iddnwserver 3644194209 1362492833  C:\ddlib
clone.001   05/03/2015   bu_iddnwserver 3644194209 1362493448  C:\ddlib

To recover the all the data from a backup save set, type the following command:

recover -S ssid - i_recover_option

where:

-S ssid is the SSID of the backup save set.

-i_recover_option

• iN does not recover the file when a conflict occurs.
• iY overwrites the existing file when a conflict occurs.
• iR renames the file when a conflict occurs. The recover process appends a .R to each recovered file name.


recover -S 3644194209 -iR

To recover the all the data from a clone save set, type the following command:

recover -S ssid/cloneid

*ssid is the SSID of the backup save set.

*clonied is the cloneid of the clone save set.

recover -S 3644194209/1362493448

To recover a single directory from the clone save set and relocate the data to a new directory location, type the following command:

recover -S ssid/cloneid -d destination_dir original_dir


Example : Performing a recover in interactive mode

1. Type: recover

2. To select the files or directories to recover:

a. Specify the directory to browse, for example:

recover> cd /var/adm

b. Select the file or directory for recovery:

recover> add file_name

3. To view the selected files or directory:

recover> list

4. To view the list of the volumes that NetWorker requires to recover the data:

recover> volumes

5. To recover the files to a location that differs from the original location, type:

recover> relocate path

6. To start the recovery operation, type:

recover> recover

7. When the recovery process completes, messages similar to the following appear:

Received 1 file(s) from NSR server `jupiter'

Recover completion time: Tue Jan 21 08:33:04 2009

recover>

8. To close the recover program, type Quit.


Example: Performing a recover in non-interactive mode:
To recover a directory /testdir on client mar

To recover a directory /testdir on client mars to a new directory /newdir on client mars, type:

recover -d /newdir -a /testdir


Example: Save set recover from command line


To recover a directory /testdir from a save set with a ssid of 12345678 and overwrite any existing files with the same name, type this command:

recover -S 12345678 -iY /testdir

Example: Performing a directed recover

To recover a directory c:\mydir that was backed up on client mars to client jupiter, type:


1. On client mars, type:

recover -R jupiter -iY

2. Add the c:\mydir directory:

recover> add c:\mydir

3. Recover the files:

recover> recover

Also check below commands:

recover -a -s NetWorker_servername -c source_host -S ssid/cloneid -d destination_directory -R destination_host -i_recover_option [directory_name]

-a used to perform the recovery in non-interactive mode. You cannot use the -a option with the -R option. You cannot use the -a option to restore 
 Windows SYSTEM or VSS save sets.

-s NetWorker_servername specifies the name of the source client’s NetWorker server.

-c source_host—Specifies the source host.

-S ssid/cloneid used to perform a save set recover.

-d destination_directory

- i_recover_option

• iN does not recover the file when a conflict occurs.
• iY overwrites the existing file when a conflict occurs.
• iR renames the file when a conflict occurs. The recover process appends a .R to each recovered file name.

◆ source_directory—Specifies the initial directory in which to begin browsing.

Note ; Also check below links

<https://community.emc.com/blogs/moinmansuri/2015/11/05/directed-saveset-recovery-command-line>

How to check Avamar node capacity- MSG_ERR_DISKFULL

Last GC: finished Wed Aug 25 01:01:04 2010 after 00m 50s >> recovered 0.00 KB (MSG_ERR_DISKFULL)

The total grid utilization is currently at 88% and I also saw some Unacknowledged Events with the following information:

Code: 4202 Message: failed garbage collection with error MSG_ERR_DISKFULL

This error is a direct result of the garbage collect run limit being reached or exceeded due to excessive checkpoint overhead. To verify and check all of the node capacities use the following commands; also if this is a single node you will not have to use the mapall command.

su – admin

ssh-agent bash

ssh-add ~admin/.ssh/admin_key


mapall –noerror ‘df -h’

This should give you the filesystem for each node including the sizes, used, and space available. Then run:

avmaint nodelist | grep percent-full

This output gives you a true reading of the OS capacity utilization.  On a grid where data nodes use a file pool, the Linux "df" command is not meaningful because the stripes are pre-allocated in the file pool, and many of the stripes might not be in use.

Data Domain Data Invulnerability Architecture Lightboard Video

Deduplication with Data Domain Lightboard Video

Saveset Recover to restore NDMP data with following command

Saveset Recover to restore NDMP data with following command

  nsrndmp_recover  -s <networker_server_name> -c <client_name>  -S <ssid> -v off  -m <hostname::/path_to_recover_to> <path_to be_recovered>

  Where:     -s <networker_server_name> is the hostname of the NetWorker server,

                   -c <client_name> is the client hostname (NDMP server: filer/datamover ) that this data originated from

                 -m option uses for new destination either on the same NDMP server but different directory, or the different NDMP server

                    <hostname::/path_to_recover_to> hostname of the NDMP server (filer/datamover) you are recovering to followed by 2 colons and

                               the full path starting with the mountpoint down to the directory you would like to recover this data to

                   The directory in path-to-recover-to MUST be existence on the host.

                 <path_to_be_recovered> is the full path name starting at the mountpoint down to the directory you would like to restore.

−c client Client is the name of the machine that saved the files. 

−F This option specifies that a file level recovery is going to be performed. This option should only be specified by nwrecover(8) or recover(8). 

−m mount point The mount point of the raw device specified by the -r option. The filesystem will be unmounted for the recover operation and mounted after the operation is complete. 

−r raw device Specifies the pathname of the raw device the data is to be recovered to. This option is mandatory for destructive recovers. 

−s server Selects which NetWorker server to use. 

−S ssid This mandatory option is used to specify save set recover mode. This mode can be used to implement fast batch file recovery without requiring the NetWorker file index entries. Ssid specifies the save set id for the save set to be recovered. NDMP generated save sets cannot be cloned, so no cloneid applies as it does for the standard recover(8) command.

EMC Networker Recovery Command Line

·         Run  the below commands from backup server where client resides.

recover -c "source client name" -R "destination client name" -iR

y - overwrite the file, continue to prompt if more duplicates are found

Y - overwrite the file, do not continue to prompt

n - do not overwrite, continue to prompt

N - do not overwrite the file, do not continue to prompt

r - rename recovered file, continue to prompt

R - rename recovered file, do not continue to prompt

·         If your source and destination is same e.g abc.com

recover -c abc.com -R abc.com -iR

·         If your source and destination diffrent e.g source : abc.com, Destination: xyz.com ( but here you should check server file entry as well as remote host entry)

recover -c abc.com -R xyz.com -iR

·         then go to required save set path with cd command (e.g c:\data\vd\path1)

recover -c abc.com -R abc.com -iR

·         you will get the prompt

recover>cd c

recover>cd data

recover>cd vd

recover>cd path1

·         you can directly go to actual path cd "c:\data\vd\path1"

check the version if you want to browse the old version.

recover>version

·         select version which you want. (e.g 08/30/2015)

recover>changetime 08/30/2015

·         select which file you want to restore or if you restore all files.

recover> add *

·         if you want to restore on original location run command recover

recover> recover

·         if you wan to restore other location

recover> relocate

6479:recover: <return> will recover files into their original location

New destination directory:c:\data\vd\restored

recover> recover

Basic trouble shooting steps

Basic trouble shooting steps:

We have assumed below details for backup server and client name:

abcbackupserver.com=  Backup Server
xyzclient.com =            Client name


On Client:

• Ensure networker client package is installed:

• Ensure networker services are running on client:

• Networker requires that both forward and reverse FQDN resolution is working fine.  Nslookup should be checked from both backup server and client (Both reverse and forward lookup):

         nslookup  abcbackupserver.com
         nslookup  "IP address" {Reverse Lookup}

• Check and update the hosts file entries on the client and backup server if you have DNS issue.

        Unix/Linux : /etc/hosts
        Windows : C:\Windows\System32\drivers\etc

• You should check if your client/backup server can contact each other.

From client :
tracert  or traceroute  abcbackupserver.com


From Backup server:

• ping -a  xyzclient.com
• rpcinfo -p xyzclient.com

    program vers proto   port  service
    100000    2   tcp   7938  portmapper
    100000    2   udp   7938  portmapper
    390436    1   tcp   9842
    390435    1   tcp   9164
    390113    1   tcp   7937  nsrexecd

• Networker requires that both forward and reverse FQDN resolution is working fine.  Nslookup should be checked from both backup server and client (Both reverse and forward lookup):

        nslookup  xyzclient.com
        nslookup  "IP address" {Reverse Lookup}

• If networker client services are running fine even though you are getting failed error like nothing to save or not properly configured, then you should check nslookup as well as below command and check for any canonical name or any aliases name for failed clients. If found, they should be added to the Aliases entry.

        nsradmin -s  xyzclient.com  -p 390113

bash-4.1$ nsradmin -s xyzclient.com  -p 390113
NetWorker administration program.
Use the "help" command for help, "visual" for full-screen mode.
nsradmin> p
                        type: NSRLA;
                        name: xyzclient.com ;

• Here you will also get the any virtual or cluster configuration IP associated with failed client in IP address section. If you observed different IP’s, nslookup them and add those in Remote Access entry.

        IP address: x.x.x.x, y.y.y.y, z.z.z.z

• If you are getting connection refused or backup server not present in client server file. Also check your backup server's entry should present in client server file which is located

       Unix: /nsr/res/servers
       Windows: C:\\installation directory\\Legato\\nsr\\res\\ servers

• You should check if your client/backup server can contact each other.

From Backup Server :

• tracert  or traceroute  xyzclient.com  

• If you are getting connection refused or Time out error make sure below port range should open Bi-Directional

• Need to allow bi-directional TCP traffic on ports 7937-9936 to backup data from the Client to the Backup Server

• Need to allow bi-directional TCP traffic on ports 7937-9936 to backup data from the Client to the Storage Node

EMC NetWorker Snapshot Management for NAS Configuration, Backup, and Rec...

EMC NetWorker 8.1 for Oracle Video

Protecting NAS Systems with EMC NetWorker Snapshot Management

Networker Command mminfo output details meaning

mminfo output and Meaning of saveset flags:

The save set ssflags summary are as follows: CvrENiRPKIFk

C = Continued

v = valid

r = purged (recoverable)

E = Eligible for recycling
N = NDMP generated
i = incomplete
R = Raw (not for savesets backed up using rawasm)
P = snapshot
K = cover
I = In-progress
F = Finished (ended)
k = checkpoint restart enabled

The save set sumflags field summary are as follows:

The first flag in the sumflags field shows which part of the saveset resides on the specified tape. The possible values are:

c = Complete (the entire saveset resides on this tape volume)

h = Head (the head, or start of the saveset resides on this tape volume)

m = Middle (the middle of the saveset resides on this tape volume)

t = Tail (the end of the saveset resides on this volume)

The second flag byte shows the status of the saveset. We will not be using this flag in our cloning script, the possible values for this flag byte are:

b = Browsable

r = Recoverable

a = Saveset was aborted

I = Saveset is still in progress

E = Eligible for recycling

S = Saveset was scanned in

i = Incomplete saveset

The clone's clflags summary are as follows: aios

a = aborted

i = incomplete
o = opaque (NDMP clone of regular NetWorker savesets)

s = suspect (read error)

The volume's volflags summary are as follows: dr

d = dirty (in use)

r = scan and read-only

NDMP and NDMP-DSA flags:

cbN/:

         c Complete

         b Browsable

         N NDMP generated

cbNs:

        c Complete

        b Browsable

        N NDMP generated

        s NDMP-DSA

Query the media database to determine which save sets are NDMP save sets and the status of an NDMP save set in the media database. NDMP save set status information is

important when performing NDMP recoveries.

• To perform a browseable NDMP recover, the ssflags (fl) field for an NDMP save set must contain a (b). The b value denotes a brows able save set.

• To perform a save set recover from the NetWorker User program, the ssflags (fl) field for an NDMP save set must contain either (r) or (b).

• An NDMP save set contains an N attribute in the ssflags (fl) field.

• An NDMP-DSA save set contains an s attribute in the ssflags (fl) field.

In the following example, the NDMP save set status is recoverable (r). To recover the data, you can only perform a save set recovery from a command line.

In the following example, the NDMP-DSA save set status is brows able (b). Recover the data by using the NetWorker User program, or from the command line. A browseable NDMP-DSA save set supports brows able and save set recoveries.

The volume's state summary are as follows: EMXA

E = Eligible for recycling

M = Manually-recyclable
X = Both
A = Archive or migration volumes.