Slow backup Issue

Networker slow backup issue

We are facing slow backup issue many times or previously added clients suddenly slow backup issue occurs. This is either because of Network glitch at server end and on client side or configuration issue at both ends.

Scenario 1:

Network End: -

Below points should always be taken care off:

·         You should be always aware about link speed between your backup server and backup clients

·         Any network or data capped is configured as switch end

·         All network cards and switches have the same Duplex, Speed, MTU and other settings

·         Also, check latency between your backup server and backup client

·         Analyze network performance or problems with netstat

Scenario 2:

Client End: -

Below points should always be taken care off:

·         Check data size and number of files in your drives (hundred thousand or millions) of small files.

·         If you have small size huge files, then backup speed will be automatically degrading. (If you have D drive with 100 GB and 400 files but you have other drive E with 50 GB and 5000000…then E will take more time to backup)

·         Defragmented disk drives

·         Make sure that the OS is up-to-date with latest patches

·         Also, check the load on server with Task Manager

•             Networker should be up to date with latest version

·         Reboot the NetWorker server, client, and storage node operating systems.

Scenario 3:

Backup Server End: -

Below points should always be taken care off:

·         Check if any Index is corrupted

·         Check if any defunct or zombie process is related to that client

·         Check if check point enabled, disabled if not require (for file server and huge data backup it should be enable so if save fails, it can be restarted from the point of interruption). This setting is tricky you will decide as per your data type.

·         Monitor and check the device write speeds over time through NetWorker debug mode

•                  Ensure enough hardware resources (that is, tape drives) are available to handle the load.
•                 Networker should be up to date about version
•                Also, check Device configuration in terms of device session max session and target session

·         With help of nsrinfo command check the what kind of data backing up on your environment (Database, any other ongoing application). If you are taking flat file system backup with ALL save set, then you should be alert about any database or SharePoint or ongoing file access application data backup.

·         If you are backing up SQL/DB/MYSQL/SharePoint server backup using ALL save set and without any respective module (means your database backup are take care differently other than Networker) and it will be taking long time to backup up the data. Here you should mention save set of backup if you do not want to backup the any DB/Application servers (with Module)

·         Rebooted the NetWorker server, client, and storage node operating systems

Note : While trouble shooting the slow backup issue you should always coordinate with Network/Server admin and also historical logs should be checked.

Browse and retention time raised Script for multiple SSID

Today I would like to explain How Browse and retention time period extend by Script for multiple SSID

bash-$ mminfo -avot -q "savetime>=02/01/2016,savetime<=03/31/2016" -r client,ssid,group,ssbrowse,ssretent >ssid.txt 2>&1

Select the respective SSID which need to extend and put in to one text file ssid-re.txt. After that run the below script for extend the Browse and retention period;

Today Date : 04/01/2016

bash-$ for i in `cat ssid-re.txt`; do sudo nsrmm -S $i -e "6 months" -y; done

bash-$ for i in `cat ssid-re.txt`; do sudo nsrmm -S $i -w "6 months" -y; done

You will get the error browse time should be less than retention time.

You should mention commands like as 

for i in `cat ssid-re.txt`; do sudo nsrmm -S $i -w "Sun Feb 28 06:16:05 2016" -y; done

After extended the both policy verify it.

for i in `cat verifyssid.txt`; do mminfo -q ssid=$i -r 'client,ssid,sscreate,ssbrowse,ssretent' >>output.txt; done 

EMC ELECT 2016

Friends I would like to share that I am elected as EMC Elect 2016 with another 70 experts from 150 finalists.

Below is the link for your reference to see the list of elected candidates

https://community.emc.com/community/connect/emc_elect/blog/2016/01/31/the-emc-elect-of-2016--the-official-list

This is a very funny incident with me about EMC elect. When I started working on EMC product and I heard from someone that there is one kind of election EMCElect. I was very surprised because I thought EMC is also conducting election like politicians  but after reading about EMC Elect I got to know that this is an honor for EMC expertise.

So that time I was very excited to grab much information about EMC Elect. I always share my knowledge regularly on EMC Community Network, sharing on Twitter too. I also started sharing my technical tips through blogs and on EMC community forum for knowledge sharing.

I was shortlisted For EMC ELECT 2013 but was not selected, and I took it as a challenge and I said to myself that next time I will take more efforts and again participate in 2014 and finally I got first EMC honor in 2014 and again in 2016.

I would like to thanks all who voted me and fulfil to complete this program.

This is unique initiative by EMC and I am really enjoying to participate year again year.

Special thanks to Mark Browne ‏@DathBrun and all known unknown people who were involved in this activity.

How to BACK UP the NetWorker Management Console (NMC)

How to BACK UP the NetWorker Management Console (NMC):

We can backup NMC through command line as well as GUI method.

First we will see the command line method:

Open the command line interface and run the command “savepsm” which will backup the NMC database. The normal Networker “save” command cannot back up NMC database successfully if it is open during backup hence it is always better to use "Savepsm" command.

Now we will see using GUI method:

Login to NMC console

Go to SETUP

Click the SETUP button on the toolbar

Select “Set Database Backup Server”

 Mention the NetWorker server name that you want to back up the NMC database.

Check "Create client resource on this server".

Set Client name to the hostname of the machine that has the NMC database.

Click OK.

After successfully configuration connect to the NetWorker server. When you look at the list of clients, you will get there a new client.

You will also see that client configuration saveset name starting with "NMCASA:/gst_on_NMC-Server-name/lgto_gst" with Backup Command attribute: savepsm (for a Windows Console server) or savepsm.sh(for a UNIX Console server).

You should verify the backup which you configured after successful first backup with mminfo command.

On Windows: mminfo -avot -s (NetWorker server) | find /i "gst"

On UNIX: mminfo -avot -s (NetWorker server) | grep -i gst

This are the method you can use to backup your NMC database.

NSR peer information' provided did not match

NSR peer information' provided did not match

nsrexecd GSS critical An authentication request from backupclient.com  was denied

If you experience errors in the backup server daemon log such as this:

89879 07/04/14 04:04:13  nsrexecd GSS critical An authentication request from backupclient.com  was denied. The 'NSR peer information' provided did not match the one stored by backupserevr. To accept this request, delete the 'NSR peer information' resource with the following attributes from backupserevr's NSRLA database: name: backupclient.com; NW instance ID: XXXXXXXX-XXXXXXXX-XXXXXXXX-XXXXXXXX-XXXXXXXX-XXXXXXXX; peer hostname: backupclient.com

It is because a client has been re-installed, renamed or the NSRLADB has been rebuilt.

To resolve this, delete the NSR peer information for the client on the backup server.

For example:

# nsradmin -p nsrexecd

NetWorker administration program.

Use the "help" command for help, "visual" for full-screen mode.

nsradmin> . type:NSR peer information;name:backupclient.com

Current query set

nsradmin> delete

                        type: NSR peer information;

               administrator: "isroot,host= backupserevr ",

                              "isroot,host=backupserevr.com",

                              "user=root,host=localhost";

                        name: backupclient.com;

               peer hostname: backupclient.com;

          Change certificate: ;

    certificate file to load: ;

Delete? y

deleted resource id 0.0.XX.XXX.0.0.0.XXX.83.XXX.XXX.XXX.XXX.XX.XX.11(1)

nsradmin> quit

EMC Data Domain Advanced Command Troubleshoot

EMC Backup and Recovery with Data Domain

How to Delete a Networker License key

Open "Command Prompt" "Run As Administrator" or Unix "Root User"

To stop NetWorker services and dependent server services, "NetWorker Backup and 
Recover Server" and "EMC GSTService". on a Windows host:


Make sure you keep copy of res directory.

C:\>cd "C:\Program Files\EMC NetWorker\nsr\res"

C:\>nsradmin -d nsrdb

bash-$sudo nsradmin -d "/nsr/res/nsrdb"

NetWorker administration program.
Use the "help" command for help.
nsradmin>

Enter the following commands at the nsradmin prompt:

nsradmin> type: NSR license

nsradmin> print

nsradmin> delete

Delete? y

nsradmin> print

No resources found for query:

type: NSR license;

nsradmin> q

Start the NetWorker services and "EMC GSTService"

How to recover the Clone Save Set

How to test cloned save set recovery
How to test single file cloned recovery

We have two Networker Sites and Two Data Domain Sites

Networker1 : 

Data Domain 1 : Backup of Networker1 is stored on DD1 and Cloning to DD2

Data Domain 2 : Disaster Site DD 

We have data on both Sites.

But for testing we have to restore cloned save set (e.g from DD2 only single file daemon.raw file)

Backup Data Domain Volume: DD1Volume.AA
Clone Data Domain Volume:   DD2CloneVolume.BB

In this scenario  you should run first normal recovery like as below:

bash-4.1$ sudo recover -c backuptest.com -R backuptest.com -iR

Current working directory is /

recover> cd /nsr/logs  (Check your files path)

recover> add daemon.raw

1 file(s) marked for recovery

recover> volume

Volumes needed (all on-line):

        DD1Volume.AA at rd=storagenode01.backuptest.com:DD1DISK01

recover>

But above volume is belong to DD1 not Cloned DD2.

Go to following path:

Login NMC >>> Backup Server >>> Media  >>> Disk Voulumes >>> DD1Volume.AA

For single file you cannot find there SSID you can find SSID for whole save set.

Then run recover again with same procedure like above.

bash-4.1$ sudo recover -c backuptest.com -R backuptest.com -iR

Current working directory is /

recover> cd /nsr/logs  (Check your files path)

recover> add daemon.raw

1 file(s) marked for recovery

recover> volume

Volumes needed (all on-line):

        DD2CloneVolume.BB at rd=storagenode02.backuptest.com:DD2DISK01

And recovered on same server on E:\ drive

recover> relocate

6479:recover: <return> will recover files into their original location

New destination directory: E:\restored_11-11-2015

recover> recover

After Test recover is completed you can revert the changes:

 Login NMC >>> Backup Server >>> Media  >>> Disk Voulumes >>> DD1Volume.AA and mark save set from suspect to Normal.

If you want to recover whole save set like “/” you can find the SSID and Clone ID using mminfo command.

bash-4.1$ mminfo -q client=backuptest.com-ot -r "client,name,volume,sscreate,ssretent,ssid,cloneid"

client                   Save Set name         volume               created        ssid          clone id

backuptest.com      /                      DD1Volume.AA          12/02/2015 123456789  1111111111

backuptest.com     /                      DD2CloneVolume.BB   12/02/2015 123456789  2222222222

recover> recover -S 123456789/2222222222

For recovery more details check my below posts:

<http://learnbackupwithmoin.blogspot.com/2015/11/save-set-recover-from-command-prompt.html>

How to recover expired save set

• I am going to explain how to recover the expired save set or How to recover expired EMC Networker Data  or How to make Expired Save Sets to browsable for particular save set:

• Note:  If the browse and retention policy for the saveset has expired and the tape has been reused, you will be unable to recover the contents of the Saveset.

• Backup Software : EMC Networker

I will provide details as below scenario:

• Today Date : 12/05/2013

• Client want to restore data : 06/12/2013

• Browse/Retention Policy: Three month only

When I tried to browse the date 06/12/2013 with changetime command.

recover> changetime 06/12/2013

18360:recover: cannot browse /P/USER/ZZZZ55/docs/Outlook/ as of `06/12/2013 23:59:59' (mm/dd/yyyy)

recover>

Note: Need to follow below step to find the Index Tape related to that date,               ran mminfo to find the ssid of the index save set I'm looking for.

mminfo -r "name,savetime(17),ssid,cloneid,volume,location,level,ssretent" -q "client=Backup Server,name=index:client Name,savetime>06/01/13,savetime<06/30/13" -ot

name            savetime(17)      ssid     cloneid    level    volume      location    ssretent sumflags

index:BackupClient   6/12/2013   62433892   1371148564 full   11103L5      IBM3310      9/4/2013  cE

cE : E indicate Expired Save set

Make tape not recyclable as well as extend the expire time by Type -S and -o manully on command line:

nsrmm -S <ssid/CloneID> -e <new expiration date> ( DD/MM/YYYY )

nsrmm -S <ssid/CloneID> -o notrecyclable

nsrmm –S 62433892/1371148564  –e 12/12/2013

nsrmm –S 62433892/1371148564   –o notrecyclable

mminfo -r "name,savetime(17),ssid,cloneid,level,volume,location,ssretent,sumflags" -q "client=BackupServer,name=index:BackupClient,savetime>06/01/13,savetime<06/30/13" -ot

name   savetime(17) ssid cloneid    level  volume  location ssretent sumflags

index:BackupClient  6/12/2013  62433892 1371148564  full  11103L5  IBM3310  12/12/2013  cr

Cr : r indicate recoverable save set

State changed from cE to Cr Expired to recoverable and Expire date now changed from 09/04/2013 to 12/12/2013.

Then tried to rebuild index now.

nsrck -L7 -t MM/DD/YYYY <clientname> /  nsrck -v -L7 -t MM/DD/YYYY <clientname>

nsrck -t 06/12/13 -L7 BackupClient

recover> changetime 06/12/2013

18360:recover: cannot browse /P/USER/ZZZZ55/docs/Outlook/ as of `06/12/2013 23:59:59' (mm/dd/yyyy)

recover>

But we are getting same cannot browse error

Then search Data Tape and mark as recoverable and not recyclable and rebuild index.

mminfo -s BackupServer -r "client,name,savetime,ssid,cloneid,volume,location,ssretent,sumflags,level" -q "client=BackupClient,savetime>06/01/2013,savetime<06/30/2013" -ot | grep full

client             name    savetime      ssid           cloneid     volume         location   ssretent  sumflags level

BackupClient P:\user  06/12/13 414719219  1371108606 111023L5      IBM3310   09/04/13  hE        full

BackupClient P:\user  06/12/13 414719219  1371108606 111092L5      IBM3310   09/04/13  tE         full

hE : E indicate Expired Save set

tE : E indicate Expired Save set

nsrmm -S <ssid/CloneID> -e <new expiration date> ( DD/MM/YYYY )

nsrmm -S <ssid/CloneID> -o notrecyclable

nsrmm –S 414719219/1371108606  –e 12/12/2013

nsrmm –S 414719219/1371108606  –o notrecyclable

mminfo -s BackupServer -r "client,name,savetime,ssid,cloneid,volume,location,ssretent,sumflags,level" -q "client=BackupClient,savetime>06/01/2013,savetime<06/30/2013" -ot | grep full

client             name    savetime      ssid           cloneid        volume        location   ssretent    sumflags level

BackupClient P:\user  6/12/2013 414719219  1371108606 000023L5      IBM3310  12/12/2013   hr        full

BackupClient P:\user  6/12/2013 414719219  1371108606 000092L5      IBM3310  12/12/2013   tr         full

hr : r indicate recoverable save set

tr : r indicate recoverable save set

State changed from hE, tE to hr, tr Expired to recoverable and Expire date now changed from 09/04/2013 to 12/12/2013.

Then again try to rebuild indexes:

nsrck -L7 -t MM/DD/YYYY <clientname> /  nsrck -v -L7 -t MM/DD/YYYY <clientname>

nsrck -t 06/12/13 -L7 BackupClient

recover> changetime 06/12/2013

18361:recover: time changed to 06/12/2013 23:59:59 (mm/dd/yyyy)

Able to browse now 

Add all files and restored files as per my previous post which as below:

http://learnbackupwithmoin.blogspot.com/2015/11/directed-saveset-recovery-command-line.html

Also check Below Useful Commands:

• mminfo -avot (a for all, v for verbose, o for order and t for time)

• mminfo -q "volumename=<volname>" -r "name,ssid,savetime(26),ssbrowse(26),ssretent(26)" - this queries the media database for information relating to a specific volume

• mminfo -q"client=<clientname>" -r"name,ssid,ssbrowse(26),ssretent(26),ssflags,savetime(26)" - this queries the media database for information relating to a specific client

•  mminfo -avV -q"ssid=<ssid>" -r"ssbrowse(26),ssretent(26),savetime(26)"

• nsrinfo -t "US date and time for saveset" <clientname> - this is to list the files in the saveset

•  nsrmm -S <ssid/CloneID> -e <new expiration date> ( DD/MM/YYYY )

• nsrmm -S <ssid/CloneID> -o notrecyclable

•  nsrck -L7 -t MM/DD/YYYY <clientname> /  nsrck -v -L7 -t MM/DD/YYYY <clientname>

• mminfo -avV -q"ssid=<ssid>" -r"ssbrowse(26),ssretent(26)"

• scanner -i -S <ssid> <device path as configured in jukebox>

• scanner -S ssid -iv \\.\device.

• nsrjb -ln -S <slot number where tape is located> -f <device path as configured in jukebox>

Data Domain Initial Configuration using Enterprise Manager

Save set recover from the command prompt

Save set recover:

The save set selection recovery method, or save set recover enables you to recover data without browsing and selecting the files for recovery. Unlike a browsable recovery, a save set recover does not inspect the client file index for information about each selected file. 


When you perform a save set recovery, NetWorker recovers the last full backup first, then recovers incremental backups in the chronological backup order. 

Use a save set recovery in the following scenarios:


• To recover many files or all the data in a save set, for example, if there is a total disk failure. When you perform a save set recovery, you do not select   individual files or directories for recovery.

• To recover data from a recyclable save set. 

• To recover data on a host with limited memory resources. A save set recovery requires less memory than a browsable recovery.




1. Connect to the target host with the root account on UNIX or the Administrator on Windows.

2. Use the mminfo command to display information about the save set of the data that you want to recover.

For example, type: mminfo -av -r volume,savetime,client,ssid,cloneid,name

Output similar to the following appears:

volume            date                 client                 ssid              clone      id name
backup.001 05/03/2015  bu_iddnwserver 3644194209 1362492833  C:\ddlib
clone.001   05/03/2015   bu_iddnwserver 3644194209 1362493448  C:\ddlib

To recover the all the data from a backup save set, type the following command:

recover -S ssid - i_recover_option

where:

-S ssid is the SSID of the backup save set.

-i_recover_option

• iN does not recover the file when a conflict occurs.
• iY overwrites the existing file when a conflict occurs.
• iR renames the file when a conflict occurs. The recover process appends a .R to each recovered file name.


recover -S 3644194209 -iR

To recover the all the data from a clone save set, type the following command:

recover -S ssid/cloneid

*ssid is the SSID of the backup save set.

*clonied is the cloneid of the clone save set.

recover -S 3644194209/1362493448

To recover a single directory from the clone save set and relocate the data to a new directory location, type the following command:

recover -S ssid/cloneid -d destination_dir original_dir


Example : Performing a recover in interactive mode

1. Type: recover

2. To select the files or directories to recover:

a. Specify the directory to browse, for example:

recover> cd /var/adm

b. Select the file or directory for recovery:

recover> add file_name

3. To view the selected files or directory:

recover> list

4. To view the list of the volumes that NetWorker requires to recover the data:

recover> volumes

5. To recover the files to a location that differs from the original location, type:

recover> relocate path

6. To start the recovery operation, type:

recover> recover

7. When the recovery process completes, messages similar to the following appear:

Received 1 file(s) from NSR server `jupiter'

Recover completion time: Tue Jan 21 08:33:04 2009

recover>

8. To close the recover program, type Quit.


Example: Performing a recover in non-interactive mode:
To recover a directory /testdir on client mar

To recover a directory /testdir on client mars to a new directory /newdir on client mars, type:

recover -d /newdir -a /testdir


Example: Save set recover from command line


To recover a directory /testdir from a save set with a ssid of 12345678 and overwrite any existing files with the same name, type this command:

recover -S 12345678 -iY /testdir

Example: Performing a directed recover

To recover a directory c:\mydir that was backed up on client mars to client jupiter, type:


1. On client mars, type:

recover -R jupiter -iY

2. Add the c:\mydir directory:

recover> add c:\mydir

3. Recover the files:

recover> recover

Also check below commands:

recover -a -s NetWorker_servername -c source_host -S ssid/cloneid -d destination_directory -R destination_host -i_recover_option [directory_name]

-a used to perform the recovery in non-interactive mode. You cannot use the -a option with the -R option. You cannot use the -a option to restore 
 Windows SYSTEM or VSS save sets.

-s NetWorker_servername specifies the name of the source client’s NetWorker server.

-c source_host—Specifies the source host.

-S ssid/cloneid used to perform a save set recover.

-d destination_directory

- i_recover_option

• iN does not recover the file when a conflict occurs.
• iY overwrites the existing file when a conflict occurs.
• iR renames the file when a conflict occurs. The recover process appends a .R to each recovered file name.

◆ source_directory—Specifies the initial directory in which to begin browsing.

Note ; Also check below links

<https://community.emc.com/blogs/moinmansuri/2015/11/05/directed-saveset-recovery-command-line>

How to check Avamar node capacity- MSG_ERR_DISKFULL

Last GC: finished Wed Aug 25 01:01:04 2010 after 00m 50s >> recovered 0.00 KB (MSG_ERR_DISKFULL)

The total grid utilization is currently at 88% and I also saw some Unacknowledged Events with the following information:

Code: 4202 Message: failed garbage collection with error MSG_ERR_DISKFULL

This error is a direct result of the garbage collect run limit being reached or exceeded due to excessive checkpoint overhead. To verify and check all of the node capacities use the following commands; also if this is a single node you will not have to use the mapall command.

su – admin

ssh-agent bash

ssh-add ~admin/.ssh/admin_key


mapall –noerror ‘df -h’

This should give you the filesystem for each node including the sizes, used, and space available. Then run:

avmaint nodelist | grep percent-full

This output gives you a true reading of the OS capacity utilization.  On a grid where data nodes use a file pool, the Linux "df" command is not meaningful because the stripes are pre-allocated in the file pool, and many of the stripes might not be in use.

Data Domain Data Invulnerability Architecture Lightboard Video

Deduplication with Data Domain Lightboard Video

Saveset Recover to restore NDMP data with following command

Saveset Recover to restore NDMP data with following command

  nsrndmp_recover  -s <networker_server_name> -c <client_name>  -S <ssid> -v off  -m <hostname::/path_to_recover_to> <path_to be_recovered>

  Where:     -s <networker_server_name> is the hostname of the NetWorker server,

                   -c <client_name> is the client hostname (NDMP server: filer/datamover ) that this data originated from

                 -m option uses for new destination either on the same NDMP server but different directory, or the different NDMP server

                    <hostname::/path_to_recover_to> hostname of the NDMP server (filer/datamover) you are recovering to followed by 2 colons and

                               the full path starting with the mountpoint down to the directory you would like to recover this data to

                   The directory in path-to-recover-to MUST be existence on the host.

                 <path_to_be_recovered> is the full path name starting at the mountpoint down to the directory you would like to restore.

−c client Client is the name of the machine that saved the files. 

−F This option specifies that a file level recovery is going to be performed. This option should only be specified by nwrecover(8) or recover(8). 

−m mount point The mount point of the raw device specified by the -r option. The filesystem will be unmounted for the recover operation and mounted after the operation is complete. 

−r raw device Specifies the pathname of the raw device the data is to be recovered to. This option is mandatory for destructive recovers. 

−s server Selects which NetWorker server to use. 

−S ssid This mandatory option is used to specify save set recover mode. This mode can be used to implement fast batch file recovery without requiring the NetWorker file index entries. Ssid specifies the save set id for the save set to be recovered. NDMP generated save sets cannot be cloned, so no cloneid applies as it does for the standard recover(8) command.