Monday, November 11, 2013

Calculating and Configuring EMC Networker port range in Firewalls

NetWorker host — A NetWorker server, storage node, or client.
 
Service port — A port on which a server process listens for requests to provide a service.
Service port = target port = destination port = listen port = inbound port
 
Connection port — A port used by a process to make requests.
Connection port = source port = outbound port
NetWorker connection flow:
 
Default NetWorker configuration results in the following connection flow for scheduled backups:
 
server:conn->client:7938 (nsrrpc)
server:conn->client:rpc/390113 (nsrexecd/7937)
server:conn->client:svc (save)
client:conn->server:7938 (nsrrpc)
client:conn->server:rpc/390119 (nsrexecd/7937)
client:conn->server:rpc/390436 (nsrauth/svc)
client:conn->server:rpc/390103 (nsrd/svc)
client:conn->server:rpc/390104 (nsrmmd/svc)
client:conn->server:rpc/390105 (nsrindexd/svc)
 
 
This shows a file-system backup with target device residing directly on the backup server.
 
Calculating and configuring port ranges
 
Reserved ports:
 
Two of the TCP ports, 7937 and 7938, are reserved by the NetWorker host
 
Port 7937 as a service port for the nsrexecd daemon.
Port 7938 as a service port for the EMC® NetWorker portmapper.
 
In addition, port 514 is used as a fallback connection if communication with nsrexecd cannot be established. To avoid potentially slow performance with the connection, ensure that port 514 is not blocked.
 
Calculating connection port ranges
 
 
In NetWorker 7.2 and earlier, the default value for this range is: 10001-30000. In NetWorker 7.3 and higher, the default value is 0-0. The 0-0 value has a special meaning: NetWorker allows the OS to select the port for TCP clients. Entering 0-0 is only allowed for NetWorker 7.3 and later.
 
From a NetWorker perspective, one connection port is required for any type of communication between the client, storage node and server. However, calculating the minimum required connection port range does not rely only on NetWorker operations because such ports are reserved for short-term re-use by the operating system. So depending on the specific operating system and operating system configuration, the number of required connection ports is always higher than highest number of parallel connections. It is best to keep the connection port range as wide as possible as there is no security concern. However, if the range is too narrow, then one may see performance problems, or random malfunctions of the NetWorker product.
 
Calculating service port ranges:
 
When NetWorker services start, they attempt to listen only in the service port range that is specified for that host. NetWorker processes attempt to connect to a service by using connection (or source) ports from the connection port range:
 
Service port ranges correspond to TCP listen ports
Connection port ranges correspond to TCP source ports
 
The NetWorker services and processes running on NetWorker servers, clients, and storage nodes listen and connect only on the specified port ranges. The minimum number of ports depends on the NetWorker configuration.
 
NetWorker client:
 
A NetWorker 7.3 or later client uses nsrexecd that requires four service ports: the reserved ports 7937 and 7938 and two user-configurable ports from the service port range.
 
As a result, a client requires a minimum of four service ports.
Note :  If the client uses NetWorker add-on products, additional ports may be required.
 
NetWorker storage node:
 
A NetWorker storage node (SN) is also a NetWorker client, and so it uses all of the ports for a client.
 
In addition to the four ports for a client, a storage node requires ports for nsrmmd and nsrlcpd daemons.  There is one nsrlcpd per robot in an autochanger.
 
 
As a result, a storage node requires a minimum of: 4 + (2 * #devices) + (#jukeboxes) service ports.
4  (client ports)+ (2 nsrmmd * #devices) + (nsrlcpd #jukeboxes) service ports
 
1 nsrmmd : There is one nsrmmd per tape or file device
2nd nsrmmd : When spanning from one device to another, a helper nsrmmd is launched to mount the new tape. Helper nsrmmd also require a port. There can be up to two mmd per device on a system.
 
NetWorker server:
 
A NetWorker server is also a NetWorker storage node, and so it uses all of the ports for a storage node.
 
In addition to the ports for a storage node, a server requires ports for nsrd, nsmmdbd, nsrindexd, nsrmmgd, and nsrjobd daemons. Each of these requires a TCP/IP port.
 
The nsrd and nsrmmgd daemons also require a UDP port.
 
As a result, a NetWorker 7.3.x server requires a minimum of:
 
11 + (2 * #devices) + (#jukeboxes) service ports.
 
11{ (4 SN’s Port + nsrd, nsmmdbd, nsrindexd, nsrmmgd, and nsrjobd + nsrd UDP and nsrmmgd UDP)}+ (2 * #devices) + (#jukeboxes)
 
NetWorker 7.4 introduces a new daemon, the client push daemon, which also consumes a TCP service port. As a result, a NetWorker 7.4 server requires a minimum of:
 
12 + (2 * #devices) + (#jukeboxes) service ports
 
NetWorker Management Console:
 
The Console server component of NMC uses 3 ports:
 
§  One port (9000 by default) is used for the web server
§  The second port (9001 by default) is used for RPC calls from the Console Java client to the Console server.
§  The last port (2638 by default) is used for database queries.
 
The Console server communicates to the NetWorker server using service ports from the standard NetWorker range (as defined by nsrports).
 
Example: Calculating service ports on a bidirectional firewall.
 

NetWorker clients A, B, C
 
NetWorker storage nodes X and Y
 
NetWorker server Z,
 
Single firewall that blocks both ways. The firewall in this example sits between the NetWorker server on the one side, and the clients and storage nodes on the other. Each storage node and the NetWorker server have a tape library and six drives, and there are no pre-NW 7.3 clients.
 
192.167.10.101 client_A
192.167.10.102 client_B
192.167.10.103 client_C
# ...
196.167.10.124 storage_node_X
192.167.10.125 storage_node_Y
 
192.167.10.126 NW_server_Z
 
11 + 2 * (num devices) + (num libraries) = 24 service ports.
11+2*(6)+(1)=24
 
Two ports must be 7937 and 7938, for example, select ports 7937–7960.
A NetWorker 7.4 server would require one additional port to accommodate the client push daemon.
 
The NetWorker server must be configured to use 24 service ports, 7937–7960, and the firewall must allow traffic leftward (to the NetWorker server's IP address) on all the service ports configured.
 
TCP, Service, src 192.167.10.*, dest 192.167.10.126, ports 7937-7960, action accept
 
dest : Backup Server
 
Storage nodes:
 
There are NetWorker storage nodes on the right of the firewall. Storage node X has six devices and one library. So it needs 4 + 2 * (num devices 6) + (num libraries 1) = 17 service ports. Two ports must be 7937 and 7938,
So, for example, select ports 7937–7953. Thus, each NetWorker SN must be configured to use 17 service ports, 7939–7953,
 
The firewall only needs to allow 17 ports for both storage node IP addresses.
 
TCP, Service, src 192.167.10.*, dest 192.167.10.124, ports 7937-7953, action accept
TCP, Service, src 192.167.10.*, dest 192.167.10.125, ports 7937-7953, action accept
 
Dest: Storage Node
 
Clients:
 
Client A needs four service ports. Two ports must be 7937 and 7938, so, for example, select ports 7937–7940. Clients B and C have the same port requirements.
 
TCP, Service, src 192.167.10.*, dest 192.167.10.101, ports 7937-7940, action accept
TCP, Service, src 192.167.10.*, dest 192.167.10.102, ports 7937-7940, action accept
TCP, Service, src 192.167.10.*, dest 192.167.10.103, ports 7937-7940, action accept
 
In the previous example, the firewall is configured to allow incoming service connections to the NetWorker server’s IP address on ports 7937–7960, from the IP addresses of each of the storage nodes or client machines (as well as any other machines on that subnet). The firewall is also configured to allow connections to the IP addresses for each storage node on ports 7937–7953, and to each client IP address on ports 7937–7940. Each NetWorker host must be configured with the appropriate port range for that machine.
 
Note : The NetWorker services must be restarted on each machine after a change to the port range is.
 
A simpler configuration to administer these machines would be to assign a range of 24 ports, 7937–7960, to all machines, and configure the firewall to allow traffic to these ports on any host, from any host.
 
TCP, Service, src 192.167.10.*, dest 192.167.10.*, ports 7937-7960, action accept
 
Example B: Calculating service ports on a unidirectional firewall:
 
The difference in this example is that one NetWorker storage node is on either side of the firewall. NetWorker clients on the left side of the firewall back up data to the storage node on the left, and clients on the right side back up data to the storage node on the right. The clients on the right side of the firewall are in a demilitarized zone (DMZ). Everything to the left of the firewall is protected and trusted. Everything to the right of the firewall is not protected and cannot be trusted. Therefore, the firewall must block network traffic from right to left.
 

192.167.10.104 client_D
192.167.10.105 client_E
192.167.10.106 client_F
 
196.167.10.124 storage_node_X
192.167.10.125 storage_node_Y
192.167.10.126 NW_server_Z
 
Clients:
TCP, Service, src 192.167.10.104, dest 192.167.10.126, ports 7937-7960, action accept
TCP, Service, src 192.167.10.105, dest 192.167.10.126, ports 7937-7960, action accept
TCP, Service, src 192.167.10.106, dest 192.167.10.126, ports 7937-7960, action accept
SN for Y:
TCP, Service, src 192.167.10.125, dest 192.167.10.126, ports 7937-7960, action accept
 
Configuring RPC:
NetWorker requires a fully functional RPC portmapper service (otherwise known as rpcbind) to discover available program services and their current connection points. NetWorker can utilize either the default operating system SunRPC portmapper on port 111 (if present) or internal NsrRPC portmapper available inside the nsrexecd process (by default on port 7938).
If not explicitly specified, the order of initial RPC connections (SunRPC or NsrRPC) is decided by operating system.
 
Note that SunRPC portmapper is not required for NetWorker operations as full functionality is provided by NsrRPC, but if SunRPC is actively blocked by a firewall rule, it can cause delays on client/server connectivity as NetWorker has to wait for operating system timeout before attempting connection to NsrRPC.
 

Default location of services file is:
 
 On Unix/Linux: /etc/services
 On Windows: %SYSTEMROOT%\System32\Drivers\etc\services
 
Example services file:
sunrpc 111/tcp rpcbind portmap #Sun RPC
sunrpc 111/udp rpcbind portmap #Sun RPC
nsrrpc 7938/tcp lgtomapper #EMC NetWorker RPC
nsrrpc 7938/udp lgtomapper #EMC NetWorker RPC
 
Diagnostic tips :
 
Before configuring NetWorker port ranges, consider the following:
 
·         Allocate some extra service ports to accommodate growth. If a new drive is added to a storage node, will the people adding the drive remember to increase the port count by two in nsrports and the firewall?
 
·         The nsrexecd daemon manages the NSR ports ranges resource. This daemon must be the first NetWorker daemon to start, as it does during system initialization. If the NetWorker software is manually started, be sure that the nsrexecd daemon is the first one started. If the nsrexecd daemon is not started first, ports may be assigned randomly.
 
·         After changing the service or connection port ranges, restart the NetWorker software, including nsrexecd, and make any corresponding modifications to the firewall rules.
 
·         Use the netstat -a command to determine port allocation.
 
·         The rpcinfo -p or ping commands may not always work across firewalls. RPC info requires connectivity using SunRPC on port 111, which is not required by NetWorker, while ping requires ICMP packets which may be blocked separately from TCP packets used by NetWorker.
 
·         Use the nsradmin command to carry out limited testing of the client/server connectivity through firewall:
 
·         To test the NetWorker server connection to the nsrexecd daemon running on the client, run the following command from the NetWorker server:
           nsradmin –s <client_name> -p 390113
 
·         To test the NetWorker client connections to the nsrd and nsrexecd daemons on the backup server, run the following command from the NetWorker client:
nsradmin –s <server_name>,
nsradmin –s <server_name> -p 390113
 
·         Maintain the connection port range for a NetWorker server, client, or storage node at the default range. In older versions of NetWorker, the default range was 10,001–30,000. NetWorker 7.3 or later can use a special range of 0–0 that lets the operating system pick the ports. These ports are used as connection ports only, and never as service ports.
 
·         Define port ranges with the nsrports program, or some other technique from the EMC NetWorker Multiplatform Version Administration Guide. Do not modify the nsr/res/nsrla.res file directly.
 
·         Do not assign ports from the reserved service port range (ports below 1024) in order to avoid conflict with other daemons or services on the host. Additionally, always place the starting point of the connection port range (if manual configuration is necessary) so that it starts after the range used by service ports for NetWorker or any other application.
 
Check the nsrexecd ad other services running on client as well as backup server:
 
# rpcinfo -p <client_name>
program          vers     proto   port
390113 1         tcp 7    937      nsrexecd
  
However, on a backup server:
 
# rpcinfo -p <server_name>
program          vers     proto   port
390103 2         tcp       8192    nsrd
390104 205     tcp       9847    nsrmmd
390105 5         tcp       9318    nsrindexd
390107 5         tcp       9882    nsrmmdbd
390109 2         tcp       8192    nsrstat
390110 1         tcp       8192    nsrjbd
390113 1         tcp       7937    nsrexecd
390115 1         tcp       9705    lgtolmd
390120 1         tcp       8192    nsrexecd
390402 1         tcp       9001    gstd
390433 1         tcp       9349    nsrjobd
390435 1         tcp       8070    nsrexecd
390436 1         tcp       8152    nsrd
390109 2         udp     9168    nsrstat

2 comments:

  1. Very valuable informationon port range configuration. I was searching for this info for sometime, Thanks for the guidance Moin. Please do keep on sharing more.

    ReplyDelete
  2. One of the Best blog for EMC netwoker. Keep up the good job bro.

    ReplyDelete

Slow backup Issue

Networker slow backup issue We are facing slow backup issue many times or previously added clients suddenly slow backup issue occurs. ...