NFS slowness

[Roshan]

-bash-4.2$ cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)

Hello Erman,

Striim reads files from an NFS drive. Recently I noticed the read is very slow (2-3 msgs/s). I have checked the read rate of the NFS.

-bash-4.2$ time dd if=/data2/CDR/inocs/sms/cbs_cdr_sms_20220112_601_101_614287.add of=/dev/null ibs=30000 obs=4096 count=3333
2+1 records in
21+1 records out
89820 bytes (90 kB) copied, 0.00761152 s, 11.8 MB/s

real    0m0.013s
user    0m0.002s
sys     0m0.000s


Could you please advise why Striim is reading so slow?

Regards,

Roshan

[Roshan]

10.215.228.72:/export/cdr2/data /datacdr nfs rw,nfsvers=3,soft,nosuid,rsize=65536,wsize=65536,noatime 0 0

-bash-4.2$ nfsstat
Server rpc stats:
calls      badcalls   badclnt    badauth    xdrcall
0          0          0          0          0

Client rpc stats:
calls      retrans    authrefrsh
2926655732   0          2927244913

Client nfs v3:
null         getattr      setattr      lookup       access       readlink
0         0% 1439971887 49% 1307220   0% 912143462 31% 3890458   0% 0         0%
read         write        create       mkdir        symlink      mknod
4869203   0% 1270802   0% 1207217   0% 6         0% 0         0% 0         0%
remove       rmdir        rename       link         readdir      readdirplus
0         0% 0         0% 0         0% 0         0% 98063821  3% 464608578 15%
fsstat       fsinfo       pathconf     commit
101       0% 14        0% 7         0% 0         0%

[Roshan]

I noticed when I decreased rsize from 65536 to 32768 I see a small improvement in read speed.

10.215.228.72:/export/cdr2/data /datacdr nfs rw,nfsvers=3,soft,nosuid,rsize=65536,wsize=65536,noatime 0 0

How should I know which value to set for the above parameters?

[ErmanArslansOracleBlog]

We don't have that specific info for Striim, but you may review the Goldengate NFS best practices for NFS..

Oracle GoldenGate Best Practice: NFS Mount options for use with GoldenGate (Doc ID 1232303.1)

32768 seems okay.
You may review the underlying the config of underlying storage/or server that hosts that nfs..

[Roshan]

Thanks for sharing the link. When I start loading I notice the performance is good for first 3 hours then it starts decreasing. Then after restarting pipeline I see speed much higher. What could be the reason? Is it the network link becomes saturated?

[ErmanArslansOracleBlog]

If you too see that slowness on NFS, then your saturation theory may be true.. in that case your network admin can check that..
However, if you dont see a slowness on NFS during that problemtic time interval, then probably your application is the cause.. maybe its heaps are getting crowded, maybe it slows down because of an unsupported object or type that it needs to process during that interval..

So you should analyze this problematic case with these in mind..

[Roshan]

I noticed that only during public holidays and Sundays the transfer speed is quite high and stable.

Most probably it is network saturation. I checked with network team and they told me we using 40GbE bandwith.

Do you think we need to increase to 100GbE? What are some tests we need to do?

[ErmanArslansOracleBlog]

Network admin will tell you that. Network admin can analyze that bandwith and see whether there is a saturation appearing  or not.
40gbit is quite good actually, but we dont know your workload..
There may be other servers using the same target ports and channels or paths, doing stuff like backups, bulk copying and etc.. so this may be the case. Please tell network admin to analyze those channels..