Parse server performance loss after time

Hey manuel. I made the test. Performance still decreases. I waited one minute between tests.

Here is test history. It started 318 request/second. And decreased to 103. And It was still decreasing but I stopped the test.

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   921.29ms  188.11ms   1.87s    85.78%
    Req/Sec   334.15    121.93   696.00     67.48%
  9536 requests in 30.05s, 179.25MB read
Requests/sec:    317.34
Transfer/sec:      5.97MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   908.23ms  154.12ms   1.46s    77.69%
    Req/Sec   336.41    134.10     0.87k    72.73%
  9606 requests in 30.06s, 180.56MB read
Requests/sec:    319.53
Transfer/sec:      6.01MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   918.89ms  155.10ms   1.75s    75.65%
    Req/Sec   331.93    134.94   848.00     70.83%
  9532 requests in 30.06s, 179.17MB read
Requests/sec:    317.11
Transfer/sec:      5.96MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   928.40ms  151.50ms   1.58s    75.08%
    Req/Sec   328.55    115.63   800.00     69.90%
  9472 requests in 30.05s, 178.04MB read
Requests/sec:    315.17
Transfer/sec:      5.92MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   941.49ms  146.27ms   1.51s    74.10%
    Req/Sec   323.12    111.22   828.00     73.61%
  9292 requests in 30.04s, 174.66MB read
Requests/sec:    309.27
Transfer/sec:      5.81MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   947.33ms  143.14ms   1.52s    72.83%
    Req/Sec   321.92     93.10   630.00     72.22%
  9247 requests in 30.06s, 173.82MB read
Requests/sec:    307.66
Transfer/sec:      5.78MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   968.04ms  160.94ms   1.96s    74.12%
    Req/Sec   315.72    100.05   770.00     71.33%
  9002 requests in 30.05s, 169.21MB read
  Socket errors: connect 0, read 0, write 0, timeout 3
Requests/sec:    299.56
Transfer/sec:      5.63MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   983.28ms  149.79ms   1.62s    69.66%
    Req/Sec   308.81    105.17   787.00     71.63%
  8906 requests in 30.04s, 167.41MB read
Requests/sec:    296.46
Transfer/sec:      5.57MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.01s   156.19ms   1.87s    65.25%
    Req/Sec   299.80    106.03   707.00     71.97%
  8639 requests in 30.05s, 162.39MB read
Requests/sec:    287.46
Transfer/sec:      5.40MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.02s   167.85ms   1.76s    66.68%
    Req/Sec   297.29    113.59   750.00     70.73%
  8520 requests in 30.05s, 160.15MB read
Requests/sec:    283.51
Transfer/sec:      5.33MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.06s   173.42ms   1.99s    65.74%
    Req/Sec   287.77    103.26     0.88k    72.38%
  8230 requests in 30.05s, 154.70MB read
  Socket errors: connect 0, read 0, write 0, timeout 5
Requests/sec:    273.86
Transfer/sec:      5.15MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.06s   175.03ms   1.94s    65.03%
    Req/Sec   287.26    102.01   700.00     68.64%
  8221 requests in 30.07s, 154.53MB read
Requests/sec:    273.43
Transfer/sec:      5.14MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.12s   190.55ms   1.99s    68.82%
    Req/Sec   271.80    102.13   828.00     76.57%
  7764 requests in 30.06s, 145.95MB read
  Socket errors: connect 0, read 0, write 0, timeout 10
Requests/sec:    258.33
Transfer/sec:      4.86MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.14s   230.84ms   2.00s    66.12%
    Req/Sec   266.72     86.49   700.00     76.66%
  7637 requests in 30.06s, 143.55MB read
  Socket errors: connect 0, read 0, write 0, timeout 7
Requests/sec:    254.06
Transfer/sec:      4.78MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.18s   245.31ms   2.00s    67.80%
    Req/Sec   255.79     86.33   545.00     70.77%
  7250 requests in 30.07s, 136.28MB read
  Socket errors: connect 0, read 0, write 0, timeout 98
Requests/sec:    241.14
Transfer/sec:      4.53MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.20s   251.28ms   2.00s    68.92%
    Req/Sec   250.43     87.33   700.00     69.79%
  7206 requests in 30.06s, 135.45MB read
  Socket errors: connect 0, read 0, write 0, timeout 114
Requests/sec:    239.72
Transfer/sec:      4.51MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.22s   234.71ms   2.00s    70.41%
    Req/Sec   247.76     98.26   818.00     77.70%
  7095 requests in 30.06s, 133.36MB read
  Socket errors: connect 0, read 0, write 0, timeout 78
Requests/sec:    236.07
Transfer/sec:      4.44MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.30s   264.83ms   2.00s    71.78%
    Req/Sec   219.64     77.18   480.00     65.62%
  6308 requests in 30.05s, 118.57MB read
  Socket errors: connect 0, read 0, write 0, timeout 433
Requests/sec:    209.92
Transfer/sec:      3.95MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.28s   273.28ms   2.00s    68.97%
    Req/Sec   224.68     86.50   610.00     72.47%
  6430 requests in 30.05s, 120.86MB read
  Socket errors: connect 0, read 0, write 0, timeout 423
Requests/sec:    213.98
Transfer/sec:      4.02MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.28s   297.44ms   2.00s    71.19%
    Req/Sec   216.58     76.46   505.00     70.49%
  6223 requests in 30.07s, 116.97MB read
  Socket errors: connect 0, read 0, write 0, timeout 725
Requests/sec:    206.95
Transfer/sec:      3.89MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.28s   268.14ms   2.00s    71.59%
    Req/Sec   212.54     70.55   690.00     76.12%
  6133 requests in 30.04s, 115.28MB read
  Socket errors: connect 0, read 0, write 0, timeout 769
Requests/sec:    204.13
Transfer/sec:      3.84MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.28s   248.52ms   2.00s    75.28%
    Req/Sec   205.85     73.77   770.00     77.08%
  5924 requests in 30.06s, 111.35MB read
  Socket errors: connect 0, read 0, write 0, timeout 875
Requests/sec:    197.08
Transfer/sec:      3.70MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.31s   264.71ms   2.00s    72.88%
    Req/Sec   201.56     70.79   550.00     74.56%
  5790 requests in 30.04s, 108.83MB read
  Socket errors: connect 0, read 0, write 0, timeout 872
Requests/sec:    192.73
Transfer/sec:      3.62MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.35s   281.51ms   2.00s    73.23%
    Req/Sec   175.89     61.97   646.00     79.51%
  5059 requests in 30.05s, 95.09MB read
  Socket errors: connect 0, read 0, write 0, timeout 1380
Requests/sec:    168.35
Transfer/sec:      3.16MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.36s   261.39ms   2.00s    70.81%
    Req/Sec   184.69     64.38   440.00     69.93%
  5271 requests in 30.05s, 99.08MB read
  Socket errors: connect 0, read 0, write 0, timeout 1245
Requests/sec:    175.38
Transfer/sec:      3.30MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.37s   296.79ms   2.00s    73.31%
    Req/Sec   169.93     64.26   626.00     78.32%
  4853 requests in 30.04s, 91.22MB read
  Socket errors: connect 0, read 0, write 0, timeout 1661
Requests/sec:    161.55
Transfer/sec:      3.04MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.34s   247.45ms   2.00s    74.87%
    Req/Sec   163.31     54.27   353.00     70.07%
  4626 requests in 30.04s, 86.95MB read
  Socket errors: connect 0, read 0, write 0, timeout 1514
Requests/sec:    154.01
Transfer/sec:      2.89MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.38s   290.60ms   2.00s    71.98%
    Req/Sec   158.62     68.69   520.00     79.72%
  4529 requests in 30.04s, 85.13MB read
  Socket errors: connect 0, read 0, write 0, timeout 1595
Requests/sec:    150.75
Transfer/sec:      2.83MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.37s   275.42ms   2.00s    75.76%
    Req/Sec   155.28     51.62   363.00     70.24%
  4484 requests in 30.05s, 84.29MB read
  Socket errors: connect 0, read 0, write 0, timeout 1893
Requests/sec:    149.20
Transfer/sec:      2.80MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.40s   278.83ms   2.00s    71.16%
    Req/Sec   154.91     51.04   360.00     73.43%
  4423 requests in 30.05s, 83.14MB read
  Socket errors: connect 0, read 0, write 0, timeout 1500
Requests/sec:    147.17
Transfer/sec:      2.77MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.38s   276.97ms   2.00s    76.20%
    Req/Sec   151.41     53.37   454.00     79.24%
  4378 requests in 30.05s, 82.29MB read
  Socket errors: connect 0, read 0, write 0, timeout 1954
Requests/sec:    145.68
Transfer/sec:      2.74MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.36s   246.76ms   2.00s    81.20%
    Req/Sec   147.93     58.99   480.00     82.29%
  4251 requests in 30.05s, 79.91MB read
  Socket errors: connect 0, read 0, write 0, timeout 2044
Requests/sec:    141.45
Transfer/sec:      2.66MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.35s   220.06ms   2.00s    80.37%
    Req/Sec   142.69     52.70   393.00     78.05%
  4101 requests in 30.03s, 77.10MB read
  Socket errors: connect 0, read 0, write 0, timeout 1854
Requests/sec:    136.56
Transfer/sec:      2.57MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.41s   244.83ms   2.00s    75.15%
    Req/Sec   139.88     60.57   545.00     81.34%
  3996 requests in 30.06s, 75.11MB read
  Socket errors: connect 0, read 0, write 0, timeout 1763
Requests/sec:    132.93
Transfer/sec:      2.50MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.36s   224.82ms   2.00s    81.04%
    Req/Sec   140.15     54.55   434.00     76.84%
  4011 requests in 30.03s, 75.39MB read
  Socket errors: connect 0, read 0, write 0, timeout 1912
Requests/sec:    133.55
Transfer/sec:      2.51MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.39s   197.84ms   2.00s    74.59%
    Req/Sec   141.83     71.15   545.00     78.87%
  4031 requests in 30.06s, 75.79MB read
  Socket errors: connect 0, read 0, write 0, timeout 1831
Requests/sec:    134.11
Transfer/sec:      2.52MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.38s   200.00ms   2.00s    77.80%
    Req/Sec   131.08     46.56   360.00     76.47%
  3787 requests in 30.05s, 71.20MB read
  Socket errors: connect 0, read 0, write 0, timeout 1895
Requests/sec:    126.03
Transfer/sec:      2.37MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.41s   185.00ms   2.00s    73.20%
    Req/Sec   133.85     62.00   474.00     83.69%
  3768 requests in 30.06s, 70.83MB read
  Socket errors: connect 0, read 0, write 0, timeout 1820
Requests/sec:    125.37
Transfer/sec:      2.36MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.43s   217.95ms   2.00s    72.34%
    Req/Sec   132.23     61.63   520.00     76.76%
  3751 requests in 30.05s, 70.51MB read
  Socket errors: connect 0, read 0, write 0, timeout 1806
Requests/sec:    124.84
Transfer/sec:      2.35MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.44s   208.47ms   1.99s    70.79%
    Req/Sec   127.17     53.19   363.00     73.78%
  3639 requests in 30.06s, 68.40MB read
  Socket errors: connect 0, read 0, write 0, timeout 1934
Requests/sec:    121.06
Transfer/sec:      2.28MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.46s   216.67ms   2.00s    68.67%
    Req/Sec   131.14     58.20   380.00     74.47%
  3701 requests in 30.05s, 69.57MB read
  Socket errors: connect 0, read 0, write 0, timeout 1818
Requests/sec:    123.16
Transfer/sec:      2.32MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.46s   191.29ms   2.00s    70.83%
    Req/Sec   122.80     47.96   313.00     75.61%
  3518 requests in 30.05s, 66.13MB read
  Socket errors: connect 0, read 0, write 0, timeout 1958
Requests/sec:    117.07
Transfer/sec:      2.20MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.47s   181.54ms   2.00s    73.70%
    Req/Sec   123.67     63.33   410.00     73.31%
  3473 requests in 30.07s, 65.28MB read
  Socket errors: connect 0, read 0, write 0, timeout 1701
Requests/sec:    115.48
Transfer/sec:      2.17MB

-----------------------------------------------------------------------

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.52s   180.95ms   2.00s    67.52%
    Req/Sec   111.96     51.35   330.00     75.99%
  3120 requests in 30.05s, 58.65MB read
  Socket errors: connect 0, read 0, write 0, timeout 1950
Requests/sec:    103.81
Transfer/sec:      1.95MB

Edit: I restarted all parse servers. And Result increased to 317 again:

  1 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   918.65ms  154.74ms   1.66s    84.86%
    Req/Sec   331.27    100.33   580.00     67.71%
  9531 requests in 30.05s, 179.15MB read
Requests/sec:    317.17
Transfer/sec:      5.96MB