Latency: That pesky little thing

Recently, I was helping out a client improve their performance of their ETL processes. They were loading several million rows and updating several million more and the whole process was taking better than 7 hours. Jumping in, it was apparent they didn’t spare any expense on the hardware. With 256GB of RAM and a high end SAN their server had plenty of horsepower, but they couldn’t push more than 1000 rows/second. I then took a close look at their config file and there were some changes that should be made, but they didn’t have anything that was unreasonable. I went ahead made some changes to the config that should affect the workload they are pushing on their server like shared_buffers and checkpoint_segments, but it had no effect at all. Watching the server while the load was running, I was surprised to see, the server was mostly idle. There was a little disk activity and a bit of CPU usage, but nothing that would indicate a problem. I went back and looked at the server loading the data and saw similar results, a little activity, but nothing that would pinpoint the issue. That’s when I figured I’d ping the PostgreSQL server from the ETL server and there it was, 1ms latency. That was something I would expect between availability zones on Amazon, but not in a enterprise data center. It turned out that they had a firewall protecting their database servers and the latency was adding 4-5 hours to their load times. The small amount of 1ms really adds up when you need to do millions of network round trips.

 

  1. BobBob08-29-2013

    Was removing the millions of network trips an option?

    • Jim MlodgenskiJim Mlodgenski08-29-2013

      It’s wasn’t an option in the short term. The longer term goal is to re-architect things to eliminate many of the round trips, but that needs to wait for a less busy time of the year.

  2. cialis_onlinecialis_online06-20-2014

    Hello!

  3. onlineonline06-25-2014

    Hello!

  4. cialiscialis06-27-2014

    Hello!

  5. buy_cialisbuy_cialis06-29-2014

    Hello!

  6. cheap_viagracheap_viagra06-30-2014

    Hello!

  7. toto07-02-2014

    Hello!

  8. viagraviagra07-02-2014

    Hello!

  9. generic_viagrageneric_viagra07-05-2014

    Hello!

  10. web siteweb site08-28-2014

    Magnificent website. A lot of useful information here.
    I’m sending it to several paals ans also sharing in delicious.

    And obviously, thank you in your sweat!

  1. xxl08-30-13
  2. Lennox furnace filters09-18-13
  3. ubergamez.com09-19-13
  4. Branson Missouri Private Investigator09-19-13
  5. tex4tex09-19-13
  6. generale state09-20-13
  7. walmart voucher09-21-13
  8. Breakfree herpes relief reviews09-21-13
  9. hemoroizi externi09-21-13
  10. ryan gosling surgery09-21-13
  11. smokeless cigarettes free trial09-21-13
  12. diet pills09-22-13
  13. lavoro da casa online gratis09-22-13
  14. http://jxxct.com09-23-13
  15. official site09-23-13
  16. website09-23-13
  17. Craigslist Search Engine09-23-13
  18. Durham Electrician09-24-13
  19. Charleston South Carolina09-24-13
  20. visit this website09-24-13
  21. Search Rex Craigslist Search09-24-13
  22. navigate to this web-site09-24-13
  23. money maker09-24-13
  24. compare phentermine09-25-13
  25. no fax payday loans09-25-13
  26. more info here09-25-13
  27. small business ideas09-25-13
  28. create an android app09-25-13
  29. Carpet Pet Urine Removal09-25-13
  30. work at home moms09-26-13
  31. come fare dei soldi falsi09-26-13
  32. floor and decor atlanta09-27-13
  33. more info09-27-13
  34. raspberry ketone pure09-27-13
  35. cardiologist salary in los angeles09-27-13
  36. visitare questo sito09-27-13
  37. water damage repair atlanta09-27-13
  38. los angeles cardiology09-27-13
  39. gry hazardowe online book of ra09-28-13
  40. e cig free trial09-28-13
  41. garcinia cambogia reviews09-28-13
  42. e cigarette starter kit09-28-13
  43. water ionizers09-29-13
  44. water ionizer09-29-13
  45. prix de l'immobilier09-29-13
  46. europa casino ohne einzahlung09-30-13
  47. tabela fipe carros09-30-13
  48. charlize theron sexiest09-30-13
  49. saffron extract reviews09-30-13
  50. internet marketer09-30-13
  51. jadwal premier league10-02-13
  52. view from a double decker bus10-02-13
  53. inscrições bbb 2014 globo10-02-13
  54. popart on canvas10-03-13
  55. how to make an app10-03-13
  56. visit this link for auto insurance10-05-13
  57. http://ZaufanaStrona.org10-05-13
  58. check these guys out10-06-13
  59. great post to read10-07-13
  60. find more info10-07-13
  61. facebook advertising software for mac10-08-13
  62. blu cigs e-cigarette starter kit10-08-13
  63. giochi machine slot gratis 3D10-08-13
  64. meratol reviews10-09-13
  65. rencontre gratuit10-10-13
  66. how to last longer in bed10-12-13
  67. e-cigs10-14-13
  68. e cig10-14-13
  69. ekonomia10-15-13

Leave a Reply

Please type the characters of this captcha image in the input box

Please write the answer to the math question of this captcha image in the input box

/* ]]> */