You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-02 - 23:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Dec 02, 2023 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
186222630,0sleep20-21swapper/212:09:432
679999341,6cyclictest224-21systemd-journal10:02:102
656123428,2sleep10-21swapper/107:08:031
462423428,2sleep30-21swapper/307:05:203
679999316,2cyclictest0-21swapper/211:20:242
679999272,10cyclictest500-21in:imuxsock10:43:582
679999263,22cyclictest534-21snmpd10:37:252
679999255,9cyclictest502-21rs:main2
679999252,3cyclictest32055-21diskmemload09:33:072
679999248,1cyclictest15935-21ls10:35:012
679199240,22cyclictest0-21swapper/112:03:501
678799240,20cyclictest224-21systemd-journal07:35:250
6805992318,2cyclictest224-21systemd-journal12:25:383
679999233,7cyclictest500-21in:imuxsock11:05:382
6799992318,5cyclictest0-21swapper/212:30:152
679199232,6cyclictest0-21swapper/111:12:151
6791992315,4cyclictest500-21in:imuxsock12:07:551
678799231,1cyclictest558-21lldpd12:15:520
680599226,7cyclictest224-21systemd-journal09:20:143
6805992219,2cyclictest39-21ksoftirqd/310:39:233
680599220,4cyclictest0-21swapper/311:42:203
679999222,9cyclictest0-21swapper/210:26:042
679999221,1cyclictest534-21snmpd10:55:272
679999221,1cyclictest534-21snmpd10:55:272
679999220,21cyclictest0-21swapper/212:04:332
6791992220,1cyclictest9844-21ssh10:30:291
6791992216,3cyclictest534-21snmpd10:05:501
6787992219,2cyclictest534-21snmpd10:25:050
6805992116,1cyclictest224-21systemd-journal10:05:263
680599211,14cyclictest224-21systemd-journal11:38:073
680599210,2cyclictest0-21swapper/311:15:553
679999210,2cyclictest0-21swapper/210:49:122
679999210,12cyclictest22275-21taskset08:47:002
679999210,10cyclictest224-21systemd-journal12:17:542
6791992115,2cyclictest224-21systemd-journal08:44:551
680599208,6cyclictest224-21systemd-journal11:03:133
680599203,14cyclictest224-21systemd-journal12:18:313
680599200,12cyclictest224-21systemd-journal10:50:423
6799992014,5cyclictest558-21lldpd12:26:012
6799992012,4cyclictest224-21systemd-journal11:33:072
6799992012,2cyclictest500-21in:imuxsock11:10:312
679199203,11cyclictest224-21systemd-journal09:10:461
6791992018,1cyclictest0-21swapper/111:30:231
679199200,7cyclictest32055-21diskmemload11:25:001
678799201,18cyclictest0-21swapper/012:26:140
680599197,3cyclictest224-21systemd-journal10:23:153
680599196,7cyclictest500-21in:imuxsock09:10:073
680599195,7cyclictest224-21systemd-journal12:31:543
6805991917,1cyclictest558-21lldpd10:10:233
679999194,4cyclictest30581-21apache_processe12:20:152
679999193,2cyclictest0-21swapper/209:57:002
679999191,5cyclictest224-21systemd-journal09:50:332
678799198,7cyclictest224-21systemd-journal11:49:320
678799193,14cyclictest500-21in:imuxsock07:10:310
6787991917,1cyclictest558-21lldpd10:40:180
678799191,16cyclictest0-21swapper/011:08:590
680599187,6cyclictest224-21systemd-journal11:58:523
680599187,5cyclictest224-21systemd-journal11:51:023
680599185,8cyclictest224-21systemd-journal11:05:493
680599182,6cyclictest224-21systemd-journal09:40:573
6805991817,1cyclictest0-21swapper/310:17:163
6805991814,1cyclictest224-21systemd-journal08:15:283
679999186,11cyclictest500-21in:imuxsock10:16:172
6799991812,3cyclictest534-21snmpd08:25:012
6791991815,2cyclictest534-21snmpd11:51:321
6791991815,2cyclictest500-21in:imuxsock10:30:001
679199181,10cyclictest16599-21cat07:30:001
679199180,7cyclictest32055-21diskmemload11:15:261
679199180,16cyclictest0-21swapper/108:58:271
678799187,4cyclictest224-21systemd-journal11:12:170
678799185,10cyclictest224-21systemd-journal09:17:430
678799182,14cyclictest32055-21diskmemload11:50:270
6787991812,3cyclictest22795-21apache_processe08:50:210
6787991812,2cyclictest224-21systemd-journal11:19:550
678799181,1cyclictest558-21lldpd11:30:230
284822180,2sleep2558-21lldpd10:20:382
680599178,5cyclictest224-21systemd-journal11:10:293
6805991710,1cyclictest224-21systemd-journal07:29:243
680599170,7cyclictest224-21systemd-journal08:40:293
679999179,2cyclictest15604-21rm11:41:352
679999178,2cyclictest224-21systemd-journal11:00:302
679999172,9cyclictest558-21lldpd08:05:222
6799991716,1cyclictest558-21lldpd09:40:182
679999171,1cyclictest558-21lldpd11:49:582
679999170,11cyclictest0-21swapper/211:15:212
6791991716,1cyclictest29679-21ssh09:31:481
679199171,5cyclictest558-21lldpd12:38:261
679199171,16cyclictest32055-21diskmemload10:49:081
6787991715,1cyclictest534-21snmpd12:35:120
678799171,12cyclictest224-21systemd-journal11:24:390
680599169,3cyclictest558-21lldpd07:20:253
680599169,3cyclictest224-21systemd-journal09:15:203
680599166,4cyclictest500-21in:imuxsock12:20:173
680599165,6cyclictest224-21systemd-journal10:43:583
680599163,3cyclictest224-21systemd-journal10:46:123
680599162,5cyclictest0-21swapper/308:55:223
680599162,10cyclictest500-21in:imuxsock09:57:233
680599162,10cyclictest224-21systemd-journal07:40:303
680599161,7cyclictest224-21systemd-journal11:23:363
6805991613,2cyclictest558-21lldpd09:38:403
6805991612,3cyclictest15134-21taskset10:34:153
6805991610,5cyclictest500-21in:imuxsock08:10:353
680599160,1cyclictest0-21swapper/309:48:083
679999169,3cyclictest224-21systemd-journal09:21:122
679999167,4cyclictest224-21systemd-journal07:55:532
679999166,6cyclictest224-21systemd-journal12:12:352
679999164,4cyclictest224-21systemd-journal09:16:132
679999162,2cyclictest0-21swapper/211:35:512
679199163,11cyclictest500-21in:imuxsock11:27:061
6791991613,1cyclictest534-21snmpd10:03:241
679199161,13cyclictest32055-21diskmemload09:25:301
679199160,4cyclictest534-21snmpd10:42:301
679199160,2cyclictest224-21systemd-journal10:35:121
678799169,5cyclictest500-21in:imuxsock11:39:120
678799168,6cyclictest558-21lldpd09:50:150
678799168,6cyclictest32055-21diskmemload10:00:250
678799168,3cyclictest224-21systemd-journal12:05:560
678799165,3cyclictest224-21systemd-journal11:40:240
678799162,4cyclictest224-21systemd-journal10:35:000
6787991614,1cyclictest500-21in:imuxsock09:40:240
6787991613,2cyclictest500-21in:imuxsock10:12:150
6787991610,1cyclictest224-21systemd-journal10:22:440
678799160,15cyclictest0-21swapper/008:31:110
680599156,5cyclictest558-21lldpd10:00:383
680599156,4cyclictest224-21systemd-journal12:14:463
680599156,4cyclictest224-21systemd-journal10:25:243
680599156,2cyclictest224-21systemd-journal12:01:193
680599154,3cyclictest224-21systemd-journal07:45:383
680599153,7cyclictest534-21snmpd07:32:003
680599152,2cyclictest224-21systemd-journal11:47:183
680599151,1cyclictest224-21systemd-journal08:50:523
6805991510,3cyclictest27299-21grep07:50:253
6805991510,3cyclictest27299-21grep07:50:253
680599150,14cyclictest0-21swapper/312:37:473
679999157,4cyclictest224-21systemd-journal08:02:132
679999154,4cyclictest224-21systemd-journal10:53:422
679999153,3cyclictest534-21snmpd10:14:462
679999152,6cyclictest500-21in:imuxsock10:05:372
679999151,2cyclictest224-21systemd-journal11:53:122
679199158,6cyclictest500-21in:imuxsock11:05:361
679199158,3cyclictest224-21systemd-journal10:50:221
679199156,4cyclictest534-21snmpd09:46:081
679199156,2cyclictest224-21systemd-journal11:36:121
679199155,4cyclictest534-21snmpd07:25:001
679199151,5cyclictest224-21systemd-journal10:24:131
6791991514,1cyclictest0-21swapper/110:15:561
6791991511,2cyclictest502-21rs:main1
6791991511,1cyclictest224-21systemd-journal11:45:181
678799157,1cyclictest534-21snmpd10:45:270
678799156,6cyclictest534-21snmpd11:55:020
678799153,2cyclictest558-21lldpd09:47:590
678799153,11cyclictest32055-21diskmemload09:23:130
678799152,9cyclictest32055-21diskmemload10:50:220
6787991511,3cyclictest26871-21grep08:55:380
6787991510,3cyclictest1642-21grep09:35:300
65342156,5sleep00-21swapper/007:07:420
680599147,4cyclictest224-21systemd-journal11:28:133
680599146,4cyclictest224-21systemd-journal08:07:133
680599143,4cyclictest224-21systemd-journal10:57:223
680599143,4cyclictest224-21systemd-journal10:57:223
680599143,4cyclictest224-21systemd-journal09:05:003
680599143,4cyclictest224-21systemd-journal08:00:363
680599141,2cyclictest224-21systemd-journal09:51:353
680599141,2cyclictest0-21swapper/307:55:313
6805991410,3cyclictest781-21sh11:31:313
680599140,5cyclictest0-21swapper/308:30:013
679999149,4cyclictest25653-21sed07:45:412
679999148,5cyclictest12480-21expr07:20:262
679999146,3cyclictest224-21systemd-journal09:10:312
679999142,11cyclictest534-21snmpd07:12:522
679999140,14cyclictest0-21swapper/211:25:032
679199149,1cyclictest224-21systemd-journal07:15:361
679199147,6cyclictest558-21lldpd11:56:431
679199147,4cyclictest502-21rs:main1
679199146,5cyclictest500-21in:imuxsock09:00:251
679199146,3cyclictest534-21snmpd10:13:461
679199141,4cyclictest224-21systemd-journal07:10:371
6791991412,1cyclictest0-21swapper/112:20:551
679199141,1cyclictest558-21lldpd10:56:431
679199141,1cyclictest558-21lldpd10:56:431
6791991411,2cyclictest4983-21sh09:38:081
6791991410,3cyclictest17225-21cat08:35:371
678799149,3cyclictest19146-21grep08:40:300
678799148,2cyclictest534-21snmpd09:56:210
678799147,2cyclictest224-21systemd-journal11:27:240
678799146,4cyclictest502-21rs:main0
678799146,4cyclictest224-21systemd-journal12:11:140
678799146,4cyclictest224-21systemd-journal07:25:350
678799145,3cyclictest534-21snmpd09:28:340
678799142,3cyclictest5885-21aten2_r7power_e10:05:230
6787991410,3cyclictest29708-21grep07:55:270
6787991410,3cyclictest28888-21gzip09:00:340
6787991410,3cyclictest21838-21sort07:40:230
678799140,4cyclictest0-21swapper/011:03:170
680599137,2cyclictest224-21systemd-journal08:25:323
680599137,2cyclictest224-21systemd-journal07:15:263
680599134,3cyclictest224-21systemd-journal09:00:233
680599132,6cyclictest224-21systemd-journal09:31:083
680599132,4cyclictest224-21systemd-journal07:35:463
680599131,9cyclictest0-21swapper/312:07:093
680599131,8cyclictest224-21systemd-journal08:47:323
680599131,6cyclictest224-21systemd-journal08:21:183
6805991310,2cyclictest8646-21date07:10:403
6805991310,2cyclictest22623-21ssh09:26:133
679999139,3cyclictest28309-21head09:00:272
679999139,3cyclictest15970-21ssh09:45:502
679999139,3cyclictest1167-21ssh11:55:082
679999138,4cyclictest37850irq/131-snd_hda08:38:492
679999137,2cyclictest9494-21tr07:15:232
679999137,2cyclictest224-21systemd-journal07:35:302
679999135,6cyclictest22540-21/usr/sbin/munin07:40:282
679999135,2cyclictest224-21systemd-journal09:28:002
6799991310,2cyclictest30505-21sed09:05:252
6799991310,2cyclictest27785-21grep07:50:322
6799991310,2cyclictest27785-21grep07:50:312
6799991310,2cyclictest2473-21wc09:35:412
679199139,3cyclictest23719-21latency_hist07:45:001
679199139,3cyclictest22377-21ssh11:01:541
679199134,4cyclictest12550-21apache_volume11:40:221
679199134,3cyclictest224-21systemd-journal08:10:331
679199133,6cyclictest500-21in:imuxsock09:16:261
679199133,6cyclictest224-21systemd-journal08:50:231
679199132,6cyclictest502-21rs:main1
6791991311,1cyclictest21757-21mailstats08:45:331
6791991310,2cyclictest8582-21ssh12:27:441
6791991310,2cyclictest8433-21ssh09:40:271
6791991310,2cyclictest7272-21ls08:15:351
6791991310,2cyclictest23015-21ssh09:51:241
679199130,4cyclictest0-21swapper/112:15:591
679199130,12cyclictest0-21swapper/109:06:181
678799139,3cyclictest32664-21ssh09:10:210
678799139,3cyclictest28256-21tail07:50:360
678799139,3cyclictest28256-21tail07:50:360
678799138,2cyclictest224-21systemd-journal10:15:240
678799137,4cyclictest534-21snmpd07:30:330
678799135,4cyclictest558-21lldpd08:15:000
678799134,2cyclictest224-21systemd-journal08:35:340
678799133,6cyclictest558-21lldpd07:17:260
678799133,2cyclictest224-21systemd-journal12:03:130
678799133,1cyclictest0-21swapper/010:55:390
678799133,1cyclictest0-21swapper/010:55:380
6787991310,2cyclictest32143-21head08:00:250
6787991310,2cyclictest26118-21chrt07:47:420
6787991310,2cyclictest13573-21taskset12:31:110
680599124,4cyclictest224-21systemd-journal08:35:333
679999129,2cyclictest24750-21seq08:53:182
679999128,3cyclictest18818-21grep08:40:262
679999128,3cyclictest11634-21hddtemp_smartct08:25:282
679999128,2cyclictest6666-21grep08:15:292
679999124,4cyclictest224-21systemd-journal08:55:242
679999124,4cyclictest224-21systemd-journal08:10:212
679999124,4cyclictest224-21systemd-journal07:30:212
679999122,7cyclictest13218-21apache_processe08:30:202
679199129,2cyclictest9159-21cut08:21:101
679199129,2cyclictest3158-21chrt08:06:541
679199129,2cyclictest29034-21cut07:55:241
679199129,2cyclictest28237-21grep07:50:371
679199129,2cyclictest28237-21grep07:50:361
679199129,2cyclictest27751-21sh09:56:181
679199129,2cyclictest24113-21cat07:45:021
679199129,2cyclictest17119-21tr07:30:241
679199128,3cyclictest14976-21grep08:30:361
679199127,2cyclictest224-21systemd-journal08:25:211
678799129,2cyclictest9511-21wc08:21:130
678799129,2cyclictest29918-21taskset09:05:200
678799129,2cyclictest1589-21tr08:05:230
678799129,2cyclictest14130-21sh07:25:000
678799128,3cyclictest16453-21ssh10:35:170
678799124,4cyclictest224-21systemd-journal09:32:420
678799124,1cyclictest558-21lldpd08:48:410
679999112,2cyclictest558-21lldpd07:25:362
679199113,4cyclictest224-21systemd-journal08:00:151
679199113,4cyclictest224-21systemd-journal07:35:091
678799118,2cyclictest8178-21sh08:20:010
678799118,2cyclictest12069-21grep08:25:340
67999982,3cyclictest224-21systemd-journal07:10:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional