You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 05:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6s.osadl.org (updated Mon Dec 01, 2025 00:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2437991090,6rtkit-daemon0-21swapper/322:30:563
211672970,7sleep121168-21ls21:31:051
815999915,78cyclictest0-21swapper/021:05:440
815999907,76cyclictest25963-21awk22:50:400
815999907,76cyclictest25963-21awk22:50:400
815999883,78cyclictest0-21swapper/023:50:580
8164998636,10cyclictest13669-21grep20:06:221
815999857,70cyclictest0-21swapper/019:02:490
815999856,71cyclictest0-21swapper/019:16:020
815999856,71cyclictest0-21swapper/019:16:020
815999826,68cyclictest0-21swapper/020:31:070
815999824,60cyclictest0-21swapper/020:51:180
815999806,51cyclictest0-21swapper/019:41:220
8159998034,39cyclictest17682-21crond19:10:490
815999802,72cyclictest0-21swapper/023:10:350
8159998018,45cyclictest261881munin-node22:51:100
8159998018,43cyclictest17848-21munin-node21:26:020
8159998018,43cyclictest17848-21munin-node21:26:020
815999795,67cyclictest0-21swapper/020:08:000
815999794,55cyclictest0-21swapper/019:36:010
815999783,67cyclictest11597-21sshd21:11:060
815999783,57cyclictest0-21swapper/020:43:070
815999783,57cyclictest0-21swapper/020:43:070
816499779,23cyclictest0-21swapper/120:30:461
8159997733,38cyclictest30486-21tune2fs21:51:000
815999773,67cyclictest0-21swapper/023:59:350
815999773,66cyclictest0-21swapper/019:53:080
8159997715,43cyclictest0-21swapper/021:55:560
815999766,51cyclictest0-21swapper/020:21:190
815999765,63cyclictest0-21swapper/023:39:030
8159997634,36cyclictest13791-21cut21:16:030
8159997630,40cyclictest1369-21runrttasks23:32:570
8159997630,40cyclictest1369-21runrttasks23:32:570
815999763,65cyclictest8384-21munin-node18:51:110
815999763,65cyclictest25702-21taskset00:00:490
815999754,64cyclictest0-21swapper/020:01:220
815999753,65cyclictest7297-21tail19:56:010
815999753,65cyclictest7297-21tail19:56:010
815999753,64cyclictest0-21swapper/022:10:560
8159997528,39cyclictest25904-21gltestperf19:26:070
8159997527,41cyclictest27857-21df21:45:590
815999752,64cyclictest16621-21munin-node22:31:070
815999746,61cyclictest0-21swapper/019:15:490
815999744,63cyclictest0-21swapper/021:40:550
815999743,64cyclictest223652chrt21:35:310
8159997427,40cyclictest22292-21grep20:26:070
815999737,59cyclictest0-21swapper/020:16:080
815999733,62cyclictest15432-21missed_timers20:11:120
815999733,56cyclictest28777-21open_inodes19:31:190
8159997326,41cyclictest1369-21runrttasks22:30:090
8159997322,44cyclictest5011-21grep22:05:580
8159997322,44cyclictest5011-21grep22:05:580
815999726,59cyclictest0-21swapper/023:45:390
815999724,61cyclictest0-21swapper/000:16:080
815999724,61cyclictest0-21swapper/000:16:080
815999723,54cyclictest0-21swapper/021:38:250
815999723,39cyclictest0-21swapper/022:38:340
8159997224,42cyclictest27647-21gzip20:36:170
815999713,53cyclictest0-21swapper/022:20:540
8159997126,39cyclictest9136-21tail21:06:050
8159997126,39cyclictest1369-21runrttasks19:47:540
8159997123,41cyclictest4240-21tune2fs20:56:040
8159997119,46cyclictest1369-21runrttasks23:29:070
8159997030,34cyclictest9915-21unixbench_multi23:21:120
815999697,51cyclictest0-21swapper/020:46:010
815999693,60cyclictest5598-21latency_hist23:15:400
815999693,60cyclictest0-21swapper/022:17:000
815999693,51cyclictest8141-21latency_hist18:50:510
8159996923,40cyclictest1369-21runrttasks00:09:170
815999692,60cyclictest30347-21munin-node00:11:150
8159996913,50cyclictest1369-21runrttasks23:19:030
815999686,48cyclictest0-21swapper/023:00:520
815999683,51cyclictest30018-21python22:56:120
8159996820,41cyclictest1369-21runrttasks22:42:560
815999676,55cyclictest0-21swapper/018:57:030
815999673,52cyclictest22695-21munin-node19:21:120
815999673,46cyclictest20567-21latency_hist23:50:390
816699664,42cyclictest0-21swapper/222:35:402
815999663,57cyclictest0-21swapper/021:21:240
815999663,57cyclictest0-21swapper/021:21:240
815999663,56cyclictest2383-21telnet22:00:540
243799650,5rtkit-daemon21644-21gzip21:31:133
294332640,4sleep329430-21grep20:41:053
294332640,4sleep329430-21grep20:41:053
243799640,6rtkit-daemon0-21swapper/222:29:413
243799640,4rtkit-daemon2436-21rtkit-daemon18:56:261
8167996312,33cyclictest0-21swapper/320:46:103
8167996210,44cyclictest20795-21df_abs19:16:053
8167996210,44cyclictest20795-21df_abs19:16:053
243799620,6rtkit-daemon0-21swapper/223:20:393
816799618,46cyclictest0-21swapper/319:01:123
816799608,45cyclictest0-21swapper/322:00:433
816799608,36cyclictest0-21swapper/323:31:573
816799608,36cyclictest0-21swapper/323:31:573
816799607,40cyclictest0-21swapper/319:45:273
816799605,49cyclictest0-21swapper/321:40:443
8167996021,32cyclictest31599-21unixbench_singl19:36:213
243799600,5rtkit-daemon0-21swapper/223:55:223
816699594,12cyclictest0-21swapper/223:21:082
816699593,49cyclictest9179-21grep21:06:052
243799590,5rtkit-daemon0-21swapper/223:46:533
816799589,42cyclictest0-21swapper/300:19:533
816799589,42cyclictest0-21swapper/300:19:533
816799588,35cyclictest0-21swapper/318:56:043
816799584,48cyclictest0-21swapper/300:10:383
8167995814,37cyclictest8276-21head22:11:083
8167995813,38cyclictest10464-21munin-run21:10:443
8167995810,41cyclictest0-21swapper/320:16:043
816499584,46cyclictest0-21swapper/123:56:031
243799580,5rtkit-daemon0-21swapper/222:55:473
816699573,11cyclictest0-21swapper/221:41:152
8166995721,30cyclictest2018-21crond19:45:472
291722573,12sleep20-21swapper/200:06:042
816799563,30cyclictest0-21swapper/300:10:503
8167995613,36cyclictest12566-21fschecks_count20:06:063
816699564,12cyclictest0-21swapper/221:05:452
816699563,12cyclictest0-21swapper/220:26:092
816799556,44cyclictest0-21swapper/319:26:163
8167995519,25cyclictest16788-21awk19:06:133
816699554,11cyclictest0-21swapper/219:56:132
816699554,11cyclictest0-21swapper/219:56:132
816799549,39cyclictest0-21swapper/322:40:403
816799548,40cyclictest0-21swapper/320:51:073
816799547,40cyclictest19378-21munin-node21:26:163
816799547,40cyclictest19378-21munin-node21:26:163
816799546,42cyclictest0-21swapper/319:51:023
816799546,41cyclictest2331-21munin-node19:46:183
816799544,44cyclictest0-21swapper/321:03:103
8167995412,35cyclictest1243-21snmpd22:15:503
816699544,30cyclictest0-21swapper/222:16:072
816699543,13cyclictest0-21swapper/220:55:462
8166995421,14cyclictest0-21swapper/223:26:092
8164995439,10cyclictest0-21swapper/120:50:461
8164995433,13cyclictest0-21swapper/121:06:061
8164995420,14cyclictest0-21swapper/120:45:191
8164995420,14cyclictest0-21swapper/120:45:191
23302543,43sleep22391-21aten2.4-expect19:45:592
816799538,38cyclictest15464-21munin-node21:21:183
816799538,38cyclictest15464-21munin-node21:21:183
816799538,37cyclictest1243-21snmpd19:22:523
816799536,40cyclictest11948-21taskset21:11:113
816799533,30cyclictest0-21swapper/320:11:123
8167995317,29cyclictest23334-21aten_r1power_vo23:55:473
816699534,30cyclictest0-21swapper/220:31:052
816699534,29cyclictest14417-21sendmail_mailqu23:31:092
816699534,29cyclictest14417-21sendmail_mailqu23:31:092
816699534,29cyclictest0-21swapper/219:01:152
816699533,27cyclictest26285-21chrt22:50:542
8166995320,14cyclictest0-21swapper/220:11:172
816499534,30cyclictest0-21swapper/122:16:011
8164995331,15cyclictest0-21swapper/120:50:451
816799529,35cyclictest0-21swapper/322:55:413
816799529,30cyclictest0-21swapper/320:02:593
816799529,12cyclictest16894-21sort23:36:103
816799527,30cyclictest0-21swapper/323:10:403
816799524,30cyclictest0-21swapper/322:40:533
8167995219,15cyclictest0-21swapper/321:00:033
8167995210,27cyclictest5153-21sed23:11:143
816699524,41cyclictest0-21swapper/221:56:122
816699524,30cyclictest0-21swapper/220:40:592
816699524,30cyclictest0-21swapper/220:40:592
816699524,30cyclictest0-21swapper/220:16:192
816699524,30cyclictest0-21swapper/218:56:152
816699524,29cyclictest0-21swapper/222:11:042
816699524,29cyclictest0-21swapper/221:51:022
816699524,29cyclictest0-21swapper/221:30:552
816699524,29cyclictest0-21swapper/219:26:082
8166995234,11cyclictest17099-21unixbench_multi23:36:122
816699523,28cyclictest0-21swapper/219:16:192
816699523,28cyclictest0-21swapper/219:16:192
8166995210,35cyclictest281ktimersoftd/222:06:002
8166995210,35cyclictest281ktimersoftd/222:06:002
816499525,39cyclictest0-21swapper/118:51:151
816499524,34cyclictest0-21swapper/119:51:111
816499524,23cyclictest5938-21tail21:00:551
8164995233,12cyclictest10419-21gltestperf20:01:081
8164995232,13cyclictest0-21swapper/120:16:081
816799518,36cyclictest9834-21python18:51:223
816799517,37cyclictest0-21swapper/320:36:113
816799516,28cyclictest0-21swapper/320:26:193
816799514,29cyclictest0-21swapper/322:21:013
8167995121,13cyclictest0-21swapper/321:50:433
816699515,29cyclictest0-21swapper/219:10:502
816699514,30cyclictest0-21swapper/223:00:512
816699514,29cyclictest0-21swapper/222:26:162
816699514,29cyclictest0-21swapper/220:21:012
816699514,29cyclictest0-21swapper/219:31:102
816699514,29cyclictest0-21swapper/200:20:372
816699514,29cyclictest0-21swapper/200:20:372
816699514,27cyclictest0-21swapper/219:51:032
816699514,12cyclictest0-21swapper/219:36:032
8166995135,10cyclictest13170-21nfs_client20:06:142
8166995120,13cyclictest0-21swapper/223:05:532
816499517,24cyclictest0-21swapper/123:35:501
816499514,29cyclictest0-21swapper/123:51:071
816499514,29cyclictest0-21swapper/119:11:061
816499514,28cyclictest0-21swapper/120:11:211
816499514,24cyclictest3537-21aten2.4-expect23:10:511
8164995131,13cyclictest13594-21df_abs19:01:081
8164995129,14cyclictest0-21swapper/119:21:131
8164995122,12cyclictest0-21swapper/121:46:031
816799507,26cyclictest26877-21open_inodes00:01:043
816799504,28cyclictest0-21swapper/322:05:563
816799504,28cyclictest0-21swapper/322:05:563
816799504,28cyclictest0-21swapper/322:00:553
816799504,28cyclictest0-21swapper/321:41:003
8167995027,17cyclictest1243-21snmpd23:45:273
8167995020,13cyclictest0-21swapper/323:25:563
816699505,26cyclictest0-21swapper/223:55:462
816699505,12cyclictest0-21swapper/222:23:322
816699504,29cyclictest0-21swapper/221:18:132
816699504,29cyclictest0-21swapper/200:11:032
816699504,28cyclictest0-21swapper/222:40:542
816699504,28cyclictest0-21swapper/222:00:562
816699504,28cyclictest0-21swapper/218:51:202
816699504,27cyclictest0-21swapper/221:45:572
8166995019,13cyclictest0-21swapper/220:56:142
816499506,27cyclictest0-21swapper/122:23:311
816499504,29cyclictest0-21swapper/123:01:061
816499504,29cyclictest0-21swapper/120:21:161
816499504,28cyclictest0-21swapper/123:15:541
816499504,28cyclictest0-21swapper/100:16:081
816499504,28cyclictest0-21swapper/100:16:081
816499504,27cyclictest0-21swapper/122:56:141
816499504,26cyclictest0-21swapper/122:38:341
8164995031,13cyclictest0-21swapper/100:05:381
816799499,34cyclictest0-21swapper/323:00:583
816799494,40cyclictest0-21swapper/321:15:573
816799494,28cyclictest0-21swapper/321:51:093
816799494,26cyclictest0-21swapper/323:23:413
8167994910,22cyclictest0-21swapper/319:15:183
816699499,22cyclictest18372-1kworker/2:0H21:30:432
816699499,22cyclictest18372-1kworker/2:0H21:30:432
816699496,27cyclictest0-21swapper/223:50:552
816699494,28cyclictest0-21swapper/220:48:082
816699494,28cyclictest0-21swapper/220:36:012
816699494,28cyclictest0-21swapper/219:22:522
816699494,27cyclictest0-21swapper/222:36:032
8166994924,12cyclictest0-21swapper/221:36:172
816499494,28cyclictest0-21swapper/121:28:171
816499494,28cyclictest0-21swapper/121:28:171
816499494,28cyclictest0-21swapper/119:41:221
816499494,27cyclictest0-21swapper/120:00:481
816499494,27cyclictest0-21swapper/120:00:481
816499494,27cyclictest0-21swapper/119:26:231
8164994916,24cyclictest0-21swapper/121:40:561
816799487,33cyclictest24857-21interrupts20:31:093
8167994811,29cyclictest0-21swapper/319:56:183
8167994811,29cyclictest0-21swapper/319:56:183
816699484,27cyclictest0-21swapper/223:13:402
8166994832,10cyclictest6152-21cpuspeed_turbos23:15:552
816699482,39cyclictest0-21swapper/221:13:272
8166994819,13cyclictest0-21swapper/219:11:232
8166994818,13cyclictest0-21swapper/222:45:552
8166994818,13cyclictest0-21swapper/222:45:552
816499485,13cyclictest0-21swapper/121:11:121
816499484,37cyclictest0-21swapper/122:01:021
816499484,34cyclictest17776-21munin-node23:41:061
816499484,27cyclictest0-21swapper/122:53:371
816499484,12cyclictest0-21swapper/122:31:131
8164994835,8cyclictest7864-21gltestperf22:11:021
258572482,9sleep10-21swapper/120:33:581
816699475,35cyclictest0-21swapper/223:40:502
816699474,28cyclictest0-21swapper/200:02:032
816699474,26cyclictest0-21swapper/222:55:522
8166994715,14cyclictest0-21swapper/221:21:072
8166994715,14cyclictest0-21swapper/221:21:072
816499475,35cyclictest0-21swapper/121:56:091
816499474,34cyclictest0-21swapper/122:45:421
816499474,27cyclictest0-21swapper/122:50:411
816499474,27cyclictest0-21swapper/122:50:411
816499474,26cyclictest0-21swapper/119:17:511
816499474,26cyclictest0-21swapper/119:17:511
816499473,27cyclictest0-21swapper/122:06:031
816499473,27cyclictest0-21swapper/122:06:031
8164994710,21cyclictest0-21swapper/123:25:521
816799465,36cyclictest0-21swapper/319:35:483
816799464,13cyclictest0-21swapper/322:46:153
816799464,13cyclictest0-21swapper/322:46:153
816499464,35cyclictest0-21swapper/120:36:081
816499464,26cyclictest0-21swapper/123:47:331
8164994631,10cyclictest0-21swapper/119:46:151
8164994630,10cyclictest0-21swapper/121:50:541
8164994628,12cyclictest0-21swapper/120:56:161
8164994626,14cyclictest0-21swapper/119:31:031
155282462,9sleep10-21swapper/119:05:551
8167994524,14cyclictest0-21swapper/320:21:073
816499455,14cyclictest0-21swapper/123:31:071
816499455,14cyclictest0-21swapper/123:31:071
816499454,24cyclictest9630-21rt-features23:21:081
816499454,14cyclictest0-21swapper/121:23:141
816499454,14cyclictest0-21swapper/121:23:141
816499453,23cyclictest15028-21taskset21:18:131
8164994517,12cyclictest0-21swapper/122:25:541
8164994515,13cyclictest0-21swapper/100:05:571
8164994512,26cyclictest30787-21df_abs00:10:551
816699444,25cyclictest0-21swapper/220:01:012
816499444,33cyclictest0-21swapper/121:36:111
8164994427,10cyclictest1741-21grep23:05:591
8164994326,12cyclictest0-21swapper/119:35:581
8167994210,18cyclictest0-21swapper/318:50:513
8164994119,10cyclictest0-21swapper/118:50:511
816699403,17cyclictest12712-1kworker/2:0H23:46:102
816699354,13cyclictest0-21swapper/218:50:512
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional