You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-13 - 10:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6s.osadl.org (updated Thu Nov 13, 2025 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
561121160,5sleep3300399cyclictest19:12:533
1464321090,43sleep014727-21grep20:38:010
1464321090,43sleep014727-21grep20:38:010
300399935,27cyclictest0-21swapper/319:23:453
300399933,70cyclictest25740-21needreboot00:22:583
300399894,25cyclictest0-21swapper/319:09:273
3003998922,10cyclictest5542-21aten_r1power_cu21:27:553
300399884,23cyclictest14818-21crond21:47:373
3003998428,49cyclictest22253-21runrttasks00:17:463
300399834,70cyclictest0-21swapper/300:13:003
3003998323,53cyclictest10345-21needreboot20:28:083
174992830,5sleep20-21swapper/220:43:062
3003998123,51cyclictest0-21swapper/319:47:583
300399804,12cyclictest0-21swapper/322:53:093
300399803,23cyclictest4063-21ls21:23:023
300399803,23cyclictest4063-21ls21:23:023
3003998024,49cyclictest1369-21runrttasks23:04:473
3003998019,53cyclictest1369-21runrttasks00:12:223
3003998019,53cyclictest1369-21runrttasks00:12:223
3003997926,46cyclictest1020-21df_abs21:17:553
300399785,25cyclictest0-21swapper/300:32:483
3003997823,48cyclictest20735-21gltestperf19:43:053
300399773,67cyclictest0-21swapper/323:37:513
3003997721,11cyclictest14504-21fschecks_count20:37:593
3003997721,11cyclictest14504-21fschecks_count20:37:593
3003997720,9cyclictest12230-21gltestperf22:47:543
3003997720,9cyclictest12230-21gltestperf22:47:543
3003997719,8cyclictest6292-21head23:42:533
3003997718,12cyclictest0-21swapper/319:22:303
3003997718,12cyclictest0-21swapper/319:22:303
243799770,6rtkit-daemon0-21swapper/119:08:180
300399764,23cyclictest0-21swapper/319:58:163
300399764,23cyclictest0-21swapper/319:58:163
300399764,22cyclictest0-21swapper/323:51:413
300399764,22cyclictest0-21swapper/320:52:543
300399764,22cyclictest0-21swapper/319:39:253
300399763,29cyclictest1369-21runrttasks22:22:553
300399763,25cyclictest0-21swapper/320:08:153
300399763,24cyclictest0-21swapper/322:01:423
300399763,23cyclictest1369-21runrttasks23:58:513
300399763,21cyclictest0-21swapper/300:03:123
300399763,14cyclictest0-21swapper/322:32:583
3003997623,10cyclictest10377-21aten_r1power_fr21:37:513
3003997615,10cyclictest2843-21gltestperf20:13:043
300399759,12cyclictest0-21swapper/321:50:213
300399755,16cyclictest0-21swapper/321:33:113
300399753,28cyclictest7680-21iostat_ios22:37:593
3003997522,8cyclictest11850-21sed23:53:083
3003997522,7cyclictest1369-21runrttasks20:36:583
3003997520,8cyclictest1369-21runrttasks22:19:443
3003997519,10cyclictest29012-21fschecks_time21:08:003
3003997518,9cyclictest28447-21cpuspeed_turbos23:22:533
300399745,21cyclictest0-21swapper/300:27:543
300399744,29cyclictest0-21swapper/323:07:473
300399744,26cyclictest0-21swapper/323:18:023
300399744,26cyclictest0-21swapper/319:55:173
300399744,24cyclictest17216-21sed20:43:013
300399744,24cyclictest0-21swapper/320:03:183
300399744,21cyclictest0-21swapper/321:12:543
300399743,21cyclictest0-21swapper/323:32:243
300399743,21cyclictest0-21swapper/323:32:243
3003997423,10cyclictest7094-21cpuspeed_turbos20:23:013
3003997422,9cyclictest26246-21timerwakeupswit22:08:113
3003997421,9cyclictest2375-21unixbench_singl23:33:093
3003997418,10cyclictest13567-21gltestperf19:28:033
300399733,28cyclictest0-21swapper/322:31:463
3003997323,8cyclictest1369-21runrttasks21:55:463
3003997320,8cyclictest4940-21entropy20:17:593
3003997319,10cyclictest1243-21snmpd23:14:163
300399723,21cyclictest0-21swapper/320:57:533
3003997222,8cyclictest26380-21df_abs21:02:563
3003997218,9cyclictest9372-21cpuspeed_turbos22:42:543
3003997218,9cyclictest9372-21cpuspeed_turbos22:42:543
3003997216,10cyclictest18971-21cpuspeed_turbos20:47:553
3003997210,12cyclictest0-21swapper/319:33:033
3003997121,9cyclictest1369-21runrttasks22:57:413
3003997113,12cyclictest0-21swapper/322:03:093
3003997113,12cyclictest0-21swapper/322:03:093
3003997112,8cyclictest114550irq/46-eth022:14:583
3003997020,9cyclictest3088-21latency_hist19:07:453
69992633,53sleep20-21swapper/221:28:172
300299635,13cyclictest0-21swapper/223:57:332
300299625,29cyclictest0-21swapper/219:18:092
300299625,29cyclictest0-21swapper/219:18:092
300299624,31cyclictest0-21swapper/221:38:132
300299623,37cyclictest14998-21munin-node00:02:582
300299622,24cyclictest0-21swapper/200:17:452
300299614,33cyclictest0-21swapper/221:03:102
300199605,31cyclictest0-21swapper/122:07:551
300199605,30cyclictest0-21swapper/123:37:521
300199604,31cyclictest0-21swapper/100:17:581
300199604,15cyclictest0-21swapper/121:03:071
300199604,14cyclictest0-21swapper/122:43:131
300199604,14cyclictest0-21swapper/122:43:131
52732593,48sleep10-21swapper/121:27:401
52732593,48sleep10-21swapper/121:27:401
300299594,29cyclictest0-21swapper/221:43:102
3002995922,14cyclictest0-21swapper/219:57:542
3002995922,14cyclictest0-21swapper/219:57:542
300199594,30cyclictest0-21swapper/123:07:511
300199594,30cyclictest0-21swapper/119:53:101
300199594,29cyclictest0-21swapper/119:32:571
3001995922,15cyclictest0-21swapper/119:23:051
243799590,5rtkit-daemon2436-21rtkit-daemon20:05:161
3002995838,13cyclictest0-21swapper/223:17:342
3002995822,15cyclictest0-21swapper/220:47:542
300199585,29cyclictest0-21swapper/120:29:221
300199584,31cyclictest0-21swapper/120:23:201
300199584,30cyclictest0-21swapper/123:28:061
300199584,30cyclictest0-21swapper/123:28:061
300199584,30cyclictest0-21swapper/100:33:031
300199584,29cyclictest0-21swapper/122:22:381
300299574,12cyclictest0-21swapper/220:22:412
238362574,46sleep20-21swapper/219:48:162
300299564,13cyclictest0-21swapper/221:08:042
300199564,31cyclictest0-21swapper/121:13:071
300199564,27cyclictest0-21swapper/100:24:151
3001995625,23cyclictest0-21swapper/122:16:191
243799560,5rtkit-daemon2436-21rtkit-daemon23:01:321
300299554,12cyclictest0-21swapper/222:09:062
300299553,12cyclictest0-21swapper/221:22:402
3001995540,10cyclictest20573-21gltestperf00:12:511
299999553,46cyclictest0-21swapper/000:07:560
299999553,46cyclictest0-21swapper/000:07:560
299999553,24cyclictest0-21swapper/021:53:050
300299544,42cyclictest0-21swapper/222:32:522
300299544,29cyclictest0-21swapper/223:22:522
300299544,28cyclictest0-21swapper/220:03:132
299999545,31cyclictest0-21swapper/022:17:370
2999995412,21cyclictest0-21swapper/021:22:550
2999995412,21cyclictest0-21swapper/021:22:550
264862542,45sleep026517-21fschecks_time23:17:540
133472542,10sleep20-21swapper/222:49:392
133472542,10sleep20-21swapper/222:49:392
3002995334,11cyclictest13514-21sort19:28:022
3002995333,13cyclictest14243-21grep20:37:552
3002995333,13cyclictest14243-21grep20:37:552
3002995333,12cyclictest0-21swapper/223:03:082
3001995337,10cyclictest7010-21tr22:37:501
3001995336,10cyclictest11890-21tr20:32:591
243799530,5rtkit-daemon2436-21rtkit-daemon20:01:311
243799530,5rtkit-daemon2436-21rtkit-daemon20:01:311
172292532,10sleep00-21swapper/019:36:580
300299526,28cyclictest0-21swapper/223:17:532
300299525,29cyclictest0-21swapper/223:52:332
300299525,29cyclictest0-21swapper/222:42:572
300299525,29cyclictest0-21swapper/222:42:572
300299525,29cyclictest0-21swapper/221:24:492
300299525,29cyclictest0-21swapper/221:24:492
300299525,28cyclictest0-21swapper/219:53:102
300299524,30cyclictest0-21swapper/220:33:092
300299524,30cyclictest0-21swapper/219:43:152
300299524,26cyclictest0-21swapper/221:57:512
3002995233,12cyclictest5959-21grep19:12:562
3002995232,12cyclictest0-21swapper/222:18:062
300299523,27cyclictest0-21swapper/223:37:172
300199525,29cyclictest0-21swapper/123:17:591
300199525,29cyclictest0-21swapper/121:38:131
3001995233,11cyclictest5643-21tail21:27:571
3001995222,14cyclictest0-21swapper/122:02:581
3001995222,14cyclictest0-21swapper/122:02:581
299999524,42cyclictest0-21swapper/020:17:550
299999524,40cyclictest0-21swapper/000:12:440
243452522,44sleep00-21swapper/022:07:130
243452522,44sleep00-21swapper/022:07:130
217922522,11sleep00-21swapper/019:44:110
154322523,43sleep115556-21timerwakeupswit20:38:141
154322523,43sleep115556-21timerwakeupswit20:38:141
14752522,10sleep00-21swapper/022:27:120
61562512,10sleep00-21swapper/022:34:130
300299516,27cyclictest0-21swapper/223:57:442
300299515,29cyclictest0-21swapper/219:33:162
300299514,30cyclictest0-21swapper/222:03:002
300299514,30cyclictest0-21swapper/222:03:002
300299514,29cyclictest0-21swapper/223:07:502
300299514,29cyclictest0-21swapper/222:13:072
300299514,29cyclictest0-21swapper/221:12:572
300299514,29cyclictest0-21swapper/220:23:192
300299514,29cyclictest0-21swapper/220:08:112
300299514,29cyclictest0-21swapper/219:23:052
300299513,30cyclictest0-21swapper/220:27:562
3002995118,14cyclictest0-21swapper/223:37:442
300199515,28cyclictest0-21swapper/100:12:331
300199515,28cyclictest0-21swapper/100:12:331
300199515,25cyclictest11453-21aten2.4-expect22:47:461
300199515,25cyclictest11453-21aten2.4-expect22:47:461
300199514,30cyclictest0-21swapper/122:57:361
300199514,30cyclictest0-21swapper/121:33:131
300199514,30cyclictest0-21swapper/119:28:181
300199514,29cyclictest0-21swapper/121:53:111
300199514,29cyclictest0-21swapper/121:18:061
300199514,27cyclictest29265-21runrttasks21:08:031
3001995132,11cyclictest11893-21grep23:53:091
299999514,28cyclictest0-21swapper/021:07:520
2999995137,8cyclictest22414-21sendmail_mailqu23:08:050
2999995134,10cyclictest6658-21munin-node22:38:040
299999513,41cyclictest0-21swapper/023:47:470
27092513,10sleep10-21swapper/123:36:381
254512513,9sleep10-21swapper/121:00:351
243799510,5rtkit-daemon2436-21rtkit-daemon23:59:371
243799510,5rtkit-daemon2436-21rtkit-daemon20:53:071
243799510,5rtkit-daemon2436-21rtkit-daemon20:43:521
243799510,5rtkit-daemon2436-21rtkit-daemon19:38:101
243799510,5rtkit-daemon2436-21rtkit-daemon00:03:071
144762513,10sleep10-21swapper/121:43:161
101552513,40sleep00-21swapper/020:28:050
327542503,40sleep10-21swapper/122:23:021
302502502,10sleep00-21swapper/023:27:010
300299505,29cyclictest0-21swapper/220:13:032
300299505,28cyclictest0-21swapper/221:54:552
300299505,28cyclictest0-21swapper/200:22:522
300299505,27cyclictest0-21swapper/200:12:512
300299504,30cyclictest0-21swapper/222:38:092
300299504,25cyclictest0-21swapper/200:07:452
300299504,25cyclictest0-21swapper/200:07:452
3002995032,11cyclictest13815-21munin-node22:52:572
300299503,30cyclictest0-21swapper/222:28:122
3002995019,14cyclictest0-21swapper/222:57:582
300199505,28cyclictest0-21swapper/122:30:011
300199504,29cyclictest0-21swapper/121:48:041
300199504,28cyclictest0-21swapper/122:33:081
300199504,27cyclictest0-21swapper/120:52:421
3001995036,8cyclictest9495-21tail23:48:061
299999506,27cyclictest0-21swapper/021:48:070
299999504,29cyclictest0-21swapper/020:23:130
299999504,22cyclictest0-21swapper/023:27:500
299999504,22cyclictest0-21swapper/023:27:500
2999995035,10cyclictest0-21swapper/022:18:070
2999995035,10cyclictest0-21swapper/019:52:560
299999503,29cyclictest0-21swapper/020:43:130
2999995020,12cyclictest0-21swapper/019:58:060
2999995020,12cyclictest0-21swapper/019:58:060
288442502,9sleep20-21swapper/200:29:352
188272504,10sleep00-21swapper/023:02:510
180922502,9sleep00-21swapper/022:59:040
135062502,10sleep00-21swapper/020:35:390
131582502,42sleep00-21swapper/019:27:590
300299495,13cyclictest0-21swapper/223:42:532
300299494,28cyclictest0-21swapper/223:28:082
300299494,28cyclictest0-21swapper/223:28:082
300299494,28cyclictest0-21swapper/222:22:552
300299494,25cyclictest0-21swapper/219:37:582
300299494,24cyclictest0-21swapper/221:47:522
300299494,13cyclictest0-21swapper/219:08:132
3002994928,14cyclictest0-21swapper/220:57:512
299999494,28cyclictest0-21swapper/019:18:050
299999494,28cyclictest0-21swapper/019:18:050
299999494,28cyclictest0-21swapper/000:28:070
299999494,27cyclictest0-21swapper/023:12:520
299999494,27cyclictest0-21swapper/020:13:180
2999994935,10cyclictest0-21swapper/021:33:060
2999994932,12cyclictest0-21swapper/021:12:590
2999994929,13cyclictest0-21swapper/023:57:430
265502492,42sleep00-21swapper/022:09:470
175212493,39sleep00-21swapper/019:37:460
111492492,10sleep00-21swapper/022:47:070
111492492,10sleep00-21swapper/022:47:070
74862482,39sleep00-21swapper/021:30:250
318682482,39sleep00-21swapper/020:07:540
3002994826,15cyclictest0-21swapper/221:33:012
3002994816,14cyclictest0-21swapper/200:32:472
300199485,36cyclictest0-21swapper/123:22:591
3001994834,9cyclictest21498-21sed21:58:121
3001994819,12cyclictest0-21swapper/119:08:011
3001994818,14cyclictest0-21swapper/119:42:581
3001994817,14cyclictest0-21swapper/119:13:061
3001994816,13cyclictest0-21swapper/119:49:341
299999484,27cyclictest0-21swapper/000:22:440
299999483,29cyclictest0-21swapper/021:18:130
299999483,27cyclictest0-21swapper/021:42:380
299999483,26cyclictest0-21swapper/020:52:540
2999994813,13cyclictest0-21swapper/000:22:330
288332484,39sleep10-21swapper/100:29:261
26812482,39sleep00-21swapper/022:28:010
243799480,5rtkit-daemon2436-21rtkit-daemon23:55:520
243799480,5rtkit-daemon2436-21rtkit-daemon20:10:231
243799480,4rtkit-daemon2436-21rtkit-daemon19:20:251
243799480,4rtkit-daemon2436-21rtkit-daemon19:20:251
241732482,40sleep00-21swapper/019:49:070
146282482,10sleep00-21swapper/021:45:120
300299474,26cyclictest0-21swapper/220:52:542
300199475,26cyclictest0-21swapper/120:14:381
300199474,35cyclictest25526-21crond23:17:341
299999474,37cyclictest0-21swapper/020:03:020
299999474,27cyclictest0-21swapper/000:02:510
299999474,26cyclictest0-21swapper/023:38:050
299999474,26cyclictest0-21swapper/019:24:290
299999473,28cyclictest0-21swapper/021:59:400
299999473,27cyclictest0-21swapper/023:43:050
299999473,27cyclictest0-21swapper/020:48:010
299999473,26cyclictest0-21swapper/022:48:000
299999473,26cyclictest0-21swapper/022:48:000
253492472,39sleep00-21swapper/020:59:170
77002462,38sleep00-21swapper/019:17:100
73412463,37sleep10-21swapper/123:44:441
6432464,34sleep00-21swapper/019:02:570
311222462,9sleep00-21swapper/000:33:110
26322464,6sleep20-21swapper/219:03:272
300199454,23cyclictest0-21swapper/120:17:551
299999454,22cyclictest0-21swapper/022:52:480
2999994518,9cyclictest1243-21snmpd23:35:110
2999994516,12cyclictest0-21swapper/021:03:040
196312453,10sleep10-21swapper/123:03:001
3001994115,8cyclictest2975-21latency_hist19:07:451
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional