You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-13 - 07:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed May 13, 2026 00:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
534549912211,10cyclictest131rcu_preempt00:05:021
534499911139,59cyclictest138165-21kworker/u8:000:30:020
5344999907,68cyclictest93443-21rm22:20:020
53449998912,55cyclictest138483-21ssh23:53:500
5344999748,45cyclictest0-21swapper/021:31:360
5344999730,41cyclictest0-21swapper/000:00:010
5344999723,30cyclictest0-21swapper/023:09:040
5344999686,32cyclictest0-21swapper/021:21:130
53456996644,7cyclictest105383-21kworker/2:323:25:042
53456996644,7cyclictest105383-21kworker/2:323:25:042
5344999661,29cyclictest0-21swapper/020:18:380
53449996611,16cyclictest111161-21sh22:56:590
53449996611,16cyclictest111161-21sh22:56:590
5346099657,7cyclictest112480-21cron23:00:003
5346099657,7cyclictest112480-21cron23:00:003
53449996517,10cyclictest12-21ksoftirqd/019:55:020
53449996513,19cyclictest60086-21ssh21:10:450
53449996430,6cyclictest0-21swapper/023:45:040
5346099637,7cyclictest37-21ksoftirqd/322:54:533
53449996317,10cyclictest12-21ksoftirqd/021:36:500
53449996317,10cyclictest12-21ksoftirqd/020:05:020
53449996313,15cyclictest102654-21ssh22:39:290
53456996253,5cyclictest59385-21kworker/2:221:15:202
5344999624,28cyclictest12-21ksoftirqd/022:21:050
53449996240,8cyclictest12-21ksoftirqd/000:04:330
53449996217,8cyclictest12-21ksoftirqd/022:40:450
53449996215,17cyclictest87375-21ssh22:07:270
53460996120,12cyclictest37-21ksoftirqd/300:20:003
53460996119,12cyclictest37-21ksoftirqd/322:45:013
53456996152,6cyclictest59385-21kworker/2:221:14:262
53449996118,8cyclictest12-21ksoftirqd/021:57:480
53449996117,7cyclictest12-21ksoftirqd/022:29:350
53449996116,15cyclictest108497-21sh22:51:190
53449996116,10cyclictest12-21ksoftirqd/021:42:150
53449996113,12cyclictest153542-21munin-run00:25:000
53460996015,15cyclictest37-21ksoftirqd/321:45:013
5344999607,23cyclictest120560-21ssh23:16:280
5344999607,23cyclictest120560-21ssh23:16:270
5344999606,50cyclictest89146-21ssh22:10:590
53449996017,9cyclictest12-21ksoftirqd/023:20:070
5346099598,11cyclictest37-21ksoftirqd/319:59:593
53449995910,16cyclictest148396-21sh00:14:240
53449995910,16cyclictest148396-21sh00:14:240
53456995849,5cyclictest59385-21kworker/2:222:35:592
53449995826,16cyclictest135016-21ssh23:46:270
53449995826,16cyclictest135016-21ssh23:46:270
5346099579,11cyclictest37-21ksoftirqd/322:22:583
5346099576,9cyclictest37-21ksoftirqd/322:20:013
5346099576,7cyclictest37-21ksoftirqd/322:05:003
53460995715,9cyclictest37-21ksoftirqd/321:15:183
53456995747,6cyclictest39308-21kworker/2:122:14:592
5344999577,46cyclictest79694-21ssh21:51:200
5344999577,21cyclictest124806-21ssh23:25:060
5344999577,21cyclictest124806-21ssh23:25:060
5344999573,37cyclictest127512-21ssh23:30:460
53449995714,8cyclictest12-21ksoftirqd/000:40:020
53449995713,37cyclictest309-21in:imuxsock19:40:000
5346099567,9cyclictest37-21ksoftirqd/300:35:003
5346099565,9cyclictest37-21ksoftirqd/323:50:023
5346099565,9cyclictest37-21ksoftirqd/323:50:013
5346099565,8cyclictest37-21ksoftirqd/321:31:043
5346099564,14cyclictest37-21ksoftirqd/322:40:013
53456995648,5cyclictest59385-21kworker/2:222:31:362
53456995647,5cyclictest59385-21kworker/2:221:45:572
53456995647,5cyclictest105383-21kworker/2:322:50:022
53456995640,11cyclictest30-21ksoftirqd/221:00:012
5344999567,22cyclictest12-21ksoftirqd/023:11:470
53449995612,9cyclictest12-21ksoftirqd/000:05:090
53449995612,11cyclictest157440-21ssh00:33:030
5346099556,9cyclictest37-21ksoftirqd/300:01:223
5346099554,10cyclictest37-21ksoftirqd/320:15:003
5346099553,9cyclictest37-21ksoftirqd/300:05:093
53460995514,10cyclictest37-21ksoftirqd/320:17:013
53456995547,5cyclictest129386-21kworker/2:000:00:312
53456995547,5cyclictest105383-21kworker/2:300:40:022
53456995545,6cyclictest39308-21kworker/2:120:00:012
53456995537,13cyclictest57028-21systemctl20:20:012
53456995531,19cyclictest275-21dbus-daemon20:45:012
5344999557,17cyclictest105736-21ssh22:45:320
53449995527,12cyclictest84849-21ssh22:02:110
53449995525,26cyclictest19350irq/34-mmc121:20:020
53449995516,7cyclictest12-21ksoftirqd/021:28:120
53449995512,14cyclictest99051-21sh22:31:430
5346099547,11cyclictest37-21ksoftirqd/322:11:383
5346099546,8cyclictest37-21ksoftirqd/320:05:033
5346099546,7cyclictest37-21ksoftirqd/323:25:013
5346099546,10cyclictest37-21ksoftirqd/323:02:033
5346099545,12cyclictest37-21ksoftirqd/322:32:373
5346099545,11cyclictest37-21ksoftirqd/322:28:293
53460995415,11cyclictest37-21ksoftirqd/321:40:023
53456995446,5cyclictest105383-21kworker/2:322:50:512
53456995445,5cyclictest59385-21kworker/2:222:40:142
53456995445,5cyclictest105383-21kworker/2:323:54:592
53456995444,5cyclictest105383-21kworker/2:323:45:112
53456995444,5cyclictest105383-21kworker/2:323:45:112
53454995435,14cyclictest275-21dbus-daemon23:55:001
53449995419,17cyclictest16850irq/134-800000000:15:490
53449995412,10cyclictest78830-21cron21:50:000
5346099536,9cyclictest37-21ksoftirqd/321:55:563
5346099536,9cyclictest37-21ksoftirqd/300:23:213
5346099535,9cyclictest37-21ksoftirqd/322:49:473
53460995344,7cyclictest37-21ksoftirqd/323:10:303
53460995342,8cyclictest37-21ksoftirqd/300:26:023
53460995316,9cyclictest37-21ksoftirqd/320:10:013
53456995346,4cyclictest39308-21kworker/2:121:30:082
53456995345,5cyclictest59385-21kworker/2:221:57:052
53456995345,5cyclictest39308-21kworker/2:122:07:462
53456995345,5cyclictest39308-21kworker/2:122:02:052
53456995345,5cyclictest129386-21kworker/2:000:11:362
53456995345,5cyclictest129386-21kworker/2:000:11:362
53456995344,6cyclictest59385-21kworker/2:222:56:122
53456995344,6cyclictest59385-21kworker/2:222:56:122
53456995344,5cyclictest39308-21kworker/2:122:18:522
53456995344,5cyclictest39308-21kworker/2:121:37:282
53456995344,5cyclictest129386-21kworker/2:000:18:582
53456995344,5cyclictest105383-21kworker/2:300:22:542
53456995343,7cyclictest105383-21kworker/2:323:41:342
53449995313,9cyclictest12-21ksoftirqd/023:01:560
5346099528,11cyclictest37-21ksoftirqd/321:21:133
5346099527,8cyclictest37-21ksoftirqd/300:37:563
5346099527,7cyclictest37-21ksoftirqd/300:11:263
5346099527,7cyclictest37-21ksoftirqd/300:11:263
5346099526,9cyclictest37-21ksoftirqd/323:20:023
5346099526,9cyclictest37-21ksoftirqd/323:20:023
5346099526,8cyclictest37-21ksoftirqd/323:28:243
5346099526,8cyclictest37-21ksoftirqd/323:28:243
5346099526,8cyclictest37-21ksoftirqd/321:51:593
5346099526,11cyclictest37-21ksoftirqd/321:47:243
5346099525,12cyclictest37-21ksoftirqd/320:35:013
5346099524,10cyclictest37-21ksoftirqd/323:52:103
5346099523,9cyclictest37-21ksoftirqd/322:05:463
5346099522,10cyclictest37-21ksoftirqd/323:41:243
53456995245,4cyclictest129386-21kworker/2:000:09:222
53456995244,5cyclictest59385-21kworker/2:223:05:292
53456995244,5cyclictest105383-21kworker/2:323:15:552
53456995244,5cyclictest105383-21kworker/2:323:15:552
53456995243,6cyclictest59385-21kworker/2:222:27:152
53456995243,6cyclictest59385-21kworker/2:221:45:012
53456995243,6cyclictest39308-21kworker/2:120:47:372
53449995238,10cyclictest55945-21sshd19:57:130
53449995218,17cyclictest16850irq/134-800000023:38:410
5344999521,23cyclictest0-21swapper/020:51:280
5346099516,7cyclictest37-21ksoftirqd/321:26:153
5346099515,8cyclictest37-21ksoftirqd/321:10:133
5346099511,13cyclictest140417-21ssh23:57:433
53456995143,5cyclictest39308-21kworker/2:122:20:222
53456995143,5cyclictest129386-21kworker/2:023:55:362
53456995142,5cyclictest105383-21kworker/2:300:30:352
53456995141,6cyclictest59385-21kworker/2:221:25:012
53456995141,6cyclictest105383-21kworker/2:323:38:482
53456995141,6cyclictest105383-21kworker/2:300:27:472
53454995118,6cyclictest346771850lldpd23:24:201
5346099506,11cyclictest37-21ksoftirqd/323:37:103
5346099503,9cyclictest37-21ksoftirqd/323:06:453
53456995042,5cyclictest59385-21kworker/2:223:01:562
53456995040,6cyclictest59385-21kworker/2:221:27:012
53456995033,12cyclictest58062-21run-parts20:40:002
5344999506,14cyclictest54350-21latency_hist19:25:020
5346099497,7cyclictest37-21ksoftirqd/321:10:023
5346099495,15cyclictest37-21ksoftirqd/323:31:183
53460994937,10cyclictest37-21ksoftirqd/319:40:013
53456994941,5cyclictest59385-21kworker/2:221:52:322
53456994941,5cyclictest105383-21kworker/2:323:32:282
53456994941,5cyclictest105383-21kworker/2:323:10:222
53456994940,6cyclictest39308-21kworker/2:121:06:382
53456994940,6cyclictest39308-21kworker/2:119:46:392
53456994940,6cyclictest105383-21kworker/2:323:20:152
53456994939,6cyclictest39308-21kworker/2:120:50:022
53454994934,11cyclictest275-21dbus-daemon00:00:001
53460994834,10cyclictest37-21ksoftirqd/319:55:013
5346099482,12cyclictest37-21ksoftirqd/320:25:013
53460994810,10cyclictest37-21ksoftirqd/319:14:593
53456994840,5cyclictest39308-21kworker/2:119:45:002
53454994835,8cyclictest1-21systemd00:40:001
5344999489,32cyclictest59633-21munin-run21:10:010
5346099479,5cyclictest131rcu_preempt20:45:023
5346099477,8cyclictest37-21ksoftirqd/320:40:003
53456994735,7cyclictest54927-21cron19:40:002
53456994713,7cyclictest131rcu_preempt19:15:002
53454994728,13cyclictest58065-21systemctl20:40:001
5346099464,7cyclictest37-21ksoftirqd/319:20:013
5346099464,7cyclictest37-21ksoftirqd/319:20:013
53456994638,5cyclictest39308-21kworker/2:119:10:012
53454994636,6cyclictest59097-21sendmail21:00:011
53454994634,8cyclictest3467602-21systemd-journal20:13:381
53454994632,11cyclictest23-21ksoftirqd/122:13:471
53454994632,10cyclictest1-21systemd22:30:011
53454994628,13cyclictest3467602-21systemd-journal23:40:011
5345499461,10cyclictest53909-21systemctl19:20:001
5345499461,10cyclictest53909-21systemctl19:20:001
53449994632,10cyclictest1-21systemd19:35:010
53449994632,10cyclictest1-21systemd19:35:000
53449994631,11cyclictest55434-21sshd19:45:390
5344999461,39cyclictest57507-21cat20:25:020
53456994537,5cyclictest39308-21kworker/2:119:20:012
53456994537,5cyclictest39308-21kworker/2:119:20:012
53456994536,6cyclictest39308-21kworker/2:121:04:382
53456994534,7cyclictest39308-21kworker/2:119:30:012
5345499451,3cyclictest131rcu_preempt00:32:051
5345499451,3cyclictest131rcu_preempt00:15:461
5344999458,33cyclictest0-21swapper/021:00:020
5344999453,38cyclictest0-21swapper/021:02:250
5346099446,10cyclictest37-21ksoftirqd/320:55:023
53454994431,9cyclictest1-21systemd23:30:011
53454994431,9cyclictest1-21systemd23:30:011
53454994431,10cyclictest61384-21sshd21:13:381
5345499441,3cyclictest131rcu_preempt23:15:021
53449994430,11cyclictest56477-21sshd20:07:130
5344999442,6cyclictest56764-21true20:15:010
53460994330,9cyclictest309-21in:imuxsock20:50:003
53456994335,5cyclictest39308-21kworker/2:119:25:012
53454994332,8cyclictest100464-21sh22:35:011
53454994331,8cyclictest117369-21ssh23:10:011
53454994325,12cyclictest275-21dbus-daemon23:35:001
5345499431,14cyclictest79957-21ssh21:52:021
5344999439,4cyclictest53603-21sshd19:10:390
53449994336,3cyclictest0-21swapper/020:40:020
53449994328,10cyclictest57813-21munin-run20:35:000
53449994326,13cyclictest3467602-21systemd-journal20:41:020
5346099426,7cyclictest37-21ksoftirqd/319:24:593
5346099425,7cyclictest37-21ksoftirqd/319:50:033
5346099425,7cyclictest37-21ksoftirqd/319:10:013
53456994221,15cyclictest275-21dbus-daemon20:15:012
5345499425,20cyclictest52974-21kworker/u8:319:25:011
53454994225,11cyclictest23-21ksoftirqd/121:26:521
5345499422,4cyclictest131rcu_preempt21:16:261
53454994215,5cyclictest131rcu_preempt23:48:411
53454994215,5cyclictest131rcu_preempt23:48:411
5344999428,30cyclictest0-21swapper/020:48:380
53449994229,10cyclictest53875-21sshd19:17:380
53449994229,10cyclictest53875-21sshd19:17:370
53449994228,11cyclictest54407-21sshd19:27:380
53449994227,12cyclictest55194-21sshd19:42:380
53449994227,12cyclictest364-21snmpd20:27:520
53449994227,12cyclictest364-21snmpd20:27:520
5346099415,7cyclictest37-21ksoftirqd/321:05:013
5346099414,6cyclictest37-21ksoftirqd/320:30:003
5346099414,6cyclictest37-21ksoftirqd/320:30:003
53460994127,11cyclictest275-21dbus-daemon19:35:023
53460994127,11cyclictest275-21dbus-daemon19:35:023
5346099411,9cyclictest37-21ksoftirqd/319:30:023
5345499413,5cyclictest131rcu_preempt00:15:021
5345499413,5cyclictest131rcu_preempt00:15:021
53454994114,5cyclictest131rcu_preempt22:06:581
5345499411,3cyclictest131rcu_preempt21:25:011
5346099402,8cyclictest37-21ksoftirqd/320:59:383
53460994024,11cyclictest55382-21cat19:45:023
53456994025,11cyclictest30-21ksoftirqd/219:35:012
53456994025,11cyclictest30-21ksoftirqd/219:35:012
53454994030,7cyclictest74237-21sh21:40:031
5345499402,7cyclictest76514-21munin-run21:45:011
53454994024,13cyclictest23-21ksoftirqd/100:08:241
5345499402,3cyclictest131rcu_preempt22:15:051
5344999402,28cyclictest53453-21latency_hist19:10:010
53456993927,8cyclictest30-21ksoftirqd/220:30:022
53456993927,8cyclictest30-21ksoftirqd/220:30:022
53456993924,11cyclictest30-21ksoftirqd/219:54:592
5345499395,4cyclictest131rcu_preempt22:01:051
5345499393,5cyclictest0-21swapper/100:22:561
53454993928,9cyclictest23-21ksoftirqd/122:51:041
53454993926,10cyclictest23-21ksoftirqd/120:05:031
53454993925,10cyclictest55136-21sed19:40:021
53454993912,6cyclictest131rcu_preempt23:41:291
53454993910,6cyclictest131rcu_preempt23:01:511
53456993823,12cyclictest30-21ksoftirqd/220:05:012
53456993819,13cyclictest1-21systemd20:35:002
53454993828,6cyclictest95703-21cron22:25:011
53454993825,9cyclictest23-21ksoftirqd/122:45:271
53454993825,8cyclictest57437-21grep20:25:021
53454993822,12cyclictest275-21dbus-daemon20:55:011
53454993820,13cyclictest275-21dbus-daemon19:55:011
5345499381,3cyclictest131rcu_preempt19:10:011
53454993811,6cyclictest131rcu_preempt22:37:071
5345499381,15cyclictest111033-21ssh22:56:421
5345499381,15cyclictest111033-21ssh22:56:421
5345499373,5cyclictest131rcu_preempt20:00:001
53454993725,9cyclictest53689-21sed19:15:011
53454993725,9cyclictest23-21ksoftirqd/121:30:151
53454993725,7cyclictest309-21in:imuxsock22:40:021
53454993725,7cyclictest309-21in:imuxsock19:50:011
5345699369,5cyclictest131rcu_preempt20:20:312
53454993626,7cyclictest56588-21sed20:10:011
53454993626,6cyclictest23-21ksoftirqd/100:30:021
53454993625,9cyclictest23-21ksoftirqd/121:58:311
53454993625,9cyclictest23-21ksoftirqd/121:45:241
53454993624,10cyclictest23-21ksoftirqd/120:35:001
5345499356,5cyclictest131rcu_preempt20:45:001
5345499352,7cyclictest120818-21ssh23:16:591
5345499352,7cyclictest120818-21ssh23:16:591
5345499352,3cyclictest131rcu_preempt21:10:011
53456993421,9cyclictest275-21dbus-daemon20:10:012
5345499342,4cyclictest131rcu_preempt20:47:381
53454993423,8cyclictest23-21ksoftirqd/120:30:011
53454993423,8cyclictest23-21ksoftirqd/120:30:001
53454993423,8cyclictest23-21ksoftirqd/119:30:021
53454993322,8cyclictest3467602-21systemd-journal19:30:321
53454993322,8cyclictest3467602-21systemd-journal19:30:311
53454993322,8cyclictest23-21ksoftirqd/120:20:011
5345499321,28cyclictest48-21kcompactd021:00:241
53454993019,8cyclictest55187-21sshd19:41:381
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional