You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-15 - 04:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Jul 15, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3626965999527,14cyclictest3679811-21ssh22:48:070
3626965999447,42cyclictest16950irq/224-800000000:05:000
3626965998928,12cyclictest3732365-21ssh00:38:170
3626965998827,6cyclictest3728402-21unin-run00:30:010
3626965998427,13cyclictest3723141-21ssh00:18:560
362696599831,36cyclictest3647096-21sshd21:39:200
362696599823,6cyclictest3683257-21latency_hist22:55:020
3626965998028,15cyclictest3684588-21ssh22:58:060
3626971997972,5cyclictest30-21ksoftirqd/219:10:022
3626965997934,41cyclictest16950irq/224-800000023:25:010
3626965997921,12cyclictest3673844-21ssh22:35:100
3626965997919,17cyclictest16850irq/134-800000000:08:010
3626965997829,11cyclictest3636812-21ssh21:17:240
3626965997828,16cyclictest3648632-21ssh21:42:210
3626965997822,12cyclictest3656851-21ssh21:59:530
3626965997728,7cyclictest3685813-21ssh23:00:180
3626965997728,7cyclictest3685813-21ssh23:00:180
3626965997719,18cyclictest16850irq/134-800000022:10:170
3626965997719,18cyclictest16850irq/134-800000022:10:160
362696599771,35cyclictest3709613-21ssh23:50:240
3626965997628,14cyclictest3689479-21ssh23:08:230
3626965997626,15cyclictest3676209-21ssh22:40:100
362696599761,38cyclictest3660392-21ssh22:07:080
362696599752,67cyclictest3713993-40mandb00:00:010
3626965997517,17cyclictest16850irq/134-800000022:19:280
362696599751,40cyclictest3650734-21ssh21:46:440
3626965997424,12cyclictest3671844-21ssh22:31:080
3626965997419,17cyclictest16850irq/134-800000000:33:540
3626965997418,17cyclictest16850irq/134-800000023:47:180
3626965997418,17cyclictest16850irq/134-800000023:47:180
362697199736,5cyclictest131rcu_preempt23:40:022
362697199736,5cyclictest131rcu_preempt23:40:022
3626965997327,14cyclictest3698373-21ssh23:26:550
3626965997220,13cyclictest3702505-21ssh23:35:280
3626965997220,13cyclictest3702505-21ssh23:35:280
3626965997216,17cyclictest16850irq/134-800000023:41:210
3626965997115,18cyclictest16850irq/134-800000022:20:080
3626971997025,41cyclictest3710090-21sshd23:51:332
3626965997047,20cyclictest16950irq/224-800000019:20:000
3626965997035,30cyclictest16850irq/134-800000021:55:010
362696599702,34cyclictest3670583-21ssh22:28:370
3626965997017,17cyclictest16850irq/134-800000000:20:210
362697199692,14cyclictest3629246-21munin-run19:55:022
3626965996932,11cyclictest3692615-21latency_hist23:15:020
3626965996918,17cyclictest16850irq/134-800000021:34:390
3626965996915,5cyclictest16850irq/134-800000021:28:310
362697499683,7cyclictest37-21ksoftirqd/321:25:013
3626971996823,38cyclictest309-21in:imuxsock22:15:002
3626971996823,38cyclictest309-21in:imuxsock22:15:002
3626971996720,41cyclictest3649985-21sh21:45:022
3626969996746,12cyclictest23-21ksoftirqd/119:55:021
3626974996634,21cyclictest37-21ksoftirqd/319:55:033
3626971996625,37cyclictest3644076-21sh21:32:502
362697199662,17cyclictest3728388-21munin-run00:30:002
3626971996527,34cyclictest30-21ksoftirqd/223:43:582
3626965996530,4cyclictest16950irq/224-800000000:10:320
362696599651,40cyclictest3635573-21grep21:15:020
362696599651,40cyclictest3635573-21grep21:15:010
362697199642,12cyclictest3627960-21munin-run19:30:012
362697199642,10cyclictest3673192-21ssh22:34:162
3626971996417,43cyclictest3467602-21systemd-journal19:50:022
362697199641,13cyclictest3670105-21ssh22:27:372
362697199641,11cyclictest3709064-21ssh23:49:352
362697199641,11cyclictest3709064-21ssh23:49:352
3626965996427,6cyclictest3467602-21systemd-journal22:03:200
3626965996427,6cyclictest3467602-21systemd-journal22:03:200
3626974996320,12cyclictest37-21ksoftirqd/323:20:013
3626974996316,6cyclictest131rcu_preempt21:20:013
3626971996334,24cyclictest1-21systemd00:15:012
3626971996329,30cyclictest3630809-21latency_hist20:25:022
3626971996325,34cyclictest3732782-21sh00:39:092
3626971996323,36cyclictest3693605-21sh23:16:522
362697199632,14cyclictest3630572-21munin-run20:20:012
362697199632,14cyclictest3630572-21munin-run20:20:012
362697199622,17cyclictest3679880-21ssh22:48:152
362697199622,15cyclictest3675940-21munin-run22:40:022
362697199622,15cyclictest3628420-21sshd19:36:192
362697199622,14cyclictest3628742-21/usr/sbin/munin19:45:002
362697199622,11cyclictest3696574-21sh23:23:092
362697199621,16cyclictest1-21systemd23:59:592
362697199621,14cyclictest3682662-21ssh22:54:112
362697199621,11cyclictest3730587-21ssh00:34:352
362696999624,11cyclictest23-21ksoftirqd/121:15:011
362696999624,11cyclictest23-21ksoftirqd/121:15:001
362696599621,4cyclictest3633166-21munin-run21:10:010
3626974996148,5cyclictest0-21swapper/300:25:033
3626974996111,6cyclictest131rcu_preempt22:26:173
362697199612,13cyclictest48-21kcompactd021:40:182
362697199612,11cyclictest3640165-21sh21:24:472
362697199611,17cyclictest3635526-21cron21:15:002
362697199611,17cyclictest3635526-21cron21:15:002
362697199611,12cyclictest3676143-21ssh22:40:032
3626971996110,44cyclictest3627715-21cat19:25:022
362696999613,8cyclictest23-21ksoftirqd/122:00:011
3626965996132,5cyclictest3695088-21latency_hist23:20:020
3626965996127,7cyclictest3700256-21ssh23:30:420
3626971996051,6cyclictest3628222-21kworker/2:320:30:012
362697199603,13cyclictest3631606-21munin-run20:40:012
3626971996026,30cyclictest3467602-21systemd-journal21:09:202
362697199601,14cyclictest364-21snmpd20:44:132
362697199601,12cyclictest3699716-21munin-run23:30:012
3626971995947,8cyclictest3628222-21kworker/2:321:05:002
362697199592,16cyclictest3467602-21systemd-journal23:32:382
362697199592,13cyclictest364-21snmpd20:10:532
362697199592,13cyclictest3631327-21munin-run20:35:022
362697199592,10cyclictest120-21kswapd022:05:012
362697199592,10cyclictest120-21kswapd022:05:002
362697199591,15cyclictest3690657-21ssh23:10:352
362696999598,5cyclictest131rcu_preempt22:05:021
362696999598,5cyclictest131rcu_preempt22:05:021
3626971995822,32cyclictest30-21ksoftirqd/221:00:012
3626971995822,32cyclictest30-21ksoftirqd/221:00:012
3626971995817,37cyclictest30-21ksoftirqd/222:17:522
362697199581,13cyclictest3627370-21modprobe19:15:182
362697499575,9cyclictest37-21ksoftirqd/300:20:013
362697199572,17cyclictest3684669-21ssh22:58:162
362697199572,11cyclictest48-21kcompactd023:04:572
362697199572,11cyclictest48-21kcompactd023:04:572
3626971995716,36cyclictest3655817-21sh21:57:232
362697199571,15cyclictest3721888-21rm00:16:032
362697199571,14cyclictest364-21snmpd19:58:352
362697199571,14cyclictest3632327-21sshd20:52:202
362697199571,11cyclictest3717752-21ssh00:07:372
3626965995726,9cyclictest12-21ksoftirqd/020:55:020
362697499569,6cyclictest131rcu_preempt22:20:023
362697499568,6cyclictest131rcu_preempt22:31:313
362697499566,10cyclictest37-21ksoftirqd/319:15:023
362697499564,7cyclictest37-21ksoftirqd/300:05:023
3626974995639,13cyclictest1-21systemd20:10:013
3626974995615,6cyclictest131rcu_preempt23:30:353
362697199563,7cyclictest364-21snmpd19:33:202
3626971995631,21cyclictest30-21ksoftirqd/200:20:522
362697199562,7cyclictest3647357-21cron21:40:002
362697199562,11cyclictest3715028-21ssh00:01:462
362697199562,10cyclictest3636088-21sh21:15:432
3626971995616,36cyclictest30-21ksoftirqd/221:27:312
362697199561,15cyclictest3660072-21ssh22:06:282
362697199561,14cyclictest3630117-21latency_hist20:10:022
362696999568,6cyclictest131rcu_preempt21:55:021
362697499556,11cyclictest37-21ksoftirqd/323:21:053
362697499553,22cyclictest37-21ksoftirqd/321:15:013
362697499553,22cyclictest37-21ksoftirqd/321:15:003
362697499552,8cyclictest37-21ksoftirqd/323:55:013
362697199552,12cyclictest3688179-21sh23:05:202
362697199551,14cyclictest3668069-21ssh22:23:232
362697199551,10cyclictest3632077-21modprobe20:48:192
3626969995513,5cyclictest131rcu_preempt22:47:171
3626969995512,5cyclictest131rcu_preempt23:30:231
362697499548,5cyclictest131rcu_preempt23:37:333
362697499548,5cyclictest131rcu_preempt23:37:333
362697499547,8cyclictest37-21ksoftirqd/322:04:463
362697499547,8cyclictest37-21ksoftirqd/322:04:453
362697499546,6cyclictest131rcu_preempt21:50:503
3626974995422,12cyclictest37-21ksoftirqd/322:55:293
3626974995415,8cyclictest131rcu_preempt21:39:013
3626974995415,5cyclictest131rcu_preempt22:15:033
362697199548,7cyclictest3453636-1kworker/2:1H20:01:502
362697199542,18cyclictest3653845-21ssh21:53:242
362696999548,8cyclictest23-21ksoftirqd/100:39:201
362696999547,12cyclictest23-21ksoftirqd/100:00:021
362696999545,8cyclictest23-21ksoftirqd/121:21:141
3626969995411,6cyclictest131rcu_preempt22:26:021
362697499537,6cyclictest131rcu_preempt23:47:263
362697499537,6cyclictest131rcu_preempt23:47:263
362697499536,9cyclictest37-21ksoftirqd/322:42:493
362697499536,10cyclictest37-21ksoftirqd/323:05:013
362697499536,10cyclictest37-21ksoftirqd/323:05:013
3626974995339,11cyclictest37-21ksoftirqd/323:41:583
362697499533,8cyclictest37-21ksoftirqd/320:20:013
362697499533,8cyclictest37-21ksoftirqd/320:20:003
3626974995313,6cyclictest131rcu_preempt22:15:023
3626974995310,6cyclictest131rcu_preempt00:31:553
362696999538,4cyclictest131rcu_preempt23:05:021
362696999538,4cyclictest131rcu_preempt23:05:011
362696999534,11cyclictest23-21ksoftirqd/121:50:011
3626969995314,6cyclictest131rcu_preempt00:33:241
362697499529,7cyclictest131rcu_preempt22:06:353
362697499526,5cyclictest131rcu_preempt21:49:183
362697499524,9cyclictest37-21ksoftirqd/300:15:023
362697499523,9cyclictest37-21ksoftirqd/322:55:023
362697499522,13cyclictest37-21ksoftirqd/320:00:003
3626974995210,5cyclictest131rcu_preempt00:39:133
3626971995212,35cyclictest275-21dbus-daemon19:15:012
362696999528,6cyclictest23-21ksoftirqd/123:16:441
362696999527,10cyclictest23-21ksoftirqd/121:20:011
362696999526,10cyclictest23-21ksoftirqd/122:50:231
3626969995236,12cyclictest1-21systemd19:40:001
3626969995212,6cyclictest131rcu_preempt23:07:131
362697499518,5cyclictest131rcu_preempt21:29:473
362697499517,7cyclictest37-21ksoftirqd/319:30:013
362697499516,5cyclictest131rcu_preempt23:09:273
362697499514,6cyclictest131rcu_preempt22:39:413
3626974995114,6cyclictest131rcu_preempt21:55:323
3626974995114,5cyclictest131rcu_preempt22:47:243
362696999518,7cyclictest23-21ksoftirqd/100:05:011
362696999516,9cyclictest23-21ksoftirqd/123:49:321
362696999516,9cyclictest23-21ksoftirqd/123:49:321
362696999516,8cyclictest23-21ksoftirqd/121:00:021
362696999516,8cyclictest23-21ksoftirqd/121:00:021
362696999515,8cyclictest23-21ksoftirqd/121:28:211
362696999512,6cyclictest131rcu_preempt22:44:011
3626969995112,6cyclictest131rcu_preempt00:26:151
3626969995111,7cyclictest131rcu_preempt00:20:011
362697499509,5cyclictest131rcu_preempt20:40:023
362697499509,4cyclictest131rcu_preempt00:06:533
362697499508,6cyclictest131rcu_preempt23:55:423
362697499505,7cyclictest37-21ksoftirqd/321:31:243
362697499502,11cyclictest37-21ksoftirqd/319:45:023
3626974995013,6cyclictest131rcu_preempt19:40:003
362696999508,9cyclictest23-21ksoftirqd/122:30:061
3626969995039,8cyclictest23-21ksoftirqd/100:10:011
3626969995034,11cyclictest1-21systemd21:35:011
3626969995019,17cyclictest23-21ksoftirqd/123:22:471
3626969995015,6cyclictest131rcu_preempt22:18:471
3626969995013,6cyclictest131rcu_preempt22:35:081
3626969995012,6cyclictest131rcu_preempt22:20:521
3626965995042,4cyclictest16950irq/224-800000020:40:000
362697499494,5cyclictest131rcu_preempt00:25:463
362697499492,9cyclictest37-21ksoftirqd/323:15:023
3626974994919,5cyclictest131rcu_preempt22:23:253
3626974994913,5cyclictest131rcu_preempt23:26:203
362696999498,8cyclictest23-21ksoftirqd/121:40:371
362696999498,5cyclictest131rcu_preempt00:12:001
362696999497,8cyclictest23-21ksoftirqd/123:25:371
362696999497,5cyclictest131rcu_preempt22:06:151
362696999497,10cyclictest23-21ksoftirqd/123:51:491
362696999495,10cyclictest23-21ksoftirqd/123:40:021
362696999495,10cyclictest23-21ksoftirqd/123:40:021
3626969994937,8cyclictest3467562-21sshd23:10:321
362696999493,6cyclictest131rcu_preempt22:11:291
362696999493,6cyclictest131rcu_preempt22:11:291
3626969994913,7cyclictest131rcu_preempt22:56:141
3626969994912,5cyclictest131rcu_preempt00:22:531
3626969994910,6cyclictest131rcu_preempt23:44:041
3626969994910,5cyclictest131rcu_preempt20:20:011
3626969994910,5cyclictest131rcu_preempt20:20:011
3626965994923,11cyclictest3639638-21ssh21:23:270
362697499489,10cyclictest37-21ksoftirqd/321:44:493
362696599484,3cyclictest131rcu_preempt19:45:020
362697499479,5cyclictest131rcu_preempt21:10:023
3626974994732,12cyclictest275-21dbus-daemon19:35:013
3626974994710,5cyclictest131rcu_preempt20:25:013
362696999475,10cyclictest23-21ksoftirqd/119:30:021
362696999474,7cyclictest23-21ksoftirqd/119:50:021
362696999471,8cyclictest3633136-21diskmemload21:10:011
362696999471,7cyclictest23-21ksoftirqd/119:35:031
3626969994712,5cyclictest131rcu_preempt21:35:471
3626969994634,8cyclictest221rcuc/120:10:021
362697499454,10cyclictest37-21ksoftirqd/320:41:013
3626974994511,5cyclictest131rcu_preempt19:20:013
362697499448,5cyclictest131rcu_preempt20:35:023
362697499445,5cyclictest131rcu_preempt21:00:023
362697499445,5cyclictest131rcu_preempt21:00:023
362697499441,14cyclictest37-21ksoftirqd/319:50:023
362696999445,7cyclictest23-21ksoftirqd/120:15:021
362696999441,17cyclictest0-21swapper/119:21:481
3626969994410,5cyclictest131rcu_preempt19:20:001
3626965994430,9cyclictest12-21ksoftirqd/019:15:010
3626965994427,12cyclictest275-21dbus-daemon20:15:010
3626965994420,20cyclictest12-21ksoftirqd/019:50:010
3626974994311,5cyclictest131rcu_preempt20:54:593
362696999439,5cyclictest131rcu_preempt19:12:191
362697499427,7cyclictest37-21ksoftirqd/321:05:003
362697499421,7cyclictest0-21swapper/320:15:003
362696999428,5cyclictest131rcu_preempt20:30:021
362696999428,5cyclictest131rcu_preempt20:05:031
362696999425,8cyclictest23-21ksoftirqd/120:25:011
3626969994232,8cyclictest221rcuc/120:30:031
3626965994226,12cyclictest3630534-21cat20:20:010
3626965994226,12cyclictest3630534-21cat20:20:010
3626965994226,10cyclictest12-21ksoftirqd/020:00:010
362696999418,5cyclictest131rcu_preempt20:35:201
3626969994111,5cyclictest131rcu_preempt21:03:301
362696599416,3cyclictest131rcu_preempt20:45:000
362696599413,33cyclictest0-21swapper/020:35:010
3626965994127,11cyclictest3627656-21sshd19:20:190
362696599412,6cyclictest3628459-21systemctl19:40:010
3626965994122,13cyclictest3467602-21systemd-journal21:00:010
3626965994122,13cyclictest3467602-21systemd-journal21:00:000
362696999401,4cyclictest131rcu_preempt20:00:011
3626965994032,5cyclictest3630807-21sh20:24:590
362697499399,4cyclictest131rcu_preempt20:50:013
3626965993933,4cyclictest16950irq/224-800000019:28:190
3626965993925,9cyclictest12-21ksoftirqd/019:35:020
3626965993923,12cyclictest3629984-21sshd20:05:190
362696999386,7cyclictest23-21ksoftirqd/120:47:121
362696999384,4cyclictest131rcu_preempt20:53:201
3626969993827,8cyclictest221rcuc/119:42:391
3626965993728,6cyclictest19350irq/34-mmc121:00:060
3626965993725,9cyclictest12-21ksoftirqd/020:04:480
3626965993719,14cyclictest3631031-21sshd20:25:570
3626974993624,9cyclictest37-21ksoftirqd/320:30:003
362697499361,8cyclictest37-21ksoftirqd/319:20:413
3626965993622,10cyclictest12-21ksoftirqd/019:55:020
3626974993513,5cyclictest131rcu_preempt20:05:023
3626965993518,13cyclictest3632069-21sshd20:46:210
362696999341,14cyclictest0-21swapper/120:43:451
3626965993426,6cyclictest12-21ksoftirqd/019:10:020
3626969993326,4cyclictest3626896-21latency_hist19:10:021
3626974993221,7cyclictest37-21ksoftirqd/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional