You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-24 - 06:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun May 24, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
369028699132125,4cyclictest3742277-21sh22:45:020
3690303999990,6cyclictest37-21ksoftirqd/300:30:023
3690303999990,6cyclictest37-21ksoftirqd/300:30:013
3690286998734,15cyclictest16850irq/134-800000021:40:010
369028699841,40cyclictest3774630-21ssh23:52:530
3690298997722,30cyclictest0-21swapper/220:00:002
3690286997328,15cyclictest3723197-21ssh22:05:090
3690298997120,11cyclictest3778345-40gzip00:00:262
3690298996717,11cyclictest30-21ksoftirqd/222:05:022
3690286996627,6cyclictest131rcu_preempt00:00:060
3690298996520,9cyclictest30-21ksoftirqd/222:51:512
3690298996512,14cyclictest3756842-21ssh23:15:362
369029899646,21cyclictest3712318-21ssh21:42:432
3690298996411,20cyclictest3775597-21systemd-run23:55:002
3690303996318,8cyclictest37-21ksoftirqd/300:16:173
3690298996318,10cyclictest30-21ksoftirqd/200:23:202
3690298996317,13cyclictest3784253-21ssh00:13:032
3690298996315,10cyclictest30-21ksoftirqd/221:38:272
3690298996314,15cyclictest3719836-21ssh21:58:312
369029899628,17cyclictest3754815-21ssh23:11:202
369029899628,16cyclictest3714119-21ssh21:46:232
3690298996216,8cyclictest30-21ksoftirqd/200:17:392
3690294996232,9cyclictest23-21ksoftirqd/100:28:201
3690294996232,9cyclictest23-21ksoftirqd/100:28:191
3690298996118,7cyclictest30-21ksoftirqd/200:38:552
3690298996115,9cyclictest30-21ksoftirqd/221:15:032
3690298996115,9cyclictest30-21ksoftirqd/221:15:022
3690298996114,17cyclictest3777658-21ssh23:59:242
3690298996113,10cyclictest30-21ksoftirqd/222:59:012
3690298996112,10cyclictest30-21ksoftirqd/223:49:282
369028699612,55cyclictest3768411-21ssh23:40:000
3690298996019,9cyclictest30-21ksoftirqd/223:02:382
3690298996016,11cyclictest30-21ksoftirqd/222:39:212
3690298996015,10cyclictest30-21ksoftirqd/223:37:172
3690298996015,10cyclictest30-21ksoftirqd/221:50:032
3690298996012,17cyclictest3764001-21ssh23:30:272
3690294996033,8cyclictest23-21ksoftirqd/122:35:491
369028699603,4cyclictest131rcu_preempt22:55:010
3690286996022,22cyclictest0-21swapper/021:20:450
369030399595,14cyclictest37-21ksoftirqd/321:49:593
369029899596,15cyclictest3724717-21ssh22:08:392
3690298995916,11cyclictest30-21ksoftirqd/223:42:302
3690298995916,10cyclictest30-21ksoftirqd/222:32:152
3690298995915,9cyclictest30-21ksoftirqd/200:30:232
3690298995913,14cyclictest3733715-21ssh22:27:162
3690298995913,14cyclictest3733715-21ssh22:27:152
3690286995949,5cyclictest16950irq/224-800000000:10:000
3690286995929,12cyclictest3796614-21modprobe00:39:070
3690286995923,31cyclictest0-21swapper/000:05:000
369030399585,7cyclictest37-21ksoftirqd/321:20:013
3690303995818,9cyclictest37-21ksoftirqd/322:05:023
3690303995815,11cyclictest37-21ksoftirqd/322:55:013
3690303995814,11cyclictest37-21ksoftirqd/321:39:593
3690298995817,9cyclictest30-21ksoftirqd/220:50:022
3690298995815,8cyclictest30-21ksoftirqd/223:07:102
3690286995828,15cyclictest3740757-21ssh22:41:550
3690286995816,24cyclictest3701281-21ssh21:20:000
369029899578,14cyclictest3727416-21ssh22:14:202
369029899576,15cyclictest3694227-21latency_hist20:25:012
369029899575,16cyclictest3761440-21ssh23:25:042
3690298995718,6cyclictest30-21ksoftirqd/221:15:142
3690298995715,10cyclictest30-21ksoftirqd/221:20:182
3690298995714,9cyclictest30-21ksoftirqd/200:25:342
3690298995714,9cyclictest30-21ksoftirqd/200:25:332
3690286995728,21cyclictest12-21ksoftirqd/023:05:000
3690303995647,6cyclictest361rcuc/322:19:353
3690298995615,12cyclictest30-21ksoftirqd/223:25:022
3690298995612,14cyclictest3706611-21sh21:30:352
3690298995612,14cyclictest3706611-21sh21:30:352
369029899561,14cyclictest0-21swapper/219:40:022
3690294995630,4cyclictest3713931-21ssh21:46:011
369028699563,3cyclictest131rcu_preempt22:25:500
369028699563,3cyclictest131rcu_preempt22:25:490
3690303995517,10cyclictest37-21ksoftirqd/322:57:433
3690303995516,8cyclictest37-21ksoftirqd/321:31:403
3690303995516,8cyclictest37-21ksoftirqd/321:31:393
3690298995545,6cyclictest3690056-21kworker/2:222:18:012
369028699552,7cyclictest3720491-21munin-run22:00:000
3690286995527,12cyclictest3749338-21ssh23:00:030
3690286995526,14cyclictest3772053-21ssh23:47:280
369030399546,8cyclictest37-21ksoftirqd/322:46:113
369030399545,7cyclictest37-21ksoftirqd/321:23:253
369030399544,9cyclictest37-21ksoftirqd/323:47:123
3690303995410,8cyclictest37-21ksoftirqd/322:00:013
3690303995410,7cyclictest37-21ksoftirqd/300:00:053
3690303995410,10cyclictest37-21ksoftirqd/321:44:563
369029899549,15cyclictest3744085-21ssh22:49:052
3690294995417,11cyclictest23-21ksoftirqd/100:16:001
3690294995410,9cyclictest23-21ksoftirqd/123:43:041
369028699543,7cyclictest284-21systemd-logind23:15:010
3690286995432,16cyclictest275-21dbus-daemon21:15:010
3690286995432,16cyclictest275-21dbus-daemon21:15:010
3690286995427,23cyclictest3770600-21ssh23:44:410
369028699542,20cyclictest3718314-21latency_hist21:55:030
3690286995420,30cyclictest3789843-21cron00:25:000
369030399536,8cyclictest37-21ksoftirqd/300:35:073
369030399536,10cyclictest37-21ksoftirqd/300:33:203
369030399535,12cyclictest37-21ksoftirqd/300:03:513
369029899537,15cyclictest3704925-21rm21:27:122
3690286995327,22cyclictest12-21ksoftirqd/022:11:510
3690286995321,3cyclictest131rcu_preempt23:22:090
3690286995320,8cyclictest12-21ksoftirqd/022:04:350
369030399525,6cyclictest37-21ksoftirqd/322:08:213
369029899525,15cyclictest3782812-21wc00:10:012
3690298995214,7cyclictest3692723-21cat19:55:022
3690294995217,6cyclictest131rcu_preempt22:30:031
3690294995217,6cyclictest131rcu_preempt22:30:021
369028699524,42cyclictest311-21rs:main0
369028699522,37cyclictest3467602-21systemd-journal22:16:280
369030399516,8cyclictest37-21ksoftirqd/323:17:073
369030399516,8cyclictest37-21ksoftirqd/319:35:033
369030399515,11cyclictest37-21ksoftirqd/322:23:163
369030399515,11cyclictest37-21ksoftirqd/321:12:253
369030399515,11cyclictest37-21ksoftirqd/321:12:243
369030399514,9cyclictest37-21ksoftirqd/320:45:023
3690303995112,8cyclictest37-21ksoftirqd/323:30:163
369029899515,13cyclictest3731577-21ssh22:22:552
3690294995142,7cyclictest23-21ksoftirqd/121:56:031
369029499513,11cyclictest23-21ksoftirqd/100:40:001
3690294995116,9cyclictest23-21ksoftirqd/123:37:311
3690286995147,2cyclictest0-21swapper/021:10:020
3690286995140,7cyclictest16950irq/224-800000000:19:590
3690286995134,11cyclictest3467602-21systemd-journal21:00:000
3690286995131,16cyclictest3731674-21ssh22:23:070
3690286995131,16cyclictest16850irq/134-800000019:12:180
3690286995127,9cyclictest3784799-21ssh00:14:230
369028699512,35cyclictest3791352-21ssh00:28:000
369028699512,35cyclictest3791352-21ssh00:27:590
369028699511,35cyclictest3712231-21ssh21:42:330
369030399508,10cyclictest0-21swapper/322:15:003
369030399505,8cyclictest37-21ksoftirqd/322:39:303
369030399505,7cyclictest37-21ksoftirqd/322:27:203
369030399505,7cyclictest37-21ksoftirqd/322:27:203
369030399505,5cyclictest131rcu_preempt21:52:053
369030399502,7cyclictest37-21ksoftirqd/300:05:173
369030399501,14cyclictest37-21ksoftirqd/323:55:003
369029899507,14cyclictest3696470-21diskmemload22:41:092
369029499506,5cyclictest131rcu_preempt22:50:031
3690294995041,7cyclictest221rcuc/122:00:031
3690294995015,8cyclictest23-21ksoftirqd/100:00:061
3690294995010,8cyclictest23-21ksoftirqd/121:38:001
369030399497,7cyclictest37-21ksoftirqd/323:21:403
369030399495,6cyclictest37-21ksoftirqd/323:27:593
369030399494,10cyclictest37-21ksoftirqd/323:07:453
3690303994910,5cyclictest131rcu_preempt00:12:383
3690298994930,14cyclictest3467602-21systemd-journal20:20:002
369029499496,11cyclictest23-21ksoftirqd/122:23:371
369029499492,6cyclictest131rcu_preempt00:00:031
3690294994921,21cyclictest0-21swapper/123:20:031
3690294994914,8cyclictest23-21ksoftirqd/121:40:021
3690286994928,5cyclictest0-21swapper/019:27:210
3690286994925,21cyclictest12-21ksoftirqd/022:36:590
3690286994923,11cyclictest3756948-21ssh23:15:500
369030399489,8cyclictest0-21swapper/320:22:173
369030399487,10cyclictest37-21ksoftirqd/323:44:293
369030399486,13cyclictest37-21ksoftirqd/323:04:273
369030399486,12cyclictest37-21ksoftirqd/323:35:433
369030399486,10cyclictest37-21ksoftirqd/321:25:523
369030399484,12cyclictest37-21ksoftirqd/322:31:453
369030399484,10cyclictest37-21ksoftirqd/319:50:013
3690303994833,11cyclictest3467602-21systemd-journal19:25:003
369030399481,10cyclictest3741424-21ssh22:43:263
3690298994833,11cyclictest3467602-21systemd-journal20:15:002
369029499483,9cyclictest23-21ksoftirqd/121:27:291
3690294994839,6cyclictest221rcuc/122:16:021
3690294994837,7cyclictest3701284-21cron21:19:591
369029499483,29cyclictest23-21ksoftirqd/121:05:021
3690294994820,6cyclictest131rcu_preempt00:08:421
3690294994817,22cyclictest221rcuc/100:14:211
3690294994814,6cyclictest131rcu_preempt23:10:471
3690294994812,9cyclictest23-21ksoftirqd/122:06:581
3690294994810,6cyclictest131rcu_preempt23:02:351
369028699482,42cyclictest3708473-21munin-run21:35:000
369028699482,42cyclictest3708473-21munin-run21:35:000
369030399476,5cyclictest131rcu_preempt23:14:053
369030399474,9cyclictest37-21ksoftirqd/321:10:003
369029499477,5cyclictest131rcu_preempt23:47:331
3690294994737,7cyclictest23-21ksoftirqd/120:00:001
3690294994736,8cyclictest23-21ksoftirqd/121:22:411
3690294994728,13cyclictest23-21ksoftirqd/119:20:001
3690294994713,11cyclictest23-21ksoftirqd/122:31:331
369028699474,4cyclictest131rcu_preempt23:30:130
3690286994728,15cyclictest3751935-21ssh23:05:140
369028699472,40cyclictest3706196-21cat21:30:020
369028699471,40cyclictest3693413-21latency_hist20:10:010
3690303994631,11cyclictest275-21dbus-daemon21:00:003
3690303994616,8cyclictest37-21ksoftirqd/300:21:203
369029899467,7cyclictest131rcu_preempt20:05:022
369029899467,7cyclictest131rcu_preempt20:05:012
3690298994635,6cyclictest3694648-21munin-plugin-st20:34:592
369029499465,8cyclictest23-21ksoftirqd/123:50:321
369029499464,6cyclictest131rcu_preempt20:25:021
3690294994637,5cyclictest23-21ksoftirqd/123:09:201
369029499462,11cyclictest23-21ksoftirqd/123:21:491
369029499461,7cyclictest0-21swapper/122:50:111
369028699462,39cyclictest3694893-21modprobe20:40:000
369028699462,39cyclictest3694893-21modprobe20:40:000
369028699461,24cyclictest131rcu_preempt00:32:190
369030399459,8cyclictest0-21swapper/319:25:283
3690303994529,12cyclictest275-21dbus-daemon21:05:023
3690303994529,12cyclictest275-21dbus-daemon21:05:013
369029499457,7cyclictest23-21ksoftirqd/121:31:061
369029499457,7cyclictest23-21ksoftirqd/121:31:051
369029499457,6cyclictest131rcu_preempt00:32:151
3690294994534,8cyclictest23-21ksoftirqd/123:32:061
3690294994533,9cyclictest23-21ksoftirqd/123:00:011
369029499451,12cyclictest23-21ksoftirqd/121:11:271
369029499451,12cyclictest23-21ksoftirqd/121:11:271
3690286994528,14cyclictest3761750-21ssh23:25:440
3690286994523,10cyclictest3713851-21ssh21:45:500
369030399449,6cyclictest0-21swapper/320:54:583
3690303994433,8cyclictest37-21ksoftirqd/320:50:023
369030399441,4cyclictest131rcu_preempt20:00:013
369030399441,10cyclictest37-21ksoftirqd/320:10:053
3690298994431,10cyclictest30-21ksoftirqd/221:00:022
3690298994427,14cyclictest3690465-21sshd19:13:062
3690298994427,12cyclictest1-21systemd19:45:002
3690298994426,12cyclictest309-21in:imuxsock20:55:012
3690298994420,17cyclictest275-21dbus-daemon20:40:012
3690298994420,17cyclictest275-21dbus-daemon20:40:002
369029499443,5cyclictest131rcu_preempt00:21:281
3690294994435,7cyclictest23-21ksoftirqd/120:46:051
3690294994432,8cyclictest23-21ksoftirqd/120:20:001
369029499441,8cyclictest0-21swapper/119:45:361
369029499441,6cyclictest0-21swapper/120:27:441
369029499441,6cyclictest0-21swapper/120:27:441
3690294994414,6cyclictest131rcu_preempt22:11:241
3690294994410,8cyclictest3741827-21ssh22:44:161
3690286994428,12cyclictest3735769-21ssh22:31:310
3690286994427,12cyclictest12-21ksoftirqd/020:45:010
3690303994323,15cyclictest275-21dbus-daemon19:15:013
3690303994319,16cyclictest275-21dbus-daemon20:30:013
3690303994319,16cyclictest275-21dbus-daemon20:30:013
3690298994335,5cyclictest3684002-21kworker/2:019:50:022
3690298994331,9cyclictest30-21ksoftirqd/221:10:022
369029499438,11cyclictest23-21ksoftirqd/119:25:161
3690294994325,13cyclictest3467602-21systemd-journal20:15:001
369029499431,5cyclictest0-21swapper/119:35:481
3690294994314,5cyclictest131rcu_preempt21:52:421
3690286994336,4cyclictest16950irq/224-800000020:35:010
3690286994328,11cyclictest3467718-21lldpd20:25:010
3690286994324,14cyclictest3467718-21lldpd20:30:020
3690286994324,14cyclictest3467718-21lldpd20:30:010
369030399426,9cyclictest37-21ksoftirqd/319:50:023
3690303994227,11cyclictest275-21dbus-daemon20:10:013
3690298994233,6cyclictest309-21in:imuxsock19:30:002
3690298994228,10cyclictest3696179-21sshd21:00:042
3690298994228,10cyclictest3696179-21sshd21:00:042
3690298994227,12cyclictest3694361-21sshd20:25:042
3690298994227,12cyclictest3694361-21sshd20:25:042
369029899421,12cyclictest3691009-21sshd19:24:052
369029499424,13cyclictest23-21ksoftirqd/119:22:331
369029499423,8cyclictest23-21ksoftirqd/123:27:381
3690294994226,12cyclictest3467602-21systemd-journal21:00:001
369029499421,6cyclictest0-21swapper/120:08:471
3690286994228,9cyclictest3692828-21modprobe20:00:010
369028699422,36cyclictest3690790-21munin-run19:20:010
369030399419,6cyclictest0-21swapper/320:39:173
369030399419,6cyclictest0-21swapper/320:39:163
3690303994132,6cyclictest37-21ksoftirqd/319:20:013
369030399411,5cyclictest0-21swapper/319:39:283
369029899417,5cyclictest131rcu_preempt20:05:022
369029499414,12cyclictest23-21ksoftirqd/119:11:211
369029499413,12cyclictest23-21ksoftirqd/119:53:211
369029499413,11cyclictest23-21ksoftirqd/120:36:071
369029499413,11cyclictest23-21ksoftirqd/120:36:061
369029499411,9cyclictest0-21swapper/120:50:341
3690294994112,6cyclictest131rcu_preempt20:45:001
3690303994010,4cyclictest131rcu_preempt20:30:053
3690298994032,5cyclictest3690056-21kworker/2:219:20:002
3690298994027,10cyclictest3695193-21munin-run20:45:012
369029499405,9cyclictest23-21ksoftirqd/120:32:181
369029499403,11cyclictest23-21ksoftirqd/120:01:241
369029499403,11cyclictest23-21ksoftirqd/120:01:231
3690294994025,12cyclictest3690302-21sed19:10:031
369029499402,12cyclictest23-21ksoftirqd/121:02:081
369029499402,12cyclictest23-21ksoftirqd/121:02:081
369029499401,5cyclictest0-21swapper/119:34:321
3690286994034,4cyclictest16950irq/224-800000019:50:210
3690286994024,12cyclictest3467602-21systemd-journal19:45:010
3690286993932,4cyclictest16950irq/224-800000020:05:000
3690286993932,4cyclictest16950irq/224-800000020:05:000
3690286993927,7cyclictest309-21in:imuxsock20:50:010
369028699392,6cyclictest275-21dbus-daemon19:40:010
369030399384,6cyclictest37-21ksoftirqd/320:05:013
369030399384,6cyclictest37-21ksoftirqd/320:05:013
369029499383,11cyclictest23-21ksoftirqd/119:41:301
3690286993828,6cyclictest3691632-21cat19:35:020
369028699382,7cyclictest3696216-21munin-run21:05:000
369028699382,7cyclictest3696216-21munin-run21:05:000
3690286993827,8cyclictest3693850-21sshd20:19:060
369028699382,4cyclictest3691039-21munin-run19:25:010
3690303993727,6cyclictest3693868-21sendmail20:20:003
369028699374,25cyclictest3690292-21grep19:10:010
3690303993622,10cyclictest3692213-21sed19:45:023
3690286993522,10cyclictest309-21in:imuxsock20:11:050
3690298993417,13cyclictest3691520-21cron19:35:002
369028699341,30cyclictest0-21swapper/020:52:290
3690298993221,9cyclictest30-21ksoftirqd/219:10:022
3690303993022,6cyclictest37-21ksoftirqd/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional