You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-19 - 05:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed Aug 19, 2026 00:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2201459996930,35cyclictest2234542-21sh22:05:531
2201456996920,43cyclictest2298298-21run-parts00:20:000
2201459996823,41cyclictest23-21ksoftirqd/121:31:521
2201456996619,43cyclictest0-21swapper/021:51:360
220145999651,14cyclictest2290041-21ssh00:02:211
220145999651,14cyclictest2290041-21ssh00:02:201
2201464996419,9cyclictest30-21ksoftirqd/223:12:472
2201464996416,12cyclictest30-21ksoftirqd/222:53:332
2201464996416,12cyclictest30-21ksoftirqd/222:53:322
2201464996413,15cyclictest2239453-21ssh22:16:192
220145999642,15cyclictest2204774-21munin-run20:15:001
220145699641,36cyclictest131rcu_preempt23:03:300
220145699641,36cyclictest131rcu_preempt23:03:300
220146899631,38cyclictest2204503-21kworker/u8:120:20:023
2201464996320,9cyclictest30-21ksoftirqd/222:36:012
2201464996319,10cyclictest30-21ksoftirqd/221:34:542
2201464996317,9cyclictest30-21ksoftirqd/222:01:102
2201464996316,11cyclictest30-21ksoftirqd/220:20:022
220145999632,13cyclictest2206865-21munin-run20:55:011
2201459996321,38cyclictest2217001-21sh21:29:381
220145999631,9cyclictest3467602-21systemd-journal00:33:071
220145999631,9cyclictest3467602-21systemd-journal00:33:071
220145999622,13cyclictest2279804-21ssh23:40:511
220145999622,13cyclictest2279804-21ssh23:40:511
220145999621,9cyclictest2211502-21ssh21:17:531
220145999621,8cyclictest2270063-21ssh23:20:071
220145999621,8cyclictest2270063-21ssh23:20:071
220145999621,8cyclictest2255805-21ssh22:50:331
220145999621,8cyclictest2255805-21ssh22:50:331
220145999621,11cyclictest2213131-21ssh21:21:051
220145699627,31cyclictest16950irq/224-800000022:00:020
220146499615,19cyclictest2202097-21latency_hist19:20:022
2201464996119,9cyclictest30-21ksoftirqd/221:10:032
2201464996117,10cyclictest30-21ksoftirqd/200:35:002
2201464996117,10cyclictest30-21ksoftirqd/200:35:002
2201464996116,10cyclictest30-21ksoftirqd/223:09:522
2201464996115,8cyclictest30-21ksoftirqd/222:35:002
2201464996115,11cyclictest30-21ksoftirqd/223:39:492
2201464996114,10cyclictest30-21ksoftirqd/222:27:342
2201464996112,11cyclictest30-21ksoftirqd/200:03:182
2201464996112,11cyclictest30-21ksoftirqd/200:03:182
220145999613,21cyclictest2241110-21munin-run22:20:001
220145999612,8cyclictest364-21snmpd00:16:581
220145999612,13cyclictest2255344-21munin-run22:50:011
2201459996121,35cyclictest2269894-21latency_hist23:20:011
220145999611,8cyclictest2292091-21ssh00:06:371
220145999611,8cyclictest2285399-21ssh23:52:401
220145999611,8cyclictest2273878-21ssh23:28:391
220145999611,8cyclictest2265606-21ssh23:11:001
220145999611,8cyclictest2243739-21ssh22:25:031
220145999611,8cyclictest2242277-21ssh22:22:131
220145999611,7cyclictest2300924-21ssh00:25:031
220145999611,7cyclictest2227334-21ssh21:50:461
220145999611,10cyclictest2208795-21ssh21:12:031
2201456996117,14cyclictest275-21dbus-daemon22:10:010
2201464996014,10cyclictest30-21ksoftirqd/221:45:212
2201464996012,11cyclictest30-21ksoftirqd/221:52:262
220145999602,15cyclictest2250935-21ssh22:40:051
220145999602,14cyclictest2219942-21ssh21:35:271
220145999602,13cyclictest48-21kcompactd021:55:301
220145999601,8cyclictest2262329-21ssh23:04:281
220145999601,8cyclictest2262329-21ssh23:04:281
220145999601,8cyclictest2259889-21ssh22:59:141
220145999601,7cyclictest2305875-21ssh00:35:381
220145999601,12cyclictest2286930-21ssh23:55:421
220145699601,40cyclictest131rcu_preempt22:52:430
220145699601,40cyclictest131rcu_preempt22:52:430
2201464995920,9cyclictest30-21ksoftirqd/200:25:032
2201464995916,11cyclictest30-21ksoftirqd/223:05:022
2201464995916,11cyclictest30-21ksoftirqd/223:05:012
2201464995915,10cyclictest30-21ksoftirqd/221:40:382
2201464995912,9cyclictest30-21ksoftirqd/223:55:462
220145999593,5cyclictest364-21snmpd22:36:371
2201459995931,24cyclictest2276957-21sed23:35:021
220145999592,6cyclictest2295943-21munin-run00:15:011
220145999592,15cyclictest2236906-21ssh22:10:501
220145999591,8cyclictest2248117-21ssh22:34:441
220145999591,8cyclictest2223362-21ssh21:42:411
220145999591,7cyclictest364-21snmpd19:46:021
220145999591,7cyclictest2300039-21ssh00:23:301
220145999591,7cyclictest2300039-21ssh00:23:301
220145999591,7cyclictest2279053-21ssh23:39:391
2201459995916,38cyclictest1-21systemd19:15:011
220145999591,13cyclictest364-21snmpd23:06:081
220146499586,17cyclictest2204269-21/usr/sbin/munin20:05:012
2201464995815,8cyclictest30-21ksoftirqd/222:49:052
2201464995813,11cyclictest30-21ksoftirqd/223:49:372
220145999582,8cyclictest2226758-21munin-run21:50:001
220145999582,6cyclictest2205567-21munin-run20:30:001
220145999581,7cyclictest2232962-21ssh22:02:441
2201456995828,14cyclictest2307227-21ssh00:38:470
2201456995827,16cyclictest2254613-21ssh22:48:180
2201456995826,16cyclictest2273416-21ssh23:27:280
2201456995815,38cyclictest2279424-21cat23:40:040
2201464995717,8cyclictest30-21ksoftirqd/221:39:302
2201464995716,9cyclictest30-21ksoftirqd/222:42:162
2201464995715,10cyclictest30-21ksoftirqd/223:50:422
2201464995713,8cyclictest30-21ksoftirqd/200:37:092
220145999572,6cyclictest2205832-21munin-run20:35:011
220145999572,6cyclictest2204269-21munin-run20:05:001
2201459995717,35cyclictest2202709-21systemd-run19:35:011
2201456995730,23cyclictest19350irq/34-mmc122:13:470
2201456995728,13cyclictest2244977-21ssh22:27:560
2201456995724,3cyclictest22132961ssh21:21:250
2201464995614,9cyclictest30-21ksoftirqd/222:10:082
2201464995614,11cyclictest30-21ksoftirqd/221:22:382
220145999568,43cyclictest0-21swapper/121:06:201
220145999565,41cyclictest2203724-21modprobe19:55:001
220145999565,41cyclictest2203724-21modprobe19:55:001
2201459995631,21cyclictest2204789-21kworker/u8:420:20:001
220145999562,12cyclictest2205326-21munin-run20:25:001
220145999561,7cyclictest2282809-21ssh23:47:141
2201456995625,15cyclictest364-21snmpd22:40:040
2201464995513,8cyclictest30-21ksoftirqd/221:25:312
2201464995513,8cyclictest30-21ksoftirqd/200:11:362
2201459995529,23cyclictest2206070-21sendmail20:40:001
2201456995545,5cyclictest16850irq/134-800000023:55:010
2201456995527,13cyclictest2247179-21ssh22:32:260
2201456995527,12cyclictest2249049-21ssh22:36:190
220145699552,11cyclictest2276902-21latency_hist23:35:010
2201464995410,9cyclictest30-21ksoftirqd/223:41:072
2201464995410,9cyclictest30-21ksoftirqd/223:41:072
2201459995418,32cyclictest2202959-21/usr/sbin/munin19:40:011
2201456995444,7cyclictest16950irq/224-800000020:55:000
2201456995428,11cyclictest2223851-21ssh21:43:500
220146499539,14cyclictest2207638-21diskmemload21:57:292
220145999535,40cyclictest2204521-21modprobe20:10:001
220145999535,40cyclictest2204521-21modprobe20:10:001
2201459995319,30cyclictest2202403-21sshd19:25:381
2201459995315,33cyclictest1-21systemd19:25:011
2201456995329,9cyclictest2233708-21ssh22:04:250
220145699532,3cyclictest131rcu_preempt22:59:380
2201456995319,29cyclictest12-21ksoftirqd/000:27:030
2201464995217,6cyclictest131rcu_preempt21:00:002
2201464995217,6cyclictest131rcu_preempt21:00:002
220145999521,11cyclictest364-21snmpd20:59:011
220145999521,11cyclictest364-21snmpd20:59:011
2201456995237,11cyclictest1-21systemd20:50:010
2201456995236,12cyclictest275-21dbus-daemon00:00:020
2201456995230,16cyclictest275-21dbus-daemon21:20:010
2201456995228,20cyclictest12-21ksoftirqd/021:27:050
2201456995227,10cyclictest2303555-21ssh00:30:470
2201456995227,10cyclictest2303555-21ssh00:30:470
2201456995226,10cyclictest2220921-21ssh21:37:390
2201456995217,29cyclictest12-21ksoftirqd/000:00:280
2201456995217,29cyclictest12-21ksoftirqd/000:00:280
2201468995136,12cyclictest37-21ksoftirqd/323:05:033
2201468995136,12cyclictest37-21ksoftirqd/323:05:023
2201468995136,10cyclictest37-21ksoftirqd/323:30:033
220146499513,10cyclictest2241669-21ssh22:20:472
220145999512,6cyclictest2201947-21systemd-run19:20:001
220145999512,4cyclictest2206349-21munin-run20:45:011
2201456995130,16cyclictest1-21systemd21:10:010
2201456995126,17cyclictest275-21dbus-daemon19:15:010
220145699512,34cyclictest2208944-21ssh21:12:280
2201456995122,14cyclictest2298965-21ssh00:20:570
2201456995122,14cyclictest2298965-21ssh00:20:570
2201468995036,10cyclictest2212445-21cron21:20:003
2201464995014,6cyclictest30-21ksoftirqd/223:34:312
220145999502,7cyclictest2203968-21sshd19:58:381
2201456995028,7cyclictest2218483-21ssh21:32:350
220145699501,36cyclictest151rcuc/019:20:020
220146899494,30cyclictest361rcuc/319:45:033
2201464994913,7cyclictest30-21ksoftirqd/222:56:322
220145999493,40cyclictest0-21swapper/121:05:001
2201456994922,12cyclictest2240204-21ssh22:18:000
220145999482,8cyclictest2203216-21cron19:45:011
220145999482,6cyclictest275-21dbus-daemon20:50:011
2201456994837,9cyclictest12-21ksoftirqd/019:45:010
2201456994821,8cyclictest12-21ksoftirqd/000:13:280
2201468994733,10cyclictest3467602-21systemd-journal21:35:003
2201468994731,11cyclictest309-21in:imuxsock20:09:383
2201468994731,11cyclictest309-21in:imuxsock20:09:383
2201464994739,5cyclictest2230804-21kworker/2:423:23:072
2201464994739,5cyclictest2230804-21kworker/2:423:23:072
220146499473,13cyclictest2210284-21ssh21:15:042
2201456994738,5cyclictest0-21swapper/021:46:260
2201456994734,10cyclictest16950irq/224-800000023:50:020
2201456994730,13cyclictest2268414-21ssh23:16:490
220145699472,7cyclictest12-21ksoftirqd/023:22:280
220145699472,7cyclictest12-21ksoftirqd/023:22:280
2201456994722,10cyclictest2264897-21ssh23:09:490
220146899465,4cyclictest131rcu_preempt22:20:003
220146899463,3cyclictest131rcu_preempt22:31:333
2201464994636,7cyclictest2206070-21sendmail20:40:002
2201456994639,4cyclictest16850irq/134-800000023:45:020
2201456994639,4cyclictest16850irq/134-800000023:45:010
220145699462,39cyclictest2205300-21cron20:25:000
2201468994532,8cyclictest2279228-21cron23:40:003
2201468994530,11cyclictest37-21ksoftirqd/321:40:383
220146499455,12cyclictest0-21swapper/200:17:152
220146499452,21cyclictest2195073-21kworker/u8:019:10:022
220145999452,5cyclictest2201568-21latency_hist19:10:011
2201456994528,13cyclictest2292051-21ssh00:06:320
2201456994524,6cyclictest12-21ksoftirqd/023:12:320
2201456994523,9cyclictest2207638-21diskmemload22:23:350
2201468994432,9cyclictest1-21systemd23:35:013
2201468994426,13cyclictest2202948-21cron19:40:003
2201468994421,17cyclictest275-21dbus-daemon23:20:013
220146899441,3cyclictest131rcu_preempt22:47:483
220146899441,3cyclictest131rcu_preempt00:27:393
220145699443,7cyclictest2205561-21cron20:30:000
2201456994425,14cyclictest1-21systemd20:10:010
2201456994425,14cyclictest1-21systemd20:10:000
2201456994423,16cyclictest275-21dbus-daemon19:55:010
2201456994423,16cyclictest275-21dbus-daemon19:55:010
2201468994335,5cyclictest2207638-21diskmemload21:10:003
2201468994327,10cyclictest309-21in:imuxsock21:05:003
220146899432,3cyclictest131rcu_preempt00:39:033
2201468994323,15cyclictest3467602-21systemd-journal23:00:003
220146899431,3cyclictest131rcu_preempt21:25:393
2201464994335,5cyclictest2204547-21kworker/2:320:25:382
220146499432,9cyclictest30-21ksoftirqd/223:25:332
2201464994327,11cyclictest309-21in:imuxsock21:10:002
2201464994310,5cyclictest0-21swapper/222:08:192
2201456994336,4cyclictest16950irq/224-800000020:00:000
2201456994327,11cyclictest275-21dbus-daemon19:50:010
2201468994231,5cyclictest2298297-21munin-plugin-st00:20:003
2201468994227,11cyclictest37-21ksoftirqd/322:25:213
2201468994226,12cyclictest37-21ksoftirqd/322:25:013
2201468994226,12cyclictest275-21dbus-daemon23:50:003
220146899422,3cyclictest131rcu_preempt00:23:383
220146899422,3cyclictest131rcu_preempt00:23:373
220146899422,3cyclictest131rcu_preempt00:08:123
220146899421,3cyclictest131rcu_preempt23:20:353
220146899421,3cyclictest131rcu_preempt23:20:353
2201464994227,11cyclictest30-21ksoftirqd/223:18:262
2201456994235,4cyclictest16950irq/224-800000020:45:010
220146899413,3cyclictest131rcu_preempt00:11:213
2201468994130,9cyclictest37-21ksoftirqd/319:25:023
2201468994126,10cyclictest1-21systemd22:10:013
2201468994124,13cyclictest3467602-21systemd-journal19:59:593
220146899412,3cyclictest131rcu_preempt22:37:593
2201468994122,14cyclictest1-21systemd19:55:013
2201468994122,14cyclictest1-21systemd19:55:013
220146899411,3cyclictest131rcu_preempt00:04:233
220146899411,3cyclictest131rcu_preempt00:04:233
2201456994134,4cyclictest16950irq/224-800000021:05:000
2201456994127,10cyclictest364-21snmpd20:31:110
220146899403,3cyclictest131rcu_preempt23:50:533
220146899403,3cyclictest131rcu_preempt22:55:023
220146899403,3cyclictest131rcu_preempt22:55:023
2201468994029,9cyclictest37-21ksoftirqd/321:21:363
2201468994027,10cyclictest37-21ksoftirqd/322:12:233
2201464994028,8cyclictest2202157-21sshd19:20:552
2201464994027,9cyclictest309-21in:imuxsock19:14:592
2201464994022,12cyclictest2300732-21sed00:25:012
2201464994022,12cyclictest2300732-21sed00:25:012
2201464994019,17cyclictest2291961-21ssh00:06:212
220146499401,13cyclictest0-21swapper/219:35:002
2201456994030,7cyclictest2207107-21sendmail21:00:000
2201456994030,7cyclictest2207107-21sendmail21:00:000
2201456994026,10cyclictest12-21ksoftirqd/019:25:020
2201456994024,12cyclictest275-21dbus-daemon19:40:010
2201456994015,6cyclictest131rcu_preempt19:35:010
2201468993931,6cyclictest37-21ksoftirqd/322:00:023
2201468993929,7cyclictest2204243-21sshd20:02:383
2201468993925,11cyclictest37-21ksoftirqd/323:43:303
2201468993925,11cyclictest37-21ksoftirqd/323:43:303
2201468993925,11cyclictest37-21ksoftirqd/322:00:123
2201468993924,12cyclictest37-21ksoftirqd/320:55:013
2201468993923,12cyclictest1-21systemd00:00:023
2201468993923,11cyclictest275-21dbus-daemon21:50:013
2201464993920,13cyclictest3467602-21systemd-journal19:55:012
2201464993920,13cyclictest3467602-21systemd-journal19:55:012
2201456993928,8cyclictest12-21ksoftirqd/020:05:020
2201456993924,11cyclictest309-21in:imuxsock20:36:380
220146899386,7cyclictest37-21ksoftirqd/321:53:383
2201468993828,7cyclictest2209858-21sshd21:14:383
2201468993825,9cyclictest2253269-21sed22:45:033
2201468993820,12cyclictest309-21in:imuxsock20:50:003
220146899381,3cyclictest131rcu_preempt23:10:113
2201464993825,10cyclictest2204487-21sshd20:06:082
2201464993825,10cyclictest2204487-21sshd20:06:082
2201464993823,10cyclictest3467602-21systemd-journal20:00:002
2201456993827,6cyclictest2205050-21sendmail20:19:590
2201468993718,15cyclictest275-21dbus-daemon19:35:003
2201464993726,8cyclictest3467602-21systemd-journal20:44:022
2201464993726,8cyclictest30-21ksoftirqd/219:45:012
2201456993723,10cyclictest3467602-21systemd-journal20:13:380
220145699371,33cyclictest0-21swapper/019:10:020
220146899364,3cyclictest131rcu_preempt21:38:163
220146899363,3cyclictest131rcu_preempt00:31:323
220146899363,3cyclictest131rcu_preempt00:31:313
2201468993622,10cyclictest37-21ksoftirqd/323:05:183
2201468993621,11cyclictest1-21systemd19:45:013
2201464993627,6cyclictest2206838-21cron20:55:002
2201464993624,9cyclictest30-21ksoftirqd/220:32:062
2201464993610,5cyclictest131rcu_preempt19:48:572
220146899354,3cyclictest131rcu_preempt21:00:003
220146899354,3cyclictest131rcu_preempt21:00:003
2201468993526,5cyclictest2201907-21sendmail19:20:003
220146899352,5cyclictest131rcu_preempt20:40:003
220146899351,3cyclictest131rcu_preempt20:25:003
2201464993526,7cyclictest30-21ksoftirqd/220:50:032
2201464993514,17cyclictest2207398-21unin-run21:05:002
220146499351,30cyclictest2202944-21modprobe19:40:002
2201468993423,7cyclictest2202497-21wc19:30:023
220146499346,23cyclictest2202442-21cut19:30:002
2201456993416,13cyclictest275-21dbus-daemon19:30:010
220146899333,3cyclictest131rcu_preempt20:45:003
2201468993326,4cyclictest2205583-21latency_hist20:30:013
2201468993322,7cyclictest37-21ksoftirqd/320:35:023
2201464993319,11cyclictest2205285-21sshd20:21:382
220146899314,3cyclictest131rcu_preempt19:12:383
2201468993116,11cyclictest275-21dbus-daemon20:15:003
220146499311,5cyclictest0-21swapper/220:14:042
2201468992922,4cyclictest2201470-21latency_hist19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional