You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-14 - 06:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Aug 14, 2026 00:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
56206199992,84cyclictest657614-21ssh00:15:022
56206199992,84cyclictest657614-21ssh00:15:012
562055998840,15cyclictest1-21systemd00:00:010
562055998329,25cyclictest0-21swapper/023:00:010
562058998217,20cyclictest658668-21rm00:17:001
562061997929,45cyclictest3467562-21sshd23:40:262
562058997919,17cyclictest668395-21ssh00:37:131
562058997919,17cyclictest668395-21ssh00:37:121
562058997826,15cyclictest574812-21ssh21:23:231
562058997826,15cyclictest574812-21ssh21:23:221
562058997718,22cyclictest584044-21ssh21:42:191
562058997717,21cyclictest634072-21ssh23:25:501
562058997716,18cyclictest639744-21ssh23:37:451
562058997715,26cyclictest630084-21ssh23:17:461
562055997728,25cyclictest597626-21ssh22:10:160
562058997622,17cyclictest623392-21ssh23:03:581
562058997617,20cyclictest578324-21ssh21:30:261
562058997520,15cyclictest655781-21ssh00:11:001
562058997520,15cyclictest655781-21ssh00:10:591
562058997519,16cyclictest618933-21ssh22:54:571
562058997514,16cyclictest603484-21ssh22:22:301
562058997419,18cyclictest576396-21ssh21:26:291
562058997417,18cyclictest580866-21ssh21:35:391
562055997418,6cyclictest16850irq/134-800000000:31:200
562055997418,19cyclictest16850irq/134-800000021:50:050
562055997418,19cyclictest16850irq/134-800000021:50:050
562058997218,17cyclictest598538-21ssh22:12:231
562058997218,15cyclictest608792-21ssh22:33:451
562058997217,16cyclictest609925-21ssh22:35:491
562058997215,15cyclictest571959-21ssh21:17:341
562055997223,11cyclictest629055-21ssh23:15:260
562055997217,6cyclictest16850irq/134-800000021:26:150
562055997217,18cyclictest16850irq/134-800000021:49:190
562055997117,17cyclictest16850irq/134-800000023:00:430
562058997013,17cyclictest662431-21ssh00:24:571
562055997017,16cyclictest16850irq/134-800000000:13:380
562055997017,16cyclictest16850irq/134-800000000:13:370
562061996934,29cyclictest587639-21munin-plugin-st21:49:592
562061996925,7cyclictest603526-21sshd22:22:372
562058996919,14cyclictest626768-21ssh23:10:471
562058996918,18cyclictest566404-21munin-plugin-st20:35:001
562058996917,14cyclictest632602-21ssh23:23:031
562058996916,15cyclictest586012-21ssh21:46:201
562058996915,15cyclictest649167-21ssh23:57:131
562055996917,18cyclictest16850irq/134-800000023:47:530
56206199681,32cyclictest30-21ksoftirqd/221:52:262
56206199681,32cyclictest30-21ksoftirqd/221:52:252
562058996821,16cyclictest0-21swapper/123:43:441
562058996817,14cyclictest595876-21ssh22:06:441
562058996815,16cyclictest625541-21ssh23:08:221
562058996724,12cyclictest666244-21ssh00:32:401
562058996720,15cyclictest620677-21ssh22:58:201
562058996717,15cyclictest593233-21ssh22:01:201
562058996717,15cyclictest593233-21ssh22:01:191
562058996717,15cyclictest588158-21ssh21:50:381
562058996717,15cyclictest588158-21ssh21:50:381
562058996716,15cyclictest638143-21ssh23:34:321
562058996714,20cyclictest601470-21ssh22:18:321
562061996521,5cyclictest643860-21sh23:46:042
562058996518,18cyclictest654838-21ssh00:09:191
562058996518,14cyclictest613255-21rm22:42:521
562058996514,14cyclictest652045-21ssh00:03:181
562058996514,14cyclictest652045-21ssh00:03:181
562058996513,20cyclictest647904-21ssh23:54:531
562061996425,5cyclictest669851-21cat00:40:032
562061996425,5cyclictest669851-21cat00:40:032
562058996419,14cyclictest663175-21ssh00:26:111
562058996416,16cyclictest644313-21ssh23:47:121
562061996332,26cyclictest1-21systemd21:40:012
562061996323,4cyclictest562508-21latency_hist19:20:022
562061996323,4cyclictest562508-21latency_hist19:20:012
562061996322,5cyclictest616578-21latency_hist22:50:022
562061996320,14cyclictest275-21dbus-daemon21:05:022
562058996316,15cyclictest566152-21munin-run20:30:011
562058996315,13cyclictest567961-21modprobe21:05:001
562058996314,14cyclictest591765-21ssh21:58:281
562058996314,14cyclictest569790-21ssh21:13:121
562061996225,5cyclictest567573-21latency_hist20:55:022
562061996223,9cyclictest30-21ksoftirqd/221:56:452
562061996222,10cyclictest30-21ksoftirqd/223:58:492
562061996121,4cyclictest568225-21diskmemload22:05:022
562061996121,4cyclictest568225-21diskmemload22:05:012
562055996122,3cyclictest0-21swapper/020:40:030
562058996014,13cyclictest605573-21ssh22:26:471
562061995926,30cyclictest30-21ksoftirqd/200:27:122
562061995924,5cyclictest611610-21sh22:39:462
562061995923,8cyclictest30-21ksoftirqd/200:23:212
562061995922,9cyclictest30-21ksoftirqd/200:16:442
562061995912,19cyclictest578093-21ssh21:30:012
562055995930,13cyclictest631792-21ssh23:21:110
562061995824,5cyclictest570127-21sh21:13:552
562061995822,12cyclictest30-21ksoftirqd/223:54:422
562061995820,12cyclictest30-21ksoftirqd/223:03:402
562061995815,11cyclictest275-21dbus-daemon23:25:012
562055995821,34cyclictest12-21ksoftirqd/023:09:220
562061995731,23cyclictest0-21swapper/200:05:022
562061995731,23cyclictest0-21swapper/200:05:012
562061995727,9cyclictest30-21ksoftirqd/223:34:542
562061995726,5cyclictest570819-21sh21:15:062
562061995726,5cyclictest3467562-21sshd20:11:192
562061995724,5cyclictest573943-21sh21:21:262
562061995724,5cyclictest573943-21sh21:21:252
562055995728,25cyclictest604431-21ssh22:24:400
56205599572,20cyclictest565624-21munin-plugin-st20:20:000
562055995714,11cyclictest12-21ksoftirqd/021:15:400
56205599571,40cyclictest613368-21ssh22:43:170
562061995618,6cyclictest0-21swapper/222:59:592
562061995617,11cyclictest30-21ksoftirqd/200:31:282
562058995612,15cyclictest615803-21ssh22:48:151
562058995611,15cyclictest566949-21munin-run20:45:011
56205599562,8cyclictest0-21swapper/021:13:340
562055995628,13cyclictest634700-21ssh23:27:150
562055995627,11cyclictest597457-21ssh22:10:010
56205599562,38cyclictest590436-21ssh21:55:270
56205599561,42cyclictest655361-21sh00:10:020
562055995612,41cyclictest602109-21/usr/sbin/munin22:20:010
562061995549,3cyclictest567225-21kworker/2:321:33:292
562061995523,10cyclictest30-21ksoftirqd/222:14:572
562061995521,8cyclictest30-21ksoftirqd/222:19:312
562055995529,11cyclictest647487-21ssh23:53:530
562055995519,6cyclictest16850irq/134-800000021:37:020
56205599551,4cyclictest131rcu_preempt22:34:180
562065995435,15cyclictest3467602-21systemd-journal22:20:023
562061995445,5cyclictest567225-21kworker/2:321:10:022
562061995425,5cyclictest629048-21sh23:15:272
562055995428,11cyclictest659430-21ssh00:18:410
562055995426,13cyclictest594059-21ssh22:03:140
562055995426,13cyclictest594059-21ssh22:03:130
562055995420,17cyclictest16850irq/134-800000021:44:360
562055995419,18cyclictest16850irq/134-800000022:26:490
562055995419,17cyclictest16850irq/134-800000000:21:570
562055995418,18cyclictest16850irq/134-800000023:38:530
562055995418,18cyclictest16850irq/134-800000000:29:460
562061995323,9cyclictest30-21ksoftirqd/222:54:242
562061995322,4cyclictest568225-21diskmemload21:44:272
562061995320,9cyclictest563253-21sshd19:30:132
562058995338,12cyclictest563792-21sshd19:42:041
562058995338,12cyclictest563792-21sshd19:42:031
562055995318,18cyclictest16850irq/134-800000021:20:160
562055995318,18cyclictest16850irq/134-800000021:20:150
562055995318,17cyclictest16850irq/134-800000023:33:490
562055995317,18cyclictest16850irq/134-800000022:36:380
562065995235,12cyclictest597709-21sshd22:10:263
562061995216,11cyclictest622184-1kworker/u9:023:15:012
562058995239,10cyclictest565913-21munin-run20:25:011
562058995239,10cyclictest565913-21munin-run20:25:011
562058995238,10cyclictest567439-21munin-run20:55:001
562058995236,13cyclictest566633-21sshd20:38:191
562055995226,21cyclictest12-21ksoftirqd/022:55:020
562055995217,18cyclictest16850irq/134-800000021:33:150
562055995217,17cyclictest16850irq/134-800000000:00:120
562055995217,17cyclictest16850irq/134-800000000:00:110
562065995129,16cyclictest275-21dbus-daemon22:40:003
562061995123,10cyclictest30-21ksoftirqd/200:09:312
562061995116,11cyclictest275-21dbus-daemon22:30:012
562058995135,12cyclictest563580-21munin-run19:40:011
562058995135,11cyclictest567201-21munin-run20:50:001
562055995128,7cyclictest12-21ksoftirqd/023:15:020
562055995123,12cyclictest616590-21ssh22:50:000
562061995023,8cyclictest30-21ksoftirqd/222:30:062
562061995022,9cyclictest30-21ksoftirqd/222:06:062
562061995020,11cyclictest3467602-21systemd-journal20:32:152
562058995031,15cyclictest364-21snmpd20:04:041
562065994931,14cyclictest564587-21systemctl20:00:013
56206599492,13cyclictest576228-21ssh21:26:093
562058994937,9cyclictest364-21snmpd19:10:181
562061994838,6cyclictest555147-21kworker/2:019:40:002
562061994835,10cyclictest3467602-21systemd-journal20:27:202
562058994836,9cyclictest565080-21sshd20:07:201
562058994836,9cyclictest565080-21sshd20:07:191
562058994836,9cyclictest3467602-21systemd-journal19:47:511
562058994835,9cyclictest3467602-21systemd-journal20:12:541
562058994835,10cyclictest364-21snmpd20:15:361
562058994827,17cyclictest364-21snmpd19:20:481
562055994839,5cyclictest16850irq/134-800000021:10:010
562055994813,18cyclictest16850irq/134-800000023:41:450
562058994733,11cyclictest3467602-21systemd-journal19:55:211
562065994636,6cyclictest568225-21diskmemload00:00:003
562065994629,13cyclictest3467602-21systemd-journal20:50:003
562065994628,11cyclictest3467602-21systemd-journal22:35:013
562065994627,14cyclictest275-21dbus-daemon19:20:023
562065994627,14cyclictest275-21dbus-daemon19:20:013
56206599461,4cyclictest131rcu_preempt22:54:523
562061994638,5cyclictest555147-21kworker/2:023:27:202
562061994637,5cyclictest621449-21kworker/2:223:39:412
562061994632,11cyclictest3467602-21systemd-journal20:18:322
562058994634,9cyclictest48-21kcompactd019:33:341
562058994625,17cyclictest563034-21munin-run19:30:001
562055994628,14cyclictest668046-21ssh00:36:170
562055994628,14cyclictest668046-21ssh00:36:160
562065994532,10cyclictest3467602-21systemd-journal22:20:023
562065994528,13cyclictest275-21dbus-daemon21:40:003
562061994532,9cyclictest1-21systemd19:25:002
562055994532,9cyclictest1-21systemd19:30:010
562055994531,11cyclictest3467602-21systemd-journal20:42:170
562055994529,13cyclictest565867-21sshd20:21:210
562055994529,13cyclictest565867-21sshd20:21:200
562055994528,12cyclictest1-21systemd21:05:020
56206599443,3cyclictest131rcu_preempt23:12:113
562065994431,10cyclictest275-21dbus-daemon22:00:003
562065994427,13cyclictest3467602-21systemd-journal21:30:023
562061994436,5cyclictest555147-21kworker/2:020:01:112
562058994433,8cyclictest3467602-21systemd-journal19:15:591
562058994433,8cyclictest3467602-21systemd-journal19:15:591
562058994416,22cyclictest568342-21latency_hist21:10:031
562055994437,4cyclictest16950irq/224-800000020:05:000
56205599443,5cyclictest562515-21munin-run19:20:010
56205599443,5cyclictest562515-21munin-run19:20:000
56205599442,7cyclictest563668-21wc19:40:010
56206599433,4cyclictest131rcu_preempt22:45:503
562065994326,13cyclictest275-21dbus-daemon21:55:023
562065994326,13cyclictest275-21dbus-daemon21:55:013
562065994326,12cyclictest638487-21sed23:35:023
56206599432,3cyclictest131rcu_preempt21:47:433
56206199433,4cyclictest131rcu_preempt20:00:012
562058994330,9cyclictest364-21snmpd20:55:061
562058994320,19cyclictest562077-21latency_hist19:10:011
562055994330,10cyclictest275-21dbus-daemon20:00:010
56205599431,17cyclictest0-21swapper/020:15:020
562065994229,10cyclictest37-21ksoftirqd/300:35:033
562065994227,10cyclictest568425-21wc21:10:033
562065994223,14cyclictest275-21dbus-daemon00:25:013
562065994223,13cyclictest3467602-21systemd-journal22:30:013
562065994222,17cyclictest1-21systemd23:05:013
562065994222,15cyclictest275-21dbus-daemon19:40:003
56206599421,8cyclictest0-21swapper/300:39:183
56206599421,8cyclictest0-21swapper/300:39:173
56206199427,6cyclictest131rcu_preempt20:44:202
562061994234,5cyclictest567225-21kworker/2:322:41:542
562058994229,10cyclictest364-21snmpd19:50:261
562055994233,6cyclictest19350irq/34-mmc120:05:040
562055994233,6cyclictest19350irq/34-mmc120:05:030
562055994228,11cyclictest1-21systemd21:00:000
562065994132,5cyclictest309-21in:imuxsock21:19:593
562065994129,10cyclictest37-21ksoftirqd/323:19:283
562065994112,25cyclictest1-21systemd23:50:023
56205599412,5cyclictest563832-21munin-run19:45:020
56205599412,5cyclictest563832-21munin-run19:45:010
56205599411,35cyclictest1-21systemd20:30:000
562065994029,8cyclictest564050-21sshd19:48:123
562065994027,10cyclictest275-21dbus-daemon23:45:013
562065994026,10cyclictest1-21systemd20:10:013
562065994026,10cyclictest1-21systemd20:10:003
562065994025,12cyclictest37-21ksoftirqd/300:16:023
562065994025,12cyclictest275-21dbus-daemon00:05:013
562065994025,12cyclictest275-21dbus-daemon00:05:003
562065994025,11cyclictest37-21ksoftirqd/319:30:003
56206599401,4cyclictest131rcu_preempt00:09:123
56206599401,3cyclictest131rcu_preempt22:03:043
56206599401,3cyclictest131rcu_preempt22:03:033
562055994034,3cyclictest16950irq/224-800000020:32:060
562055994025,11cyclictest1-21systemd19:15:000
56206599396,22cyclictest566648-21modprobe20:40:013
562065993929,8cyclictest37-21ksoftirqd/323:52:063
56206599391,4cyclictest131rcu_preempt23:38:103
562055993925,11cyclictest567420-21sshd20:54:210
56206599386,8cyclictest37-21ksoftirqd/323:20:583
56206599385,3cyclictest131rcu_preempt22:08:203
562065993830,5cyclictest1-21systemd21:15:003
562065993828,8cyclictest37-21ksoftirqd/321:40:263
562065993828,6cyclictest309-21in:imuxsock23:10:003
562065993825,10cyclictest563448-21sed19:35:033
562065993824,7cyclictest309-21in:imuxsock19:24:593
562061993824,10cyclictest30-21ksoftirqd/223:06:562
562061993812,5cyclictest131rcu_preempt20:37:162
562055993828,7cyclictest12-21ksoftirqd/020:50:020
562055993827,8cyclictest12-21ksoftirqd/019:32:270
562065993730,4cyclictest566952-21latency_hist20:45:013
562065993727,8cyclictest37-21ksoftirqd/320:20:013
562065993726,9cyclictest37-21ksoftirqd/300:11:413
562065993726,9cyclictest37-21ksoftirqd/300:11:403
562065993724,11cyclictest37-21ksoftirqd/322:57:003
562065993723,11cyclictest3467602-21systemd-journal23:29:033
562065993723,10cyclictest37-21ksoftirqd/321:00:013
56206199379,6cyclictest131rcu_preempt20:07:192
56206199379,6cyclictest131rcu_preempt20:07:192
562061993725,9cyclictest567808-21sed21:00:012
562055993726,8cyclictest12-21ksoftirqd/019:47:120
562055993724,9cyclictest3467602-21systemd-journal19:22:130
56206599364,4cyclictest131rcu_preempt22:43:153
562065993622,10cyclictest566390-21cron20:35:003
562065993617,15cyclictest275-21dbus-daemon21:05:003
562065993616,12cyclictest37-21ksoftirqd/320:15:003
56206599361,3cyclictest131rcu_preempt00:26:123
562055993623,9cyclictest564343-21wc19:55:020
56206599353,3cyclictest131rcu_preempt21:21:023
56206599353,3cyclictest131rcu_preempt21:21:013
562065993527,6cyclictest564853-21latency_hist20:05:013
562065993524,9cyclictest37-21ksoftirqd/319:55:023
56206199355,7cyclictest30-21ksoftirqd/219:11:132
562061993522,9cyclictest1-21systemd19:45:012
562061993522,9cyclictest1-21systemd19:45:002
562061993522,8cyclictest30-21ksoftirqd/219:30:022
562061993520,11cyclictest3467602-21systemd-journal20:25:012
562061993520,11cyclictest3467602-21systemd-journal20:25:002
56206599345,21cyclictest567427-21modprobe20:55:013
562065993418,8cyclictest0-21swapper/320:25:023
562065993418,8cyclictest0-21swapper/320:25:013
562061993425,6cyclictest564058-21cron19:50:002
562065993320,10cyclictest562104-21date19:10:023
562065993311,4cyclictest346771850lldpd19:43:243
562065993311,4cyclictest346771850lldpd19:43:233
562061993226,4cyclictest564322-21latency_hist19:55:012
562055993225,5cyclictest12-21ksoftirqd/019:10:020
56206599314,4cyclictest131rcu_preempt19:14:593
56206599314,3cyclictest131rcu_preempt20:30:013
562061993121,8cyclictest567188-21/usr/sbin/munin20:50:022
562061992921,6cyclictest30-21ksoftirqd/219:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional