You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-20 - 09:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Jul 20, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
277104521270,3sleep00-21swapper/019:10:130
284165421250,5sleep33056-21PK-Backend22:30:013
279165821220,2sleep20-21swapper/220:05:242
283847521170,2sleep10-21swapper/122:20:131
287532221140,4sleep33056-21PK-Backend00:05:013
276873721130,21sleep126-21ksoftirqd/119:05:111
276873721130,21sleep126-21ksoftirqd/119:05:111
278774121120,4sleep33056-21PK-Backend19:55:193
276959921110,19sleep342-21ksoftirqd/319:05:263
276959921110,19sleep342-21ksoftirqd/319:05:253
279820721030,3sleep20-21swapper/220:25:132
288019421010,2sleep20-21swapper/200:15:222
276991727553,14sleep00-21swapper/019:06:370
276991727553,14sleep00-21swapper/019:06:370
276873826543,15sleep20-21swapper/219:05:132
276873826543,15sleep20-21swapper/219:05:132
28087602590,2sleep30-21swapper/320:55:143
28193622440,3sleep10-21swapper/121:25:161
28073972380,3sleep03056-21PK-Backend20:50:170
27996302360,2sleep00-21swapper/020:30:130
27843202350,2sleep22784322-21cat19:45:202
2770408993011,17cyclictest2840200-21expr22:25:121
277040899289,5cyclictest2790449-21seq20:02:461
277040899280,26cyclictest26-21ksoftirqd/122:40:131
2770413992712,13cyclictest2866299-21unin-run23:40:002
277040899279,4cyclictest2830580-21seq21:57:191
277040899279,4cyclictest1095-21runrttasks23:20:151
277040899278,5cyclictest2886827-21if_enp2s000:35:151
277040899278,5cyclictest2802914-21date20:40:001
277040899278,3cyclictest2793221-21missed_timers20:10:181
2770408992710,2cyclictest2848417-21latency_hist22:50:011
139099270,4rtkit-daemon1389-21rtkit-daemon22:25:540
277040899268,3cyclictest2870158-21cpuspeed_turbos23:50:121
277040899267,5cyclictest2822590-21seq21:35:141
277040899266,5cyclictest2875726-21find00:05:161
139099260,4rtkit-daemon1389-21rtkit-daemon19:39:140
277040899259,3cyclictest2779075-21swap19:30:231
277040899257,5cyclictest2829201-21sed21:55:001
277040899257,4cyclictest2835172-21expr22:10:121
277040899256,5cyclictest2855445-21seq23:08:081
277040899254,6cyclictest2833973-21cat22:05:231
277040899253,10cyclictest2881678-21chrt00:20:181
139099250,4rtkit-daemon1389-21rtkit-daemon21:33:550
139099250,4rtkit-daemon1389-21rtkit-daemon20:37:560
277040899247,5cyclictest26-21ksoftirqd/119:45:121
277040899247,4cyclictest2855989-21cpuspeed_turbos23:10:121
277040899247,3cyclictest2815326-21cat21:15:011
277040899247,3cyclictest2796065-21sed20:20:011
277040899247,3cyclictest2791816-21switchtime20:05:261
277040899245,4cyclictest2843839-21expr22:35:131
277040899245,3cyclictest0-21swapper/120:30:131
277040899240,8cyclictest436-20systemd-journal23:35:001
277040899240,12cyclictest0-21swapper/123:59:591
2770403992415,5cyclictest1095-21runrttasks22:55:110
2770403992414,7cyclictest436-20systemd-journal00:40:000
139099240,4rtkit-daemon1389-21rtkit-daemon22:17:470
28284392230,5chrt0-21swapper/021:50:200
2770423992321,1cyclictest3056-21PK-Backend00:10:153
2770423992320,2cyclictest3056-21PK-Backend21:52:463
2770413992310,11cyclictest436-20systemd-journal20:34:592
277040899239,4cyclictest26-21ksoftirqd/121:20:011
277040899239,3cyclictest2803255-21cstates20:40:121
277040899238,3cyclictest2864999-21cstates23:35:131
277040899237,3cyclictest2770786-21cpu19:10:121
277040899236,4cyclictest2846926-21seq22:45:101
277040899236,4cyclictest2786698-21seq19:53:071
277040899236,4cyclictest0-21swapper/120:45:091
277040899236,3cyclictest28587992chrt23:16:431
277040899236,3cyclictest2776217-21cut19:25:001
277040899232,7cyclictest2824455-21expr21:40:121
277040899232,4cyclictest2841751-21apt22:30:141
277040899232,4cyclictest2825834-21/usr/sbin/munin21:45:231
277040899231,8cyclictest2874230-21irqstats00:00:191
277040899231,8cyclictest2776473-21apt19:25:131
277040899230,8cyclictest2796954-21/usr/sbin/munin20:20:201
277040899230,8cyclictest2788008-21/usr/sbin/munin19:55:231
277040899230,8cyclictest2782004-21expr19:40:141
2770403992310,11cyclictest2878928-21cron00:15:000
139099230,15rtkit-daemon0-21swapper/223:15:052
28078472220,4sleep10-21swapper/120:50:221
2770423992219,2cyclictest3056-21PK-Backend23:08:373
2770423992219,2cyclictest3056-21PK-Backend22:35:453
2770423992219,2cyclictest3056-21PK-Backend00:37:053
2770423992212,8cyclictest883-21dbus-daemon21:13:443
277042399220,20cyclictest3056-21PK-Backend21:33:533
277042399220,20cyclictest3056-21PK-Backend19:44:023
277040899225,15cyclictest0-21swapper/119:15:141
277040899221,9cyclictest2837406-21python322:15:221
277040899221,8cyclictest2884885-21irqstats00:30:171
277040899221,8cyclictest2884885-21irqstats00:30:171
277040899221,8cyclictest2831449-21diskstats22:00:131
277040899221,4cyclictest2850621-21cnf-update-db22:55:011
277040899220,9cyclictest2798544-21/usr/sbin/munin20:25:191
277040899220,8cyclictest2820996-21expr21:30:141
277040899220,20cyclictest0-21swapper/123:40:301
277040899220,20cyclictest0-21swapper/123:40:301
2770403992219,2cyclictest0-21swapper/022:49:280
2770403992219,2cyclictest0-21swapper/019:41:570
2770403992216,4cyclictest2877195-21cron00:10:000
2770403992212,7cyclictest2846549-21sshd22:43:290
139099220,14rtkit-daemon0-21swapper/200:06:542
2770423992119,1cyclictest3056-21PK-Backend22:42:563
2770423992119,1cyclictest0-21swapper/323:42:033
2770423992119,1cyclictest0-21swapper/323:42:033
2770423992119,1cyclictest0-21swapper/321:04:193
2770423992119,1cyclictest0-21swapper/320:48:243
2770423992119,1cyclictest0-21swapper/319:16:313
2770423992119,1cyclictest0-21swapper/300:30:223
2770423992119,1cyclictest0-21swapper/300:30:223
2770423992119,1cyclictest0-21swapper/300:25:293
2770423992118,2cyclictest3056-21PK-Backend20:10:323
277042399210,20cyclictest0-21swapper/300:24:283
277042399210,19cyclictest3056-21PK-Backend23:30:203
277042399210,19cyclictest3056-21PK-Backend23:26:413
277042399210,19cyclictest3056-21PK-Backend21:26:573
277042399210,19cyclictest3056-21PK-Backend19:36:063
277042399210,19cyclictest3056-21PK-Backend00:09:133
2770413992112,7cyclictest2855598-21cron23:10:002
277041399210,20cyclictest3056-21PK-Backend20:38:032
277041399210,20cyclictest0-21swapper/222:20:182
277040899216,4cyclictest2861083-21cpu23:25:131
277040899212,5cyclictest2174-21JS10:27:481
277040899211,7cyclictest2818064-21python321:20:301
277040899210,7cyclictest2780583-21/usr/sbin/munin19:35:191
2770403992119,1cyclictest0-21swapper/023:54:100
2770403992119,1cyclictest0-21swapper/023:04:010
2770403992119,1cyclictest0-21swapper/021:10:110
2770403992119,1cyclictest0-21swapper/020:27:530
2770403992119,1cyclictest0-21swapper/019:57:520
2770403992110,8cyclictest436-20systemd-journal21:30:000
277040399210,20cyclictest0-21swapper/021:08:160
277040399210,20cyclictest0-21swapper/020:21:180
277040399210,1cyclictest0-21swapper/023:08:480
277040399210,19cyclictest3056-21PK-Backend00:22:500
277040399210,19cyclictest0-21swapper/021:22:560
2770423992018,1cyclictest3056-21PK-Backend22:30:103
2770423992018,1cyclictest3056-21PK-Backend22:11:193
2770423992018,1cyclictest3056-21PK-Backend21:36:573
2770423992018,1cyclictest3056-21PK-Backend19:21:383
2770423992018,1cyclictest0-21swapper/323:45:413
2770423992018,1cyclictest0-21swapper/323:35:473
2770423992018,1cyclictest0-21swapper/321:56:513
2770423992018,1cyclictest0-21swapper/300:16:573
2770423992017,2cyclictest3056-21PK-Backend21:40:033
2770423992017,2cyclictest3056-21PK-Backend20:35:533
2770423992017,2cyclictest3056-21PK-Backend20:18:573
277042399200,1cyclictest3056-21PK-Backend23:18:233
277042399200,1cyclictest3056-21PK-Backend23:04:383
277042399200,1cyclictest3056-21PK-Backend19:46:253
277042399200,1cyclictest3056-21PK-Backend19:33:413
277042399200,19cyclictest3056-21PK-Backend23:53:453
277042399200,19cyclictest0-21swapper/322:55:523
277042399200,19cyclictest0-21swapper/321:21:433
277042399200,19cyclictest0-21swapper/320:50:343
2770413992015,4cyclictest2881065-21tr00:20:122
2770413992013,5cyclictest2873454-21cron00:00:002
2770413992013,5cyclictest2175-21JS10:27:482
2770413992011,7cyclictest2806493-21cron20:50:012
277041399200,19cyclictest0-21swapper/222:33:422
277040899208,4cyclictest2812140-21cpuspeed_turbos21:05:131
277040899206,3cyclictest2869234-21timerandwakeup23:45:231
277040899204,5cyclictest2853212-21processes23:00:221
277040899200,19cyclictest0-21swapper/121:00:371
277040399209,9cyclictest918-21rs:main0
277040399203,13cyclictest2803046-21apt20:40:110
2770403992018,1cyclictest0-21swapper/021:17:080
2770403992018,1cyclictest0-21swapper/020:59:370
2770403992018,1cyclictest0-21swapper/020:18:070
2770403992018,1cyclictest0-21swapper/020:12:050
2770403992018,1cyclictest0-21swapper/000:02:080
2770403992017,2cyclictest0-21swapper/022:21:230
2770403992015,3cyclictest2872124-21cpuspeed_turbos23:55:120
277040399200,8cyclictest27843222cat19:45:190
277040399200,1cyclictest0-21swapper/023:14:220
277040399200,19cyclictest0-21swapper/021:37:030
277040399200,19cyclictest0-21swapper/019:16:350
277040399200,18cyclictest0-21swapper/023:47:410
139099200,14rtkit-daemon3056-21PK-Backend23:10:573
2770423991917,1cyclictest3056-21PK-Backend23:23:033
2770423991917,1cyclictest3056-21PK-Backend22:05:503
2770423991917,1cyclictest3056-21PK-Backend21:46:103
2770423991916,2cyclictest3056-21PK-Backend20:25:123
277042399190,1cyclictest3056-21PK-Backend20:31:483
277042399190,1cyclictest3056-21PK-Backend20:05:333
277042399190,1cyclictest0-21swapper/321:08:453
277042399190,1cyclictest0-21swapper/320:42:433
277042399190,1cyclictest0-21swapper/319:13:343
277042399190,18cyclictest3056-21PK-Backend22:18:073
2770413991917,1cyclictest0-21swapper/221:01:352
2770413991913,4cyclictest2777950-21grep19:30:002
2770413991912,5cyclictest2850141-21apt-get22:55:002
2770413991912,4cyclictest2803907-21python320:40:202
277041399190,1cyclictest0-21swapper/222:47:002
277041399190,17cyclictest2831815-21ntpq22:00:182
277040899190,1cyclictest0-21swapper/100:16:121
2770403991917,1cyclictest0-21swapper/023:16:590
2770403991917,1cyclictest0-21swapper/022:01:150
2770403991917,1cyclictest0-21swapper/021:02:070
2770403991917,1cyclictest0-21swapper/020:08:400
2770403991917,1cyclictest0-21swapper/019:32:250
2770403991917,1cyclictest0-21swapper/019:24:400
2770403991917,1cyclictest0-21swapper/000:34:460
2770403991917,1cyclictest0-21swapper/000:34:460
2770403991913,4cyclictest13-21ksoftirqd/022:55:000
2770403991911,5cyclictest2864534-21cut23:35:010
277040399191,13cyclictest2789141-21apt-get20:00:050
277040399190,1cyclictest0-21swapper/023:21:390
277040399190,1cyclictest0-21swapper/021:45:060
277040399190,18cyclictest0-21swapper/000:29:180
139099190,13rtkit-daemon3056-21PK-Backend00:36:002
139099190,13rtkit-daemon0-21swapper/223:50:332
2770423991816,1cyclictest0-21swapper/323:55:483
277042399181,15cyclictest2831939-21python322:00:203
277041399189,7cyclictest436-20systemd-journal21:20:002
277041399183,14cyclictest1095-21runrttasks00:33:092
277041399183,14cyclictest1095-21runrttasks00:33:092
2770413991816,1cyclictest0-21swapper/223:49:472
2770413991816,1cyclictest0-21swapper/221:34:342
2770413991813,3cyclictest2843665-21tr22:35:122
2770413991812,4cyclictest1095-21runrttasks21:20:312
2770413991811,6cyclictest82332-21update-manager21:13:442
2770413991811,6cyclictest2823460-21awk21:35:232
277041399181,16cyclictest1095-21runrttasks00:13:402
2770413991811,5cyclictest2787263-21cut19:55:142
2770413991810,6cyclictest2845772-21cat22:40:162
2770413991810,6cyclictest2771294-21wc19:10:172
277041399180,7cyclictest436-20systemd-journal19:36:182
2770408991816,1cyclictest3056-21PK-Backend00:27:351
277040399185,3cyclictest141rcu_preempt00:20:000
2770403991816,1cyclictest0-21swapper/023:41:400
2770403991816,1cyclictest0-21swapper/023:41:400
2770403991816,1cyclictest0-21swapper/019:50:420
2770403991812,5cyclictest1-21systemd22:00:000
2770403991812,3cyclictest2861212-21cpuspeed_turbos23:25:140
2770403991811,5cyclictest2843870-21awk22:35:150
2770403991810,6cyclictest2777220-21ntpq19:25:190
277040399180,1cyclictest0-21swapper/022:14:200
2770423991715,1cyclictest3056-21PK-Backend22:53:293
2770423991715,1cyclictest3056-21PK-Backend19:28:213
2770423991710,5cyclictest2816165-21ntpq21:15:193
277041399179,6cyclictest436-20systemd-journal20:55:002
277041399174,8cyclictest2852414-21apt23:00:132
277041399172,14cyclictest1095-21runrttasks20:56:352
277041399172,10cyclictest2811976-21apt21:05:122
2770413991715,1cyclictest0-21swapper/223:43:042
2770413991715,1cyclictest0-21swapper/223:43:042
2770413991715,1cyclictest0-21swapper/223:22:152
2770413991715,1cyclictest0-21swapper/222:55:252
2770413991715,1cyclictest0-21swapper/222:16:042
2770413991715,1cyclictest0-21swapper/222:12:262
2770413991715,1cyclictest0-21swapper/221:57:192
2770413991715,1cyclictest0-21swapper/221:53:402
2770413991715,1cyclictest0-21swapper/221:40:362
2770413991715,1cyclictest0-21swapper/220:18:102
2770413991715,1cyclictest0-21swapper/220:13:022
2770413991715,1cyclictest0-21swapper/220:03:022
2770413991713,3cyclictest2772485-21munin-node22:05:132
2770413991712,3cyclictest27729792sleep219:15:142
2770413991711,5cyclictest2862545-21perf23:30:012
2770413991711,5cyclictest2796497-21awk20:20:142
2770413991711,4cyclictest436-20systemd-journal23:32:002
277040399172,11cyclictest2833045-21apt-get22:05:040
2770403991714,2cyclictest3056-21PK-Backend20:45:040
277040399171,12cyclictest2824128-21apt21:40:110
2770403991710,5cyclictest2841961-21ls22:30:140
2770423991614,1cyclictest3056-21PK-Backend22:21:393
2770423991613,2cyclictest3056-21PK-Backend20:04:443
277042399160,5cyclictest2848095-21chrt22:46:053
2770413991614,1cyclictest0-21swapper/219:43:082
2770413991612,3cyclictest2882756-21cpuspeed00:25:112
2770413991612,3cyclictest2774715-21/usr/sbin/munin19:20:232
2770413991610,5cyclictest2778652-21cat19:30:172
2770408991614,1cyclictest0-21swapper/120:55:041
277041399159,5cyclictest2826764-21sed21:45:212
277041399155,9cyclictest34-21ksoftirqd/200:05:002
2770413991510,3cyclictest2156-21gnome-shell22:27:092
277042399140,12cyclictest2786862-21cron19:55:003
2770413991412,1cyclictest0-21swapper/219:51:192
139099140,7rtkit-daemon2856000-21cpuspeed_turbos23:10:122
277042399120,1cyclictest3056-21PK-Backend20:20:453
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional