You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-11 - 14:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sun Jan 11, 2026 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1463372900,7sleep14497-21PK-Backend22:20:261
7637227645,22sleep10-21swapper/119:05:131
1547382750,4sleep20-21swapper/222:45:142
1736602740,8sleep10-21swapper/123:40:141
7773827254,13sleep00-21swapper/019:09:020
1479972660,3sleep10-21swapper/122:25:211
7775026540,19sleep20-21swapper/219:09:122
7634026542,14sleep30-21swapper/319:05:123
77958996117,12cyclictest438-20systemd-journal23:40:222
197399610,9rtkit-daemon1972-21rtkit-daemon19:49:471
197399600,9rtkit-daemon1972-21rtkit-daemon00:21:301
197399600,10rtkit-daemon1972-21rtkit-daemon20:20:570
77958995918,11cyclictest123011-21seq21:15:132
197399590,9rtkit-daemon1972-21rtkit-daemon23:24:421
1171712590,5sleep3411rcuc/320:59:553
77958995821,12cyclictest923-21dbus-daemon00:10:012
197399580,9rtkit-daemon1972-21rtkit-daemon23:47:441
197399580,9rtkit-daemon1972-21rtkit-daemon00:18:461
197399580,9rtkit-daemon1972-21rtkit-daemon00:18:461
197399580,8rtkit-daemon1972-21rtkit-daemon23:37:180
197399580,8rtkit-daemon1972-21rtkit-daemon19:15:210
77958995719,9cyclictest129833-21cut21:35:022
77958995619,11cyclictest923-21dbus-daemon22:15:022
77958995517,12cyclictest153235-21sed22:40:222
77958995516,11cyclictest132904-21seq21:43:032
77958995511,12cyclictest2751-21gdbus20:25:122
197399550,9rtkit-daemon1972-21rtkit-daemon20:16:430
197399550,8rtkit-daemon1972-21rtkit-daemon20:12:200
197399550,8rtkit-daemon1972-21rtkit-daemon19:28:480
197399550,8rtkit-daemon1972-21rtkit-daemon19:28:480
1241722550,5chrt0-21swapper/121:20:011
77958995419,8cyclictest99297-21grep20:05:292
77958995416,10cyclictest139987-21cron22:05:002
77958995416,10cyclictest135606-21sed21:50:212
77958995414,12cyclictest91526-21cat19:45:132
77958995412,12cyclictest1033-21gdbus20:20:002
77958995317,10cyclictest113146-21python320:45:312
77958995313,12cyclictest146369-21cat22:20:262
77958995313,11cyclictest111004-21cat20:40:202
77958995312,11cyclictest83681-21timerandwakeup19:20:282
197399530,8rtkit-daemon1972-21rtkit-daemon00:35:380
77958995213,11cyclictest85574-21seq19:25:552
77958995213,11cyclictest85574-21seq19:25:552
7795899521,13cyclictest119112-21cat21:05:022
77958995117,7cyclictest95536-21needreboot19:55:232
77958995112,10cyclictest137398-21missed_timers21:55:222
77958995111,10cyclictest147462-21grep22:25:132
197399510,8rtkit-daemon1972-21rtkit-daemon19:11:151
7795899509,11cyclictest78613-21if_err_enp2s019:10:192
77958995018,8cyclictest4497-21PK-Backend00:17:382
77958995018,8cyclictest4497-21PK-Backend00:17:382
77958995017,7cyclictest1137-21runrttasks22:06:492
77958995017,6cyclictest1137-21runrttasks21:08:292
77958995017,6cyclictest1137-21runrttasks21:08:292
77958995016,7cyclictest157870-21acpi22:55:102
77958995010,12cyclictest109634-21processes20:35:262
77961994918,26cyclictest126111-21cut21:25:023
77958994919,6cyclictest100091-21/usr/sbin/munin20:10:302
77958994916,7cyclictest178702-21cpu23:55:122
77958994916,6cyclictest169736-21latency_hist23:30:022
77958994916,6cyclictest1137-21runrttasks23:08:182
77958994916,6cyclictest1137-21runrttasks23:08:182
77958994914,8cyclictest151855-21switchtime22:35:352
77958994914,7cyclictest114474-21interrupts20:50:192
77958994912,9cyclictest177529-21missed_timers23:50:222
77958994912,10cyclictest193966-21seq00:35:592
77955994917,6cyclictest133549-21cstates21:45:141
77955994917,11cyclictest128930-21pluginstate21:30:261
197399490,42rtkit-daemon13-21ksoftirqd/023:47:560
77961994814,12cyclictest105327-21libinput-device20:25:113
77958994818,7cyclictest82184-21unin-run19:20:022
77958994816,11cyclictest125458-21sed21:20:312
77958994813,9cyclictest104502-21cat20:20:292
77958994812,11cyclictest121334-21basename21:10:182
77958994720,8cyclictest164541-21latency_hist23:15:002
77958994717,7cyclictest86207-21/usr/sbin/munin19:30:252
77958994716,7cyclictest1137-21runrttasks20:00:102
77958994715,7cyclictest107129-21cpu20:30:122
77955994727,5cyclictest0-21swapper/121:53:501
77958994631,11cyclictest89413-21cron19:40:002
77958994615,8cyclictest192500-21sshd00:34:212
7795899461,4cyclictest0-21swapper/222:52:042
77955994615,12cyclictest155863-21pool-udisksd22:49:571
197399450,8rtkit-daemon136691-21grep21:55:023
197399450,38rtkit-daemon185315-21fuser00:11:081
77958994414,7cyclictest187785-21tr00:20:132
77958994414,7cyclictest134234-21grep21:45:282
77958994412,8cyclictest0-21swapper/222:15:172
77955994433,7cyclictest169747-21basename23:30:011
77955994416,6cyclictest141rcu_preempt00:00:011
197399440,37rtkit-daemon2078-21gmain23:29:140
197399440,37rtkit-daemon2078-21gmain20:38:060
77958994313,10cyclictest34-21ksoftirqd/219:50:482
77958994313,10cyclictest34-21ksoftirqd/219:50:482
77951994327,12cyclictest152427-21munin-run22:40:000
77951994320,19cyclictest158631-21ntp_states22:55:240
1681842430,7sleep24497-21PK-Backend23:25:012
197399410,33rtkit-daemon77945-21cyclictest20:08:580
77961994017,19cyclictest108785-21/usr/sbin/munin20:35:183
77955994022,6cyclictest26-21ksoftirqd/122:41:191
77955994022,6cyclictest26-21ksoftirqd/122:07:141
197399400,8rtkit-daemon158611-21ntpq22:55:233
1769292400,16chrt13-21ksoftirqd/023:50:110
77961993923,12cyclictest923-21dbus-daemon23:50:013
77958993921,14cyclictest34-21ksoftirqd/200:10:092
77958993911,12cyclictest3692986-21snapd21:29:212
77955993921,5cyclictest26-21ksoftirqd/119:43:111
77951993929,6cyclictest13-21ksoftirqd/023:10:010
77951993929,6cyclictest13-21ksoftirqd/023:10:010
77951993921,14cyclictest181903-21unin-run00:05:000
77951993920,6cyclictest13-21ksoftirqd/022:25:140
77961993822,12cyclictest144470-21ntp_states22:15:263
77961993821,13cyclictest190241-21python300:25:283
77961993821,12cyclictest438-20systemd-journal23:42:593
77961993813,22cyclictest173205-21latency_hist23:40:013
7795899382,31cyclictest172095-21df23:35:132
7795599387,5cyclictest141rcu_preempt23:54:341
77955993822,12cyclictest85890-21lsb_release19:29:221
77955993822,12cyclictest85890-21lsb_release19:29:221
77955993821,5cyclictest26-21ksoftirqd/100:38:171
197399380,30rtkit-daemon26-21ksoftirqd/123:04:361
197399380,26rtkit-daemon0-21swapper/322:52:313
197399380,24rtkit-daemon0-21swapper/320:50:143
77961993727,6cyclictest80824-21tr19:15:183
77961993727,6cyclictest1137-21runrttasks23:34:053
77961993727,6cyclictest1137-21runrttasks23:34:053
77958993720,12cyclictest89799-21cat19:40:132
7795599375,11cyclictest26-21ksoftirqd/119:55:341
77955993724,9cyclictest80312-21mktemp19:15:031
77955993720,13cyclictest83167-21/usr/sbin/munin19:20:191
77955993720,13cyclictest121645-21ntpq21:10:231
7795599371,14cyclictest4497-21PK-Backend20:20:481
7795599371,13cyclictest4497-21PK-Backend20:32:401
77951993721,11cyclictest438-20systemd-journal21:24:550
77951993719,5cyclictest13-21ksoftirqd/023:20:140
77951993719,13cyclictest140534-21expr22:05:140
77951993719,13cyclictest134091-21ntp_states21:45:250
77951993717,15cyclictest166280-21/usr/sbin/munin23:20:010
197399370,25rtkit-daemon0-21swapper/320:44:363
929762360,7sleep04497-21PK-Backend19:50:020
7796199362,13cyclictest119191-21apt21:05:113
7796199362,13cyclictest119191-21apt21:05:113
77961993620,12cyclictest132476-21sensors_temp21:40:263
77961993619,13cyclictest160012-21fw_packets23:00:183
77955993625,7cyclictest125053-21needreboot21:20:221
77955993622,11cyclictest132102-21cut21:40:201
77955993621,11cyclictest139983-21cron22:05:001
77955993620,11cyclictest86480-21cat19:30:141
77955993619,12cyclictest189495-21grep00:25:141
77951993623,9cyclictest146924-21dump-pmu-power22:25:000
77951993620,11cyclictest150678-21cron22:35:000
77951993619,12cyclictest171170-21sshd23:31:220
77951993619,12cyclictest171170-21sshd23:31:210
197399360,30rtkit-daemon0-21swapper/000:16:340
197399360,30rtkit-daemon0-21swapper/000:16:340
197399360,26rtkit-daemon1972-21rtkit-daemon00:05:131
197399360,25rtkit-daemon0-21swapper/322:29:483
197399360,24rtkit-daemon0-21swapper/322:44:343
197399360,24rtkit-daemon0-21swapper/321:19:243
197399360,24rtkit-daemon0-21swapper/320:46:083
7796199352,29cyclictest193235-21hddtemp_smartct00:35:193
7796199352,13cyclictest107053-21/usr/sbin/munin20:30:163
77961993521,11cyclictest42-21ksoftirqd/320:13:203
77961993520,11cyclictest103811-21expr20:20:143
77961993519,12cyclictest191406-21expr00:30:143
77961993518,13cyclictest155396-21cat22:45:293
77961993510,21cyclictest146914-21apt-get22:25:003
77955993522,9cyclictest152427-21munin-run22:40:001
77955993520,11cyclictest181313-21sed00:00:281
77955993520,11cyclictest126672-21expr21:25:151
77955993518,5cyclictest26-21ksoftirqd/120:38:521
7795599351,14cyclictest4497-21PK-Backend20:52:071
77951993520,11cyclictest163302-21expr23:10:130
77951993519,12cyclictest139022-21/usr/sbin/munin22:00:210
77951993519,12cyclictest116139-21ls20:55:180
77951993519,11cyclictest190127-21ntpq00:25:260
77951993517,5cyclictest13-21ksoftirqd/021:20:000
197399350,27rtkit-daemon1972-21rtkit-daemon00:10:170
197399350,24rtkit-daemon0-21swapper/321:39:533
197399350,24rtkit-daemon0-21swapper/221:38:502
1617242350,15chrt26-21ksoftirqd/123:05:171
1617242350,15chrt26-21ksoftirqd/123:05:161
7796199346,24cyclictest141736-21awk22:10:003
77961993426,5cyclictest0-21swapper/300:05:303
77961993424,6cyclictest42-21ksoftirqd/321:35:013
77961993424,6cyclictest42-21ksoftirqd/319:20:173
77961993422,8cyclictest89456-21munin-run19:40:013
77961993421,8cyclictest98716-21df_inode20:05:163
77961993421,10cyclictest42-21ksoftirqd/323:09:133
77961993421,10cyclictest42-21ksoftirqd/323:09:133
77961993420,10cyclictest42-21ksoftirqd/319:15:003
77961993418,12cyclictest187316-21chrt00:19:323
77961993418,12cyclictest187316-21chrt00:19:323
77961993417,13cyclictest138194-21sshd21:59:213
7795899346,9cyclictest34-21ksoftirqd/223:45:022
77958993419,11cyclictest190767-21sshd00:29:222
7795599341,9cyclictest4497-21PK-Backend23:37:071
7795599341,9cyclictest4497-21PK-Backend20:11:471
77955993419,10cyclictest117349-21wc21:00:021
7795599341,8cyclictest4497-21PK-Backend22:57:031
7795599341,8cyclictest4497-21PK-Backend20:43:381
77955993417,13cyclictest89414-21cron19:40:001
7795599341,30cyclictest4497-21PK-Backend20:06:351
77951993427,4cyclictest0-21swapper/021:33:060
77951993426,5cyclictest0-21swapper/022:55:020
77951993423,7cyclictest191256-21cpuspeed_turbos00:30:130
77951993421,8cyclictest2143-21upowerd20:25:110
77951993419,11cyclictest923-21dbus-daemon00:10:010
7795199341,7cyclictest122392-21turbostat21:15:000
77951993413,6cyclictest141rcu_preempt19:35:190
7795199341,29cyclictest4497-21PK-Backend22:15:190
197399340,26rtkit-daemon1972-21rtkit-daemon19:11:360
197399340,25rtkit-daemon1972-21rtkit-daemon23:44:300
77961993322,8cyclictest2161-21gmain21:25:023
7796199332,26cyclictest89667-21/usr/sbin/munin19:40:163
7796199332,16cyclictest141rcu_preempt00:10:223
7796199332,12cyclictest96983-21sh20:00:163
7796199332,10cyclictest93548-21/usr/sbin/munin19:50:183
7796199332,10cyclictest93548-21/usr/sbin/munin19:50:173
77961993318,11cyclictest179258-21sed23:55:213
77961993318,11cyclictest139021-21sshd22:00:213
77961993317,12cyclictest177514-21wc23:50:213
77961993317,12cyclictest121327-21idleruntime21:10:183
77961993316,13cyclictest149980-21ntp_states22:30:243
77961993316,12cyclictest142155-21grep22:10:133
77961993315,14cyclictest118550-21users21:00:333
7796199331,29cyclictest4497-21PK-Backend00:24:333
77958993320,9cyclictest34-21ksoftirqd/222:30:172
7795899331,29cyclictest0-21swapper/223:33:392
7795899331,29cyclictest0-21swapper/223:33:392
7795599337,5cyclictest141rcu_preempt22:16:101
77955993319,10cyclictest163805-21ntp_kernel_pll_23:10:241
77955993318,11cyclictest156312-21cut22:50:131
77955993316,12cyclictest170859-21awk23:30:271
77955993316,12cyclictest170859-21awk23:30:271
77955993316,12cyclictest117792-21expr21:00:151
7795599331,5cyclictest0-21swapper/120:45:391
77955993315,15cyclictest4497-21PK-Backend19:55:001
77955993315,15cyclictest4497-21PK-Backend19:55:001
7795599331,29cyclictest0-21swapper/123:17:191
77951993324,6cyclictest77945-21cyclictest19:30:120
77951993323,6cyclictest13-21ksoftirqd/000:20:130
77951993322,5cyclictest13-21ksoftirqd/020:01:170
77951993319,10cyclictest923-21dbus-daemon20:00:000
7795199331,8cyclictest133075-21turbostat21:45:000
77951993318,11cyclictest127177-21wc21:25:270
77951993317,11cyclictest160701-21awk23:00:300
7795199331,29cyclictest0-21swapper/021:37:100
7795199331,27cyclictest438-20systemd-journal00:00:000
197399330,25rtkit-daemon4497-21PK-Backend19:40:080
7796199329,20cyclictest0-21swapper/323:15:263
7796199329,19cyclictest42-21ksoftirqd/319:48:083
77961993221,7cyclictest102329-21irqstats20:15:213
77961993218,10cyclictest180893-21uname00:00:193
77961993217,11cyclictest85444-21cat19:25:293
77961993217,11cyclictest85444-21cat19:25:293
77961993217,11cyclictest133626-21expr21:45:143
7796199321,28cyclictest0-21swapper/323:26:063
7796199321,28cyclictest0-21swapper/323:24:323
7796199321,12cyclictest151950-21timerandwakeup22:35:363
77958993219,9cyclictest181584-21seq00:01:302
77958993219,9cyclictest165739-21sed23:15:292
7795599327,5cyclictest141rcu_preempt21:09:101
7795599327,5cyclictest141rcu_preempt21:09:101
77955993223,6cyclictest149371-21tr22:30:121
77955993218,10cyclictest136962-21cut21:55:131
7795599321,10cyclictest4497-21PK-Backend21:35:171
77951993222,5cyclictest13-21ksoftirqd/021:55:010
77951993221,6cyclictest13-21ksoftirqd/020:45:440
77951993221,6cyclictest13-21ksoftirqd/020:30:250
77951993215,12cyclictest923-21dbus-daemon19:55:010
77951993215,12cyclictest923-21dbus-daemon19:55:010
7795199321,28cyclictest4497-21PK-Backend19:23:190
7795199321,28cyclictest0-21swapper/021:06:310
7795199321,28cyclictest0-21swapper/021:06:310
7796199312,25cyclictest95260-21diskstats19:55:173
7796199311,27cyclictest4497-21PK-Backend19:32:123
77955993124,4cyclictest4497-21PK-Backend20:15:381
7795599311,27cyclictest4497-21PK-Backend00:34:341
7795199316,21cyclictest153432-21grep22:40:250
7795199312,24cyclictest438-20systemd-journal22:00:000
77951993121,6cyclictest13-21ksoftirqd/020:50:090
77951993120,7cyclictest119007-21sort21:05:020
77951993116,11cyclictest142208-21ls22:10:140
7795199311,27cyclictest0-21swapper/020:42:460
7796199302,24cyclictest0-21swapper/323:14:573
77955993018,8cyclictest142618-21wc22:10:221
7795199301,9cyclictest0-21swapper/022:45:500
77958992918,8cyclictest4497-21PK-Backend20:56:162
77958992912,13cyclictest34-21ksoftirqd/223:03:532
77955992916,10cyclictest97240-21ls20:00:231
7795599271,10cyclictest4497-21PK-Backend20:25:101
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional