You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 14:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Thu Apr 16, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
346433422410,19sleep126-21ksoftirqd/119:05:271
353689421190,3sleep10-21swapper/122:30:221
356076821180,4sleep13056-21PK-Backend23:40:121
353219421120,3sleep20-21swapper/222:20:012
346398321010,7sleep33463987-21latency19:05:203
35447332920,3sleep20-21swapper/222:55:012
346486026646,14sleep00-21swapper/019:09:240
346457226637,23sleep20-21swapper/219:05:402
35787552510,2sleep13056-21PK-Backend00:30:201
35575212510,3sleep00-21swapper/023:30:160
34710962430,3sleep10-21swapper/119:25:121
35679132350,4sleep33056-21PK-Backend00:00:103
3465078993110,5cyclictest3561411-21cat23:40:202
346507899309,5cyclictest3544347-21sed22:55:002
3465078993010,5cyclictest3508907-21sshd21:13:132
346507899298,5cyclictest3569693-21expr00:05:112
346507899298,4cyclictest0-21swapper/220:35:212
3465078992911,5cyclictest34-21ksoftirqd/222:05:162
346507899291,11cyclictest3509335-21apt21:15:112
3465078992910,3cyclictest3473690-21sh19:30:192
139099290,5rtkit-daemon1389-21rtkit-daemon23:51:482
3465076992823,4cyclictest26-21ksoftirqd/121:25:011
139099280,4rtkit-daemon1389-21rtkit-daemon23:18:312
139099280,4rtkit-daemon1389-21rtkit-daemon23:02:592
346507899279,4cyclictest3497320-21cpuspeed_turbos20:40:112
3465078992721,5cyclictest0-21swapper/219:29:542
3465078992711,3cyclictest3486554-21idleruntime-cro20:10:002
3465078992711,2cyclictest3481584-21/usr/sbin/munin19:55:192
346507899261,7cyclictest3576283-21apt00:25:102
346507899261,10cyclictest3526916-21munin-run22:05:002
3465076992611,13cyclictest3528675-21unin-run22:10:001
139099260,4rtkit-daemon1389-21rtkit-daemon23:31:272
139099260,4rtkit-daemon1389-21rtkit-daemon22:48:182
139099260,4rtkit-daemon1389-21rtkit-daemon21:36:002
139099260,4rtkit-daemon1389-21rtkit-daemon00:29:583
346507899251,7cyclictest3511069-21apt21:20:132
346507899250,18cyclictest0-21swapper/220:49:372
346507899250,18cyclictest0-21swapper/220:49:372
346507299259,14cyclictest3574273-21cron00:20:000
346508299240,22cyclictest3474685-21grep19:35:113
3465078992418,5cyclictest0-21swapper/221:52:242
346507899240,1cyclictest0-21swapper/223:06:182
3465076992411,9cyclictest1-21systemd19:47:131
3465076992411,9cyclictest1-21systemd19:47:131
3465072992415,7cyclictest3567115-21munin-run00:00:000
346508299230,21cyclictest3508079-21irqstats21:10:163
346507899235,3cyclictest0-21swapper/221:25:182
3465078992319,2cyclictest3521723-21munin-run21:50:002
3465078992315,6cyclictest883-21dbus-daemon00:25:002
3465076992315,6cyclictest3521979-21apt21:50:081
346507299231,20cyclictest3480741-21python319:50:200
139099230,4rtkit-daemon1389-21rtkit-daemon20:28:553
3465082992219,2cyclictest3056-21PK-Backend20:54:343
3465082992219,2cyclictest3056-21PK-Backend19:48:373
3465082992219,2cyclictest3056-21PK-Backend19:48:363
346508299220,3cyclictest3505050-21grep21:00:223
3465078992220,1cyclictest0-21swapper/223:28:492
3465078992220,1cyclictest0-21swapper/222:33:402
3465078992211,9cyclictest3465767-21grep19:10:162
346507899220,21cyclictest0-21swapper/219:42:452
3465076992212,8cyclictest3532195-21latency_hist22:20:011
3465072992220,1cyclictest0-21swapper/000:34:230
139099220,17rtkit-daemon3466952-21munin-node21:38:253
35659782210,4sleep33056-21PK-Backend23:55:133
34988412210,3chrt0-21swapper/120:45:001
3465082992119,1cyclictest3056-21PK-Backend22:41:083
3465082992119,1cyclictest3056-21PK-Backend21:25:363
3465082992119,1cyclictest3056-21PK-Backend20:18:253
3465082992119,1cyclictest3056-21PK-Backend19:52:293
3465082992119,1cyclictest3056-21PK-Backend19:13:423
3465082992119,1cyclictest0-21swapper/323:21:223
3465082992119,1cyclictest0-21swapper/323:18:453
3465082992119,1cyclictest0-21swapper/323:12:443
3465082992119,1cyclictest0-21swapper/322:07:413
3465082992119,1cyclictest0-21swapper/321:45:233
3465082992119,1cyclictest0-21swapper/320:01:273
3465082992119,1cyclictest0-21swapper/319:34:103
3465082992119,1cyclictest0-21swapper/319:23:363
3465082992119,1cyclictest0-21swapper/300:38:083
3465082992118,2cyclictest0-21swapper/319:58:233
3465082992116,3cyclictest3571948-21python300:10:203
346508299211,18cyclictest3497581-21/usr/sbin/munin20:40:133
346508299210,20cyclictest3056-21PK-Backend00:17:123
346508299210,20cyclictest0-21swapper/321:21:313
3465078992119,1cyclictest0-21swapper/223:56:342
3465078992119,1cyclictest0-21swapper/222:20:092
3465078992119,1cyclictest0-21swapper/221:43:382
3465078992119,1cyclictest0-21swapper/220:27:342
3465078992119,1cyclictest0-21swapper/219:37:512
3465078992119,1cyclictest0-21swapper/219:17:442
3465078992115,4cyclictest3540778-21sh22:45:002
3465078992115,4cyclictest2156-21gnome-shell19:51:092
3465078992113,6cyclictest3469205-21apt19:20:102
346507899210,20cyclictest3503075-21strings20:55:182
346507899210,20cyclictest0-21swapper/221:01:542
346507899210,20cyclictest0-21swapper/220:04:352
3465076992119,1cyclictest0-21swapper/122:36:081
3465076992117,2cyclictest34949462sleep120:31:541
3465076992116,3cyclictest34746432sleep119:35:041
3465076992110,4cyclictest26-21ksoftirqd/123:33:341
346507699210,20cyclictest0-21swapper/119:13:501
3465072992119,1cyclictest0-21swapper/023:52:460
3465072992119,1cyclictest0-21swapper/023:08:590
3465072992119,1cyclictest0-21swapper/022:56:560
3465072992119,1cyclictest0-21swapper/022:29:110
3465072992119,1cyclictest0-21swapper/021:39:380
3465072992119,1cyclictest0-21swapper/021:17:120
3465072992119,1cyclictest0-21swapper/021:02:390
3465072992119,1cyclictest0-21swapper/020:19:540
3465072992119,1cyclictest0-21swapper/020:00:200
3465072992119,1cyclictest0-21swapper/019:56:140
3465072992119,1cyclictest0-21swapper/019:47:300
3465072992119,1cyclictest0-21swapper/019:47:300
3465072992119,1cyclictest0-21swapper/019:26:060
3465072992119,1cyclictest0-21swapper/019:18:370
3465072992119,1cyclictest0-21swapper/000:35:250
3465072992119,1cyclictest0-21swapper/000:25:520
3465072992113,6cyclictest883-21dbus-daemon19:20:470
3465072992113,6cyclictest436-20systemd-journal22:49:550
346507299210,20cyclictest0-21swapper/022:33:580
346507299210,20cyclictest0-21swapper/022:05:050
346507299210,20cyclictest0-21swapper/020:27:580
346507299210,20cyclictest0-21swapper/019:43:370
346507299210,20cyclictest0-21swapper/019:12:220
346507299210,19cyclictest0-21swapper/019:32:490
3465082992018,1cyclictest3056-21PK-Backend22:36:203
3465082992018,1cyclictest3056-21PK-Backend20:21:553
3465082992018,1cyclictest3056-21PK-Backend19:41:063
346508299201,7cyclictest3560885-21irqstats23:40:153
346508299201,17cyclictest3562141-21apt23:45:103
346508299200,1cyclictest3056-21PK-Backend21:30:433
346508299200,1cyclictest3056-21PK-Backend20:47:183
346508299200,1cyclictest3056-21PK-Backend20:47:173
346508299200,19cyclictest3056-21PK-Backend23:28:593
346508299200,19cyclictest0-21swapper/300:22:533
3465078992018,1cyclictest0-21swapper/220:23:492
3465078992018,1cyclictest0-21swapper/220:13:132
3465078992018,1cyclictest0-21swapper/219:46:372
3465078992018,1cyclictest0-21swapper/219:46:372
3465078992018,1cyclictest0-21swapper/200:36:352
3465078992018,1cyclictest0-21swapper/200:18:172
346507899200,19cyclictest3056-21PK-Backend21:30:332
346507899200,19cyclictest0-21swapper/223:47:142
346507899200,19cyclictest0-21swapper/223:37:072
346507899200,19cyclictest0-21swapper/220:32:062
346507699209,4cyclictest26-21ksoftirqd/119:33:221
346507699209,3cyclictest26-21ksoftirqd/121:03:011
346507699206,3cyclictest141rcu_preempt23:56:231
3465076992018,1cyclictest0-21swapper/122:55:211
3465076992018,1cyclictest0-21swapper/122:22:151
3465076992018,1cyclictest0-21swapper/120:19:131
3465076992018,1cyclictest0-21swapper/100:00:101
3465076992016,2cyclictest35214622chrt21:46:551
3465076992015,3cyclictest35511872chrt23:11:281
3465076992012,6cyclictest3509057-21cron21:15:001
3465076992010,9cyclictest3507318-21cron21:10:001
3465076992010,4cyclictest26-21ksoftirqd/120:36:231
3465076992010,4cyclictest26-21ksoftirqd/119:56:001
346507699200,5cyclictest26-21ksoftirqd/100:25:001
346507699200,19cyclictest0-21swapper/122:10:191
3465072992018,1cyclictest0-21swapper/023:36:100
3465072992018,1cyclictest0-21swapper/023:29:410
3465072992018,1cyclictest0-21swapper/022:52:440
3465072992018,1cyclictest0-21swapper/022:00:180
3465072992018,1cyclictest0-21swapper/020:59:210
3465072992018,1cyclictest0-21swapper/019:35:530
3465072992018,1cyclictest0-21swapper/000:22:000
346507299200,1cyclictest0-21swapper/000:08:210
346507299200,19cyclictest0-21swapper/023:22:520
346507299200,19cyclictest0-21swapper/023:11:360
346507299200,19cyclictest0-21swapper/022:15:390
346507299200,19cyclictest0-21swapper/021:28:070
346507299200,19cyclictest0-21swapper/020:51:370
346507299200,19cyclictest0-21swapper/000:01:520
35239652190,3sleep00-21swapper/021:55:130
35028872190,2sleep33056-21PK-Backend20:55:173
3465082991917,1cyclictest3056-21PK-Backend23:02:293
3465082991917,1cyclictest3056-21PK-Backend22:33:093
3465082991917,1cyclictest3056-21PK-Backend21:50:583
3465082991917,1cyclictest3056-21PK-Backend21:05:013
3465082991917,1cyclictest0-21swapper/321:43:493
3465082991917,1cyclictest0-21swapper/320:39:193
3465082991916,2cyclictest3056-21PK-Backend19:15:323
346508299190,1cyclictest3056-21PK-Backend20:12:563
346508299190,1cyclictest0-21swapper/323:33:253
346508299190,1cyclictest0-21swapper/321:15:293
346508299190,1cyclictest0-21swapper/300:05:373
346508299190,18cyclictest3056-21PK-Backend23:52:523
346508299190,18cyclictest3056-21PK-Backend22:28:233
346508299190,18cyclictest0-21swapper/319:26:263
346507899198,9cyclictest436-20systemd-journal22:15:002
3465078991917,1cyclictest0-21swapper/223:10:442
3465078991917,1cyclictest0-21swapper/200:14:252
346507899190,1cyclictest0-21swapper/200:02:012
346507899190,18cyclictest0-21swapper/221:07:552
346507699199,4cyclictest26-21ksoftirqd/122:42:401
346507699198,4cyclictest26-21ksoftirqd/121:32:161
346507699195,4cyclictest26-21ksoftirqd/121:25:201
3465076991917,1cyclictest0-21swapper/123:06:361
3465076991917,1cyclictest0-21swapper/122:48:521
3465076991917,1cyclictest0-21swapper/120:46:241
3465076991917,1cyclictest0-21swapper/120:46:241
3465076991915,2cyclictest35598242sleep123:36:071
3465076991915,2cyclictest35178152sleep121:37:571
3465076991914,3cyclictest35811592chrt00:39:301
3465076991914,3cyclictest3547996-21turbostat.cron23:05:001
3465076991914,3cyclictest34689232sleep119:19:591
346507699190,5cyclictest3056-21PK-Backend00:05:091
346507699190,1cyclictest0-21swapper/120:23:391
346507699190,1cyclictest0-21swapper/120:11:291
3465072991917,1cyclictest0-21swapper/022:43:450
3465072991917,1cyclictest0-21swapper/021:42:000
3465072991917,1cyclictest0-21swapper/021:24:360
346507299190,8cyclictest3561859-21cron23:45:000
346507299190,1cyclictest0-21swapper/020:25:010
346507299190,1cyclictest0-21swapper/000:13:140
346507299190,18cyclictest0-21swapper/022:24:250
346508299186,11cyclictest42-21ksoftirqd/320:30:203
3465082991815,2cyclictest3056-21PK-Backend22:55:403
3465082991810,6cyclictest436-20systemd-journal20:10:003
3465082991810,6cyclictest3578599-21memory00:30:193
3465082991810,6cyclictest3544242-21cron22:55:003
346508299180,1cyclictest3056-21PK-Backend22:24:243
346507899183,13cyclictest3537716-21cpuspeed_turbos22:35:122
3465078991816,1cyclictest0-21swapper/222:26:302
3465078991816,1cyclictest0-21swapper/220:53:232
346507699185,11cyclictest3477873-21sh19:45:001
346507699183,8cyclictest26-21ksoftirqd/100:17:121
3465076991814,3cyclictest251rcuc/122:00:001
3465076991813,4cyclictest26-21ksoftirqd/119:55:001
3465076991812,5cyclictest3564809-21sed23:50:211
3465076991811,5cyclictest3525449-21apt22:00:121
3465076991811,5cyclictest3500848-21grep20:50:121
3465076991810,2cyclictest141rcu_preempt22:26:041
346507699180,5cyclictest3056-21PK-Backend23:20:091
346507699180,2cyclictest3056-21PK-Backend20:00:211
3465072991816,1cyclictest3056-21PK-Backend21:10:040
3465072991816,1cyclictest3056-21PK-Backend20:35:420
3465072991816,1cyclictest0-21swapper/021:33:280
3465072991812,5cyclictest13-21ksoftirqd/022:10:150
3465072991812,5cyclictest13-21ksoftirqd/020:45:130
3465072991812,5cyclictest13-21ksoftirqd/020:45:120
346507299180,16cyclictest13-21ksoftirqd/023:00:200
3465082991715,1cyclictest3056-21PK-Backend23:05:063
346508299170,15cyclictest35601152sleep323:39:533
3465078991713,3cyclictest3056-21PK-Backend20:15:302
346507699179,6cyclictest3503109-21ntp_kernel_err20:55:181
346507699177,2cyclictest141rcu_preempt21:19:371
346507699174,2cyclictest141rcu_preempt23:15:581
3465076991713,2cyclictest35633892chrt23:47:251
346507299179,6cyclictest3522347-21df_inode21:50:110
346507299179,6cyclictest3505860-21awk21:05:110
3465072991715,1cyclictest0-21swapper/020:34:070
3465072991714,2cyclictest3498667-21perf20:45:010
3465072991711,5cyclictest3538623-21ls22:35:220
3465072991710,5cyclictest883-21dbus-daemon21:50:000
35297662160,3sleep33056-21PK-Backend22:10:193
3465078991613,2cyclictest3525176-21dump-pmu-power22:00:002
346507899161,13cyclictest3578689-21grep00:30:192
3465078991610,5cyclictest3554597-21seq23:20:462
346507699167,3cyclictest26-21ksoftirqd/122:53:111
346507699161,1cyclictest141rcu_preempt20:25:201
346507699160,3cyclictest3056-21PK-Backend19:25:001
3465072991614,1cyclictest0-21swapper/023:45:300
3465072991614,1cyclictest0-21swapper/020:09:460
3465072991613,2cyclictest1095-21runrttasks20:12:200
3465082991513,1cyclictest3056-21PK-Backend22:00:313
3465076991513,1cyclictest3056-21PK-Backend00:26:121
3465076991511,2cyclictest26-21ksoftirqd/100:13:201
3465076991510,4cyclictest26-21ksoftirqd/123:30:001
346507699150,7cyclictest3486408-21chrt20:08:311
3465072991513,1cyclictest0-21swapper/023:16:020
346508299140,3cyclictest978-21snmpd22:15:283
346507699140,2cyclictest3056-21PK-Backend21:40:101
346508299139,3cyclictest3056-21PK-Backend21:55:243
346508299134,2cyclictest141rcu_preempt22:45:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional