You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-05 - 16:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue May 05, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2286210185,9sleep00-21swapper/019:06:250
228329986,9sleep20-21swapper/219:06:242
182829853,41sleep10-21swapper/119:05:251
114829551,39sleep30-21swapper/319:05:203
268299260,26cyclictest0-21swapper/221:38:352
268299250,22cyclictest0-21swapper/223:51:412
75252220,0sleep00-21swapper/000:31:340
2688992219,2cyclictest39-21ksoftirqd/300:08:213
268899220,1cyclictest28063-21diskmemload21:49:273
268299220,21cyclictest0-21swapper/200:25:272
268299220,1cyclictest0-21swapper/221:29:332
267699220,5cyclictest6452-21kworker/1:300:37:561
128192220,0sleep30-21swapper/323:36:283
2688992118,1cyclictest2669-21cyclictest21:58:323
268299210,21cyclictest0-21swapper/200:34:542
267499210,4cyclictest0-21swapper/022:51:540
268899202,7cyclictest0-21swapper/319:30:143
268899200,20cyclictest0-21swapper/322:52:243
268299200,20cyclictest0-21swapper/222:59:412
268299200,19cyclictest0-21swapper/223:38:402
2676992018,1cyclictest0-21swapper/100:00:141
267499200,19cyclictest0-21swapper/019:45:120
267699190,19cyclictest0-21swapper/121:11:161
2682991817,1cyclictest0-21swapper/200:05:492
268299180,2cyclictest0-21swapper/223:13:072
267699180,16cyclictest0-21swapper/121:23:571
2688991717,0cyclictest0-21swapper/322:41:063
2682991715,1cyclictest0-21swapper/223:05:242
268299171,2cyclictest0-21swapper/223:20:132
268299170,17cyclictest0-21swapper/223:56:582
2676991715,1cyclictest0-21swapper/123:37:301
267699171,15cyclictest0-21swapper/123:10:061
267699170,1cyclictest0-21swapper/121:52:011
267699170,16cyclictest0-21swapper/122:10:231
267499175,6cyclictest18328-21munin-run20:50:000
267499173,10cyclictest21259-21sensors00:20:210
2674991714,2cyclictest0-21swapper/023:27:040
268899163,7cyclictest0-21swapper/319:10:143
2688991616,0cyclictest0-21swapper/323:41:063
2688991616,0cyclictest0-21swapper/323:41:053
268299161,14cyclictest6223-21ssh21:55:142
268299160,16cyclictest0-21swapper/222:46:112
268299160,16cyclictest0-21swapper/222:43:202
268299160,16cyclictest0-21swapper/222:02:572
268299160,15cyclictest0-21swapper/200:02:292
268299160,14cyclictest0-21swapper/221:13:592
268299160,0cyclictest0-21swapper/223:31:582
267699165,3cyclictest30689-21/usr/sbin/munin21:30:221
267699162,13cyclictest28063-21diskmemload21:41:381
2676991614,1cyclictest25-21ksoftirqd/121:26:491
2676991613,2cyclictest13899-21/usr/sbin/munin22:00:161
267699160,16cyclictest0-21swapper/123:17:421
267699160,13cyclictest0-21swapper/121:55:281
267499166,5cyclictest28625-21perf21:30:000
2674991610,4cyclictest15634-21ls22:00:270
107622160,1sleep339-21ksoftirqd/322:17:263
2688991514,1cyclictest0-21swapper/322:37:433
268299159,3cyclictest225-21systemd-journal21:34:552
268299155,4cyclictest1436-21sendmail-mta21:41:242
268299151,12cyclictest8909-21wc20:30:012
268299150,1cyclictest0-21swapper/222:07:372
268299150,15cyclictest0-21swapper/222:11:492
268299150,15cyclictest0-21swapper/200:17:162
268299150,0cyclictest0-21swapper/222:15:062
267699159,6cyclictest1646-21apache223:34:431
2676991513,2cyclictest25-21ksoftirqd/122:23:021
2676991513,1cyclictest0-21swapper/122:35:431
267699151,13cyclictest0-21swapper/121:45:151
267699150,2cyclictest0-21swapper/121:15:151
267499151,13cyclictest0-21swapper/022:22:430
268899144,6cyclictest12132-21cron19:29:593
2688991414,0cyclictest0-21swapper/322:09:403
2688991411,2cyclictest15475-21wc22:00:243
2688991410,3cyclictest28173-21sed23:45:263
268899140,13cyclictest17167-21cat20:45:183
268299141,7cyclictest4234-21ntpq00:10:202
2682991414,0cyclictest0-21swapper/220:05:132
2682991410,3cyclictest31968-21grep22:30:152
268299140,12cyclictest0-21swapper/223:25:082
267699149,5cyclictest1646-21apache223:03:101
267699145,6cyclictest31648-21/usr/sbin/munin20:10:111
267699143,4cyclictest0-21swapper/123:25:131
2676991412,1cyclictest0-21swapper/123:41:031
2676991412,1cyclictest0-21swapper/123:41:021
2676991410,3cyclictest26191-21grep19:55:251
2676991410,3cyclictest13127-21tail19:30:161
267699140,2cyclictest141rcu_preempt20:40:141
267699140,10cyclictest0-21swapper/120:05:141
2674991410,2cyclictest6895-21grep21:35:270
2674991410,2cyclictest4038-21wc00:10:190
267499140,8cyclictest94-21kswapd021:55:010
268899139,3cyclictest4526-21grep20:20:133
268899134,7cyclictest19764-21apt-get19:45:133
2688991310,2cyclictest9785-21grep20:30:173
2688991310,2cyclictest28597-21df00:25:163
2688991310,2cyclictest19306-21grep20:50:183
2688991310,2cyclictest17085-21grep23:00:173
2688991310,2cyclictest15617-21grep22:20:173
2688991310,2cyclictest12575-21grep20:35:193
268899130,1cyclictest0-21swapper/323:20:473
268899130,12cyclictest29244-21tr21:30:143
268899130,0cyclictest0-21swapper/323:56:393
2682991310,2cyclictest31936-21grep20:10:142
2682991310,2cyclictest20356-21grep19:45:162
2682991310,2cyclictest16387-21grep19:35:262
268299130,13cyclictest0-21swapper/200:24:162
267699138,3cyclictest25-21ksoftirqd/100:24:301
2676991311,1cyclictest0-21swapper/123:20:151
2676991310,2cyclictest6021-21sed19:15:171
2676991310,2cyclictest25825-21head22:45:251
2676991310,2cyclictest24099-21grep22:05:271
2676991310,2cyclictest21934-21grep19:50:011
2676991310,2cyclictest10736-21tail19:25:171
267699130,2cyclictest0-21swapper/122:32:401
267499139,3cyclictest9392-21tr22:55:230
267499139,3cyclictest10604-21cat23:35:180
267499139,2cyclictest9162-21grep20:30:140
267499137,3cyclictest20778-21rm23:02:260
267499131,7cyclictest3917-21munin-run21:15:000
267499131,2cyclictest0-21swapper/022:42:350
2674991310,2cyclictest22191-21tr21:45:150
2674991310,2cyclictest13184-21ssh21:20:150
267499130,12cyclictest6680-21ssh21:55:170
267499130,11cyclictest0-21swapper/019:35:140
268899129,2cyclictest7304-21chrt21:35:423
268899129,2cyclictest5432-21grep00:30:213
268899129,2cyclictest4712-21grep21:15:143
268899129,2cyclictest3758-21chrt21:14:553
268899129,2cyclictest28038-21ssh23:25:553
268899129,2cyclictest2422-21head20:15:183
268899129,2cyclictest2340-21grep23:10:253
268899129,2cyclictest21229-21sensors_fan00:20:203
268899129,2cyclictest20424-21cat00:00:273
268899129,2cyclictest13470-21grep21:40:143
268899129,2cyclictest11759-21grep00:15:153
268899128,3cyclictest6453-21ls19:15:213
268899128,3cyclictest32273-21sort20:10:163
268899128,3cyclictest31928-21ssh23:09:463
268899128,3cyclictest31633-21grep22:49:593
268899128,3cyclictest2918-21head23:50:153
268899128,3cyclictest22143-21grep21:25:193
268899128,3cyclictest12983-21grep21:20:153
268899128,2cyclictest25666-21cron21:04:593
268299129,2cyclictest9755-21cron19:25:002
268299129,2cyclictest7532-21cut22:35:122
268299129,2cyclictest5785-21ls21:15:212
268299129,2cyclictest26340-21cut23:45:142
268299129,2cyclictest22865-21head19:50:182
268299129,2cyclictest19999-21chrt23:40:362
268299129,2cyclictest19999-21chrt23:40:352
268299129,2cyclictest19691-21uname20:50:202
268299129,2cyclictest1915-21grep20:15:152
268299129,2cyclictest14059-21sed19:30:262
268299129,2cyclictest10318-21tr19:25:142
2682991210,1cyclictest9134-21expr23:15:142
2682991210,1cyclictest12915-21ssh21:20:152
268299120,3cyclictest0-21swapper/222:51:502
268299120,11cyclictest22276-21uname20:55:202
268299120,0cyclictest0-21swapper/221:49:272
267699129,2cyclictest7095-21sed22:15:161
267699129,2cyclictest5884-21grep20:20:251
267699129,2cyclictest4200-21fgrep19:10:231
267699129,2cyclictest3501-21dpkg00:30:001
267699129,2cyclictest3104-21chrt22:51:291
267699129,2cyclictest27672-21wc23:45:221
267699129,2cyclictest25049-21wc21:00:261
267699129,2cyclictest24306-21sh22:25:221
267699129,2cyclictest2376-21grep20:15:181
267699129,2cyclictest23370-21grep19:50:221
267699129,2cyclictest19226-21cut20:50:171
267699129,2cyclictest18110-21tail19:40:151
267699128,3cyclictest28539-21grep20:00:271
267699128,2cyclictest2381-21tr23:50:131
267699127,4cyclictest10795-21taskset20:30:481
267699126,2cyclictest21428-21ssh22:43:281
2676991211,1cyclictest0-21swapper/123:09:471
267499129,2cyclictest6681-21sort19:15:250
267499129,2cyclictest453-21grep23:10:140
267499129,2cyclictest27605-21grep00:05:200
267499129,2cyclictest25946-21grep23:05:240
267499129,2cyclictest13243-21cut21:40:140
267499129,2cyclictest11123-21grep19:25:200
267499128,3cyclictest8254-21grep19:20:170
267499126,5cyclictest1646-21apache222:46:580
267499125,5cyclictest1646-21apache222:38:030
267499123,5cyclictest225-21systemd-journal21:05:000
267499120,2cyclictest24181-21memory22:25:210
267499120,1cyclictest0-21swapper/000:39:040
165932120,0sleep00-21swapper/023:58:490
268899118,2cyclictest9756-21cron19:25:003
268899118,2cyclictest8758-21sh20:30:013
268899118,2cyclictest8525-21ssh23:15:093
268899118,2cyclictest4801-21rm23:31:383
268899118,2cyclictest3275-21sh22:32:173
268899118,2cyclictest31814-21ssh21:50:373
268899118,2cyclictest26759-21ssh22:26:503
268899118,2cyclictest26174-21sed19:55:253
268899118,2cyclictest22798-21tune2fs19:50:173
268899118,2cyclictest18105-21head19:40:153
268899118,2cyclictest12757-21ssh00:35:153
268899118,2cyclictest10121-21chrt22:55:333
268899115,2cyclictest225-21systemd-journal00:10:033
268899110,10cyclictest30691-21grep22:10:143
268299118,2cyclictest27084-21tr21:05:232
268299118,2cyclictest23306-21cut22:25:162
268299118,2cyclictest19311-21ssh23:01:162
268299118,2cyclictest18034-21sort19:40:152
268299118,2cyclictest16870-21ssh22:20:302
268299118,2cyclictest1605-21apache219:55:112
268299118,2cyclictest12582-21grep00:35:142
268299117,3cyclictest24274-21head21:00:162
268299110,10cyclictest0-21swapper/220:20:122
267699119,1cyclictest28063-21diskmemload00:11:361
267699118,2cyclictest7949-21sh00:31:541
267699118,2cyclictest28086-21cut00:05:231
267699118,2cyclictest16889-21dump-pmu-power19:40:001
267699118,2cyclictest12566-21ssh22:57:301
267699118,2cyclictest11508-21cut20:35:121
267699116,2cyclictest0-21swapper/100:18:551
267699115,4cyclictest6287-21ssh21:35:211
267699110,1cyclictest141rcu_preempt23:56:261
267499119,1cyclictest23843-21grep19:50:270
267499119,1cyclictest13-21ksoftirqd/021:05:160
267499118,2cyclictest9043-21ssh22:16:030
267499118,2cyclictest6718-21awk22:35:010
267499118,2cyclictest5966-21cut21:15:220
267499118,2cyclictest4878-21cut20:20:160
267499118,2cyclictest483-21ssh21:31:530
267499118,2cyclictest3666-21grep23:50:210
267499118,2cyclictest31184-21head22:10:160
267499118,2cyclictest30225-21sh00:25:390
267499118,2cyclictest30150-21ssh23:46:590
267499118,2cyclictest2958-21aten2_r7power_p19:10:150
267499118,2cyclictest25419-21ssh22:06:260
267499118,2cyclictest2263-21sort20:15:170
267499118,2cyclictest20021-21sed20:50:250
267499118,2cyclictest1901-21ssh23:30:150
267499118,2cyclictest18454-21ssh00:00:060
267499118,2cyclictest17673-21tr19:40:140
267499118,2cyclictest17084-21grep23:20:140
267499118,2cyclictest16116-21ssh00:17:340
267499118,2cyclictest13350-21cut19:30:180
267499118,2cyclictest10477-21grep23:15:250
267499117,3cyclictest30514-21head20:05:220
267499117,3cyclictest28516-21grep20:00:270
267499117,2cyclictest18214-21sh23:40:140
267499117,2cyclictest18214-21sh23:40:130
268899108,1cyclictest29581-21cat20:05:143
268899108,1cyclictest167202sleep319:38:023
268899107,2cyclictest21569-21expr20:55:153
268299108,1cyclictest8997-21basename20:30:112
268299108,1cyclictest28392sleep219:10:042
268299108,1cyclictest27340-21expr20:00:162
268299108,1cyclictest18061-21chrt20:46:532
268299108,1cyclictest14237-21expr20:40:142
268299107,2cyclictest30479-21which21:50:172
268299107,2cyclictest12161-21grep20:35:152
267699108,1cyclictest22861-21head20:55:271
267699108,1cyclictest17845-21grep20:45:261
267699107,2cyclictest8339-21awk20:25:261
267699107,2cyclictest27655-21sed21:05:281
267699103,5cyclictest7901-21grep19:20:161
267499108,1cyclictest32641-21cat20:10:180
267499108,1cyclictest21459-21cut20:55:150
267499108,1cyclictest13347-21chrt20:37:320
267499107,2cyclictest6927-21grep20:25:130
267499107,2cyclictest25479-21sed19:55:180
26889997,1cyclictest27031-21uname21:05:233
26889997,1cyclictest14341-21sed20:40:153
26889996,2cyclictest28025-21tr20:00:223
26829997,1cyclictest5256-21apache_accesses19:15:112
26749997,1cyclictest13956-21grep20:40:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional