You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-10 - 14:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Apr 10, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
939528875,9sleep20-21swapper/219:09:522
797028467,12sleep00-21swapper/019:05:170
916728168,9sleep30-21swapper/319:06:563
921227759,14sleep10-21swapper/119:07:311
948999340,5cyclictest28955-21apt23:40:001
948999330,3cyclictest12319-21ntp_kernel_pll_23:10:201
948999314,26cyclictest2081-21diskmemload23:26:291
948999311,2cyclictest0-21swapper/121:50:001
949499300,27cyclictest0-21swapper/221:43:212
948999274,22cyclictest24777-21cut22:20:181
9489992725,2cyclictest25-21ksoftirqd/121:42:441
948999270,7cyclictest4600-21ssh21:30:181
948999270,2cyclictest0-21swapper/100:06:581
948999270,1cyclictest0-21swapper/121:15:121
948599270,2cyclictest0-21swapper/023:34:210
949499260,25cyclictest0-21swapper/222:32:022
948999261,2cyclictest2081-21diskmemload22:05:011
948999260,4cyclictest0-21swapper/100:10:181
948999254,1cyclictest0-21swapper/100:15:271
948999253,1cyclictest2081-21diskmemload23:52:531
948999250,5cyclictest23450-21fschecks_time20:45:161
949499241,18cyclictest10994-21/usr/sbin/munin21:15:122
949499240,22cyclictest0-21swapper/222:41:382
948999242,3cyclictest0-21swapper/121:39:221
948999240,4cyclictest0-21swapper/122:33:291
948999240,23cyclictest0-21swapper/123:40:141
949699230,22cyclictest0-21swapper/322:49:233
948999234,5cyclictest2081-21diskmemload22:58:031
948999233,1cyclictest2081-21diskmemload22:05:511
948999233,1cyclictest2081-21diskmemload21:11:591
948999231,1cyclictest2081-21diskmemload23:06:071
948999230,23cyclictest0-21swapper/100:04:541
948999230,22cyclictest0-21swapper/122:52:451
948999230,1cyclictest0-21swapper/100:20:161
949699220,22cyclictest0-21swapper/321:27:083
949499220,22cyclictest0-21swapper/222:19:582
949499220,20cyclictest0-21swapper/223:25:562
948999227,7cyclictest17299-21rm22:35:161
948999221,5cyclictest8358-21/usr/sbin/munin00:25:281
948999221,4cyclictest0-21swapper/123:31:321
948999221,3cyclictest0-21swapper/120:25:281
948999220,1cyclictest0-21swapper/122:18:391
948999220,1cyclictest0-21swapper/122:14:311
948599220,5cyclictest0-21swapper/023:13:250
949699212,1cyclictest0-21swapper/321:20:303
949699210,3cyclictest0-21swapper/322:02:383
9494992118,2cyclictest32-21ksoftirqd/223:35:012
948999214,1cyclictest2081-21diskmemload21:22:251
948999214,16cyclictest7175-21sh22:29:291
948999210,4cyclictest0-21swapper/121:53:231
948999210,3cyclictest0-21swapper/120:20:171
948999210,0cyclictest0-21swapper/123:01:111
949499200,20cyclictest0-21swapper/200:37:412
949499200,16cyclictest0-21swapper/221:45:122
948999206,6cyclictest27654-21ssh23:57:501
948999205,7cyclictest20904-21ssh21:59:311
948999201,4cyclictest27709-21wget20:55:141
948999201,18cyclictest0-21swapper/100:30:131
948999200,2cyclictest0-21swapper/123:19:191
948599200,18cyclictest0-21swapper/022:36:510
948599200,16cyclictest0-21swapper/022:04:050
9496991914,4cyclictest0-21swapper/300:19:303
949699191,10cyclictest7864-21idleruntime-cro22:30:013
949699190,15cyclictest0-21swapper/300:10:133
9494991918,1cyclictest0-21swapper/220:58:452
948999193,3cyclictest28598-21ssh21:25:171
948999191,1cyclictest24770-21apt-get00:35:011
948999190,4cyclictest0-21swapper/123:20:471
948999190,0cyclictest0-21swapper/122:40:171
9485991911,4cyclictest23935-21apache223:25:210
948599190,4cyclictest0-21swapper/022:57:330
948599190,0cyclictest0-21swapper/023:48:590
9496991817,1cyclictest0-21swapper/322:53:223
949699181,3cyclictest0-21swapper/322:05:153
949699180,1cyclictest0-21swapper/321:44:173
949499188,5cyclictest2593-21sh23:05:122
949499180,18cyclictest0-21swapper/200:03:332
949499180,15cyclictest0-21swapper/221:57:432
948999189,8cyclictest30429-21grep19:50:241
948599180,1cyclictest0-21swapper/023:53:340
948599180,11cyclictest0-21swapper/020:50:120
949499170,17cyclictest0-21swapper/222:23:272
949499170,17cyclictest0-21swapper/221:23:382
949499170,17cyclictest0-21swapper/200:05:362
948999179,7cyclictest18101-21head19:25:211
948999171,2cyclictest0-21swapper/119:10:141
949499160,16cyclictest0-21swapper/223:48:232
948999168,7cyclictest21368-21basename19:34:591
948999161,3cyclictest0-21swapper/119:40:121
948599160,14cyclictest0-21swapper/022:13:290
25112160,2sleep232-21ksoftirqd/222:25:542
949699153,4cyclictest30434-21cron00:00:013
949699152,11cyclictest12749-21/usr/sbin/munin21:15:273
949699151,0cyclictest0-21swapper/321:56:393
949699150,1cyclictest0-21swapper/322:41:253
9494991512,2cyclictest32-21ksoftirqd/221:40:002
949499150,3cyclictest24819-21cat22:40:002
948999159,5cyclictest27676-21tr19:45:231
948999158,6cyclictest5520-21chrt20:05:391
948999158,6cyclictest5465-21ssh23:45:041
948999157,7cyclictest9774-21grep20:15:221
948999157,7cyclictest9774-21grep20:15:221
948999157,7cyclictest18014-21basename20:35:111
948999157,7cyclictest1018-21chrt19:59:141
948999154,11cyclictest2081-21diskmemload22:47:261
948999151,4cyclictest22258-21fschecks_count19:35:161
948599158,3cyclictest25874-21unixbench_multi22:20:260
948599150,14cyclictest0-21swapper/000:07:190
9496991414,0cyclictest0-21swapper/323:25:363
9496991414,0cyclictest0-21swapper/321:38:093
9496991411,2cyclictest2508-21tr20:00:203
9496991410,3cyclictest7845-21sed20:10:283
9496991410,3cyclictest28702-21cat19:50:003
9494991410,1cyclictest0-21swapper/223:51:562
949499140,13cyclictest0-21swapper/222:02:262
948999147,6cyclictest31700-21sed21:00:271
948999147,6cyclictest14769-21expr19:20:131
948999146,7cyclictest15915-21cut20:30:151
948999141,2cyclictest225-21systemd-journal21:10:001
948999140,4cyclictest225-21systemd-journal20:55:011
948599149,3cyclictest32393-21basename22:25:150
948599145,5cyclictest15384-21apache_accesses22:15:130
9485991410,3cyclictest22879-21wc19:35:220
9485991410,3cyclictest2036-21grep22:45:190
948599140,1cyclictest0-21swapper/021:41:010
949699139,3cyclictest29922-21grep19:50:183
949699139,2cyclictest0-21swapper/321:45:153
949699135,7cyclictest2081-21diskmemload23:19:003
9496991311,1cyclictest0-21swapper/322:12:193
9496991310,2cyclictest28326-21grep20:55:163
9496991310,2cyclictest25102-21cat19:40:183
9496991310,2cyclictest17837-21cat20:35:013
9496991310,2cyclictest16943-21grep19:25:133
9496991310,2cyclictest13780-21grep20:25:173
9496991310,2cyclictest11748-21grep23:10:173
949499139,2cyclictest19122-21grep20:35:202
949499131,7cyclictest0-21swapper/221:32:542
9494991313,0cyclictest0-21swapper/222:47:422
9494991310,2cyclictest30779-21grep21:00:172
9494991310,2cyclictest24277-21wc20:45:252
9494991310,2cyclictest22803-21cut23:55:142
948999136,6cyclictest6428-21wc20:10:131
948999135,7cyclictest2443-21wc20:00:191
948999135,7cyclictest13329-21head19:15:211
948599139,3cyclictest5762-21grep21:30:290
948599139,3cyclictest10573-21wc23:10:000
948599139,3cyclictest10296-21grep00:25:260
948599139,3cyclictest0-21swapper/023:55:230
948599133,8cyclictest0-21swapper/000:30:130
948599133,6cyclictest225-21systemd-journal20:57:130
948599131,6cyclictest10637-21munin-run20:20:000
948599131,6cyclictest10637-21munin-run20:20:000
9485991310,2cyclictest29960-21tr19:50:190
9485991310,2cyclictest22107-21grep20:40:250
948599130,4cyclictest0-21swapper/000:21:400
948599130,1cyclictest0-21swapper/000:16:100
948599130,11cyclictest0-21swapper/023:20:490
948599130,11cyclictest0-21swapper/019:40:120
949699129,2cyclictest7571-21sed23:45:273
949699129,2cyclictest24622-21sort22:20:173
949699129,2cyclictest22861-21basename20:45:143
949699129,2cyclictest22255-21tr23:35:203
949699129,2cyclictest20914-21df20:40:143
949699129,2cyclictest2070-21cron23:05:003
949699129,2cyclictest18875-21vmstat00:30:273
949699129,2cyclictest18475-21expr22:55:133
949699129,2cyclictest18029-21strings22:35:213
949699129,2cyclictest12873-21grep19:15:173
949699129,2cyclictest12515-21grep20:20:243
949699129,2cyclictest1072-21cat00:40:013
949699128,3cyclictest8646-21grep20:15:153
949699128,3cyclictest8646-21grep20:15:143
949699128,3cyclictest20019-21grep19:30:183
9496991211,1cyclictest0-21swapper/321:52:053
9496991210,1cyclictest2643-21apt-get21:10:163
9496991210,1cyclictest13330-21sh00:27:403
949699120,11cyclictest8828-21grep00:05:263
949499129,2cyclictest8230-21cron20:15:002
949499129,2cyclictest31613-21tail22:05:162
949499129,2cyclictest17878-21wc20:35:012
949499129,2cyclictest11623-21grep23:10:152
949499128,3cyclictest17739-21hddtemp_smartct00:30:172
949499124,4cyclictest225-21systemd-journal21:25:132
9494991212,0cyclictest0-21swapper/223:21:052
9494991211,1cyclictest22608-21ssh00:14:182
9494991210,1cyclictest25988-21sed20:50:172
949499120,2cyclictest0-21swapper/221:51:022
949499120,12cyclictest0-21swapper/222:14:062
949499120,10cyclictest0-21swapper/223:00:542
948999123,3cyclictest20635-21grep20:40:131
948599129,2cyclictest8536-21cut22:30:140
948599129,2cyclictest7270-21tr20:10:180
948599129,2cyclictest4610-21sed21:10:270
948599129,2cyclictest32612-21grep21:05:140
948599129,2cyclictest28787-21sed21:25:170
948599129,2cyclictest26744-21ssh00:35:190
948599129,2cyclictest12808-21grep19:15:170
948599128,3cyclictest7902-21ssh21:51:000
948599128,2cyclictest0-21swapper/021:56:380
948599124,7cyclictest28460-21munin-node00:03:230
948599123,7cyclictest3491-21sh20:05:000
948599120,2cyclictest141rcu_preempt23:36:150
949699119,1cyclictest31792-21head00:00:173
949699119,1cyclictest14037-21cat23:30:253
949699118,2cyclictest3974-21ssh00:21:563
949699118,2cyclictest3730-21basename20:05:093
949699118,2cyclictest31485-21expr21:00:263
949699118,2cyclictest31009-21tr23:40:223
949699118,2cyclictest26728-21cat20:50:263
949699118,2cyclictest21923-21cat19:35:153
949699118,2cyclictest18914-21ssh22:16:363
949699118,2cyclictest15004-21ssh23:50:183
949699118,2cyclictest14506-21apt-get19:20:123
949699118,2cyclictest10503-21dump-pmu-power23:10:003
949699118,2cyclictest10486-21sed19:10:163
949699110,4cyclictest0-21swapper/323:20:143
949499118,2cyclictest9293-21sed20:15:182
949499118,2cyclictest9293-21sed20:15:172
949499118,2cyclictest32635-21grep19:55:212
949499118,2cyclictest32371-21ssh00:20:102
949499118,2cyclictest31050-21cat19:55:002
949499118,2cyclictest2884-21chrt21:10:162
949499118,2cyclictest276042sleep219:45:222
949499118,2cyclictest26174-21cron19:45:002
949499118,2cyclictest25130-21rm23:37:032
949499118,2cyclictest24599-21df00:15:162
949499118,2cyclictest22600-21wc20:45:012
949499118,2cyclictest21442-21grep23:15:272
949499118,2cyclictest17938-21rm22:55:082
949499118,2cyclictest14050-21grep20:25:182
949499118,2cyclictest10395-21grep00:25:272
949499117,3cyclictest2201-21cat21:10:002
949499113,4cyclictest225-21systemd-journal22:52:142
9494991110,1cyclictest0-21swapper/223:40:312
948599119,1cyclictest16615-21sleep19:24:590
948599118,2cyclictest9780-21aten2_r7power_e19:10:130
948599118,2cyclictest32203-21ssh21:46:130
948599118,2cyclictest32142-21grep19:55:170
948599118,2cyclictest31807-21uname22:05:180
948599118,2cyclictest30732-21ssh23:02:050
948599118,2cyclictest30019-21grep23:40:150
948599118,2cyclictest20278-21grep21:20:170
948599118,2cyclictest19346-21hostname19:30:150
948599118,2cyclictest19334-21ssh23:15:090
948599118,2cyclictest18708-21fschecks_time20:35:160
948599118,2cyclictest17597-21grep19:25:160
948599118,2cyclictest15946-21tr00:10:150
948599118,2cyclictest14447-21ssh21:16:320
948599118,2cyclictest13397-21ssh22:51:470
948599118,2cyclictest13393-21ssh21:35:190
948599117,3cyclictest15942-21grep20:30:150
948599110,10cyclictest32116-21dump-pmu-power21:05:000
948599110,10cyclictest30415-21ssh22:42:530
949699108,1cyclictest31811-21expr19:55:153
949699108,1cyclictest18812sleep321:07:483
949699108,1cyclictest18245-21cat20:35:153
949699105,1cyclictest3839-21apt-get21:30:133
949699100,9cyclictest8306-21ssh22:30:113
949499108,1cyclictest233802sleep219:35:402
949499108,1cyclictest1696-21cstates20:00:152
949499108,1cyclictest16942-21wget19:25:132
949499108,1cyclictest144612sleep219:20:062
949499107,2cyclictest3793-21grep20:05:102
949499107,2cyclictest11644-21tail20:20:162
949499106,3cyclictest13645-21wc19:15:252
949499106,2cyclictest19532-21tr19:30:142
948599107,2cyclictest26414-21chrt19:45:070
948599107,2cyclictest11477-21taskset20:20:150
948599106,3cyclictest4357-21grep20:05:140
94949997,1cyclictest9682-21chrt19:10:042
94859997,1cyclictest227612sleep020:45:060
94859997,1cyclictest13770-21df20:25:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional