You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 14:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Dec 10, 2025 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1403929682,10sleep10-21swapper/119:09:121
1389729682,9sleep30-21swapper/319:07:223
1395828565,16sleep20-21swapper/219:08:092
1382828463,6sleep00-21swapper/019:06:290
181142600,1sleep20-21swapper/222:44:052
14193993330,2cyclictest7245-21diskmemload00:00:183
14193993029,1cyclictest0-21swapper/321:29:183
14193992524,1cyclictest0-21swapper/321:52:383
1419399250,24cyclictest0-21swapper/319:25:143
1418799240,1cyclictest0-21swapper/123:33:571
1418799240,0cyclictest0-21swapper/121:44:381
1419399231,4cyclictest141rcu_preempt00:08:453
14191992322,1cyclictest0-21swapper/223:29:122
1419199220,22cyclictest0-21swapper/222:48:042
1419199220,1cyclictest0-21swapper/221:29:442
14193992019,1cyclictest0-21swapper/322:20:133
14193992018,1cyclictest18487-21ssh21:16:183
14193992018,1cyclictest18487-21ssh21:16:173
1419199200,2cyclictest0-21swapper/223:10:292
1419199200,20cyclictest0-21swapper/221:41:232
1418799200,2cyclictest0-21swapper/121:56:451
14193991919,0cyclictest0-21swapper/321:48:343
1419399190,3cyclictest0-21swapper/323:15:573
14191991910,7cyclictest27937-21/usr/sbin/munin20:45:192
1419199190,1cyclictest2954-21switchtime19:50:252
14187991918,1cyclictest0-21swapper/121:12:201
14193991817,1cyclictest0-21swapper/300:34:583
1419399180,18cyclictest0-21swapper/322:42:583
1419199181,4cyclictest0-21swapper/200:31:562
221372170,4sleep125-21ksoftirqd/122:03:181
14193991716,1cyclictest0-21swapper/321:36:173
1419399170,10cyclictest0-21swapper/300:25:153
1419199170,17cyclictest0-21swapper/222:22:022
1419199170,17cyclictest0-21swapper/221:38:072
1418799174,1cyclictest141rcu_preempt23:03:121
14187991716,1cyclictest0-21swapper/122:43:291
1418799170,16cyclictest0-21swapper/123:26:161
1418599175,6cyclictest224-21systemd-journal20:27:410
1418599174,6cyclictest18197-21idleruntime-cro20:24:590
1418599172,9cyclictest31134-21perf19:45:000
1419399161,4cyclictest0-21swapper/322:46:023
1419199169,6cyclictest8319-21wc00:00:272
1419199160,15cyclictest0-21swapper/222:08:242
14187991613,2cyclictest25-21ksoftirqd/120:30:121
1418599160,1cyclictest0-21swapper/021:12:180
14193991514,1cyclictest0-21swapper/323:33:583
14193991514,1cyclictest0-21swapper/300:19:473
14191991513,1cyclictest19754-21sh00:09:462
1419199151,14cyclictest30487-21ssh22:51:172
14187991512,2cyclictest25-21ksoftirqd/122:46:081
1418599156,8cyclictest4876-21munin-run21:05:010
1419399144,7cyclictest20549-21idleruntime-cro20:29:593
14193991413,1cyclictest0-21swapper/323:39:433
14193991413,1cyclictest0-21swapper/321:44:383
14193991412,1cyclictest0-21swapper/323:08:083
1419199149,3cyclictest1422-21wc19:50:012
14191991410,3cyclictest32306-21tr22:10:142
14191991410,3cyclictest30645-21tr20:50:182
1419199140,14cyclictest0-21swapper/222:30:382
1419199140,13cyclictest0-21swapper/221:50:132
14187991413,1cyclictest0-21swapper/100:02:131
1418799141,1cyclictest0-21swapper/123:55:161
14187991410,3cyclictest2255-21grep19:50:171
1418599147,6cyclictest30113-21apache222:03:400
14185991413,1cyclictest0-21swapper/023:37:140
14185991410,3cyclictest22010-21cut00:35:140
1418599140,11cyclictest0-21swapper/023:05:260
1419399139,3cyclictest523-21grep22:10:163
14193991310,2cyclictest4677-21sed19:55:173
14193991310,2cyclictest31810-21tr19:45:163
14193991310,2cyclictest27421-21cat23:10:293
14193991310,2cyclictest23749-21grep00:35:283
14193991310,2cyclictest17594-21idleruntime-cro23:05:013
1419199139,4cyclictest37850irq/131-snd_hda23:45:132
1419199139,2cyclictest1679-21cut23:15:172
1419199135,7cyclictest25131-21rm22:26:222
14191991310,2cyclictest4372-21tr22:55:242
14191991310,2cyclictest29486-21grep19:40:142
14191991310,2cyclictest29428-21ssh23:53:322
14191991310,2cyclictest28212-21wc00:15:182
14191991310,2cyclictest25166-21sed23:30:172
14191991310,2cyclictest18982-21grep22:00:272
1418799134,5cyclictest224-21systemd-journal23:17:001
14187991311,1cyclictest85082sleep123:20:151
14187991310,2cyclictest4181-21wc21:50:261
14187991310,2cyclictest31915-21tr21:25:251
14187991310,2cyclictest26698-21ssh00:15:001
14187991310,2cyclictest21305-21cron19:25:011
14187991310,2cyclictest21305-21cron19:25:001
14187991310,2cyclictest19454-21head23:05:231
14187991310,2cyclictest16533-21grep23:45:181
14187991310,2cyclictest14315-21cut20:15:221
1418799130,12cyclictest29458-21sh23:12:071
1418599139,3cyclictest5875-21grep21:05:170
1418599134,6cyclictest16245-21df23:45:160
14185991312,1cyclictest13-21ksoftirqd/023:19:330
14185991310,2cyclictest8453-21sed00:00:270
14185991310,2cyclictest7900-21tail22:15:160
14185991310,2cyclictest32761-21cat23:55:220
14185991310,2cyclictest28646-21cron19:40:000
14185991310,2cyclictest10718-21cut21:55:180
1418599130,9cyclictest0-21swapper/000:30:140
1418599130,7cyclictest8473-21perf00:25:010
1419399129,2cyclictest9553-21grep20:05:193
1419399129,2cyclictest7271-21cat21:10:013
1419399129,2cyclictest32509-21sh20:55:013
1419399129,2cyclictest27158-21cat20:40:273
1419399129,2cyclictest15769-21cron20:20:003
1419399126,2cyclictest0-21swapper/322:53:103
1419399125,2cyclictest24745-21apt-get22:05:153
1419399123,7cyclictest4541-21/usr/sbin/munin22:35:243
14193991211,1cyclictest8570-21ssh21:31:383
14193991211,1cyclictest0-21swapper/322:16:203
14193991210,1cyclictest8572-21grep23:40:193
14193991210,1cyclictest7074-21df20:00:173
14193991210,1cyclictest19237-21cut19:20:143
14193991210,1cyclictest19237-21cut19:20:133
1419399120,3cyclictest0-21swapper/322:00:313
1419199129,2cyclictest4398-21grep21:00:282
1419199129,2cyclictest24200-21grep19:30:132
1419199129,2cyclictest23467-21grep21:20:152
1419199129,2cyclictest23030-21cat20:35:012
1419199129,2cyclictest18910-21cut23:05:172
1419199129,2cyclictest15745-21ssh21:15:162
1419199129,2cyclictest15745-21ssh21:15:152
1419199129,2cyclictest10000-21cat20:05:222
1419199128,3cyclictest32431-21tail23:55:202
1419199127,3cyclictest18362-21wc20:25:002
1419199121,9cyclictest7245-21diskmemload00:36:042
14191991210,2cyclictest32-21ksoftirqd/200:25:352
1419199120,5cyclictest0-21swapper/221:10:212
1418799129,2cyclictest5558-21grep19:55:261
1418799129,2cyclictest5369-21grep22:35:211
1418799129,2cyclictest31840-21tr19:45:161
1418799129,2cyclictest29010-21ssh22:07:341
1418799129,2cyclictest2847-21grep00:20:201
1418799129,2cyclictest28264-21sed19:35:281
1418799129,2cyclictest28157-21cut22:50:161
1418799129,2cyclictest25819-21tail19:30:251
1418799129,2cyclictest24410-21tr20:35:211
1418799129,2cyclictest24377-21uniq23:50:171
1418799129,2cyclictest22694-21tail00:35:181
1418799129,2cyclictest21612-21latency_hist22:24:591
1418799129,2cyclictest17865-21systemd-cgroups00:07:581
1418799129,2cyclictest1686-21cat23:35:271
1418799129,2cyclictest15317-21wc23:44:591
1418799129,2cyclictest12948-21tail20:10:291
1418799128,3cyclictest4320-21wc21:00:271
1418799128,3cyclictest29144-21cron22:30:001
1418799128,3cyclictest27941-21grep00:15:161
1418799128,3cyclictest13036-21tr21:35:161
1418799123,4cyclictest0-21swapper/121:05:141
1418599129,2cyclictest4912-21tr22:35:180
1418599129,2cyclictest27763-21tr22:50:140
1418599129,2cyclictest27540-21grep23:10:290
1418599129,2cyclictest27459-21grep00:15:140
1418599129,2cyclictest20340-21tr19:20:210
1418599129,2cyclictest20340-21tr19:20:200
1418599128,3cyclictest7744-21snmpd23:23:200
1418599128,3cyclictest32073-21grep20:50:290
1418599121,10cyclictest10645-21/usr/sbin/munin23:00:130
1418599120,10cyclictest0-21swapper/000:13:160
1419399119,1cyclictest26269-21cron19:35:003
1419399118,2cyclictest8618-21ssh21:10:183
1419399118,2cyclictest625-21rm22:31:443
1419399118,2cyclictest5989-21ssh00:22:303
1419399118,2cyclictest3164-21grep19:50:263
1419399118,2cyclictest24433-21ssh21:20:223
1419399118,2cyclictest23514-21tr22:25:233
1419399118,2cyclictest2068-21grep20:55:303
1419399118,2cyclictest16124-21ssh23:25:023
1419399118,2cyclictest11325-21gzip21:55:243
1419399114,5cyclictest0-21swapper/323:47:003
1419399114,4cyclictest224-21systemd-journal23:25:013
1419399110,10cyclictest2716-21taskset21:00:103
1419199119,2cyclictest0-21swapper/223:21:342
1419199119,1cyclictest26690-21grep20:40:212
1419199118,2cyclictest8289-21sed23:40:162
1419199118,2cyclictest7963-21grep22:15:172
1419199118,2cyclictest6894-21expr20:00:172
1419199118,2cyclictest6539-21vmstat22:35:312
1419199118,2cyclictest4184-21expr19:55:152
1419199118,2cyclictest22899-21uname19:25:212
1419199118,2cyclictest22337-21grep00:10:272
1419199118,2cyclictest1970-21ssh23:35:282
1419199118,2cyclictest14400-21ssh21:57:532
1419199118,2cyclictest13712-21hostname20:15:192
1419199117,3cyclictest26780-21grep19:35:162
1419199116,4cyclictest5357-21ssh21:30:142
1419199110,7cyclictest37850irq/131-snd_hda20:10:162
1419199110,0cyclictest0-21swapper/200:23:322
1418799118,2cyclictest7912-21tail22:15:171
1418799118,2cyclictest6632-21cut20:00:161
1418799118,2cyclictest5683-21wc21:30:161
1418799118,2cyclictest4760-21wc22:55:291
1418799118,2cyclictest30691-21grep20:50:181
1418799118,2cyclictest30691-21grep19:40:251
1418799118,2cyclictest30374-21ssh21:46:151
1418799118,2cyclictest26613-21ssh21:21:491
1418799118,2cyclictest23915-21basename19:30:011
1418799118,2cyclictest15860-21cut21:15:151
1418799118,2cyclictest15860-21cut21:15:151
1418799118,2cyclictest1084-21grep20:55:191
1418799117,3cyclictest519-21gmain00:30:141
1418799117,3cyclictest4076-21cron22:34:591
1418799112,5cyclictest19017-21df_inode20:25:191
1418799110,10cyclictest4008-21sh22:12:351
1418599119,1cyclictest32458-21tr22:10:150
1418599118,2cyclictest9144-21ls23:40:230
1418599118,2cyclictest7620-21grep20:00:230
1418599118,2cyclictest5212-21ssh21:30:070
1418599118,2cyclictest5110-21ssh21:51:060
1418599118,2cyclictest32236-21grep19:45:170
1418599118,2cyclictest31142-21grep22:30:240
1418599118,2cyclictest25778-21uname22:05:190
1418599118,2cyclictest25675-21sed23:50:260
1418599118,2cyclictest2475-21uname19:50:200
1418599118,2cyclictest21953-21ssh22:25:070
1418599118,2cyclictest21030-21wc20:30:140
1418599118,2cyclictest20483-21ssh22:45:170
1418599118,2cyclictest18167-21sh21:16:020
1418599118,2cyclictest18167-21sh21:16:010
1418599118,2cyclictest16860-21aten2_r7power_e19:15:160
1418599118,2cyclictest15045-21sort19:10:170
1418599118,2cyclictest13451-21ssh00:05:070
1418599118,2cyclictest12586-21sed22:40:160
1418599118,2cyclictest11337-21grep20:10:170
1418599118,2cyclictest10444-21sendmail_mailtr00:25:240
1418599117,3cyclictest25388-21ssh23:30:190
1418599117,2cyclictest31347-21tr21:25:200
1418599112,7cyclictest16396-21ssh23:25:150
1419399108,2cyclictest26059-21ssh23:50:443
1419399108,2cyclictest12006-21head20:10:203
1419399108,1cyclictest6248-21rm22:56:273
1419399108,1cyclictest31285-21ssh23:55:013
1419399108,1cyclictest265112sleep319:35:103
1419399108,1cyclictest20389-21sh00:10:113
1419399107,2cyclictest16449-21expr20:20:153
1419399100,9cyclictest28292-21expr20:45:163
1419399100,6cyclictest7744-21snmpd19:10:393
1419199108,1cyclictest18405-21taskset20:25:092
1419199108,1cyclictest17100-21ls19:15:172
1419199108,1cyclictest150002sleep219:10:162
1419199107,2cyclictest27427-21ssh21:45:092
1419199107,2cyclictest2457-21cron21:00:002
1419199107,2cyclictest19175-21basename19:20:112
1419199107,2cyclictest19175-21basename19:20:112
1419199107,2cyclictest12384-21wc23:00:262
1419199101,5cyclictest224-21systemd-journal21:10:002
1419199100,9cyclictest23255-21cut20:35:132
1418799108,1cyclictest8965-21ssh00:25:151
1418799108,1cyclictest25844-21tr20:40:151
1418799102,4cyclictest0-21swapper/119:10:121
1418799100,9cyclictest16683-21grep20:20:161
1418599108,1cyclictest27971-21apt20:45:150
1418599108,1cyclictest276092sleep020:43:580
1418599108,1cyclictest22973-21ssh21:20:050
1418599108,1cyclictest19960-21ssh21:40:020
1418599108,1cyclictest1797-21wc20:55:280
1418599108,1cyclictest14755-21cat22:20:140
1418599108,1cyclictest10666-21taskset20:05:430
1418599107,2cyclictest3418-21ssh22:55:180
1418599107,2cyclictest27276-21ssh21:45:010
1418599107,2cyclictest24639-21cat19:30:150
1418599107,2cyclictest23968-21tail20:35:170
1418599107,2cyclictest13752-21grep20:15:190
1418599107,2cyclictest12974-21cat21:35:160
1418599104,5cyclictest4493-21diskstats19:55:150
141939998,1cyclictest21389-21sh20:30:163
141939997,2cyclictest29795-21cat19:40:163
141939997,2cyclictest23558-21expr20:35:143
141939996,2cyclictest16853-21cut19:15:153
141879997,1cyclictest9926-21taskset20:05:211
141879997,1cyclictest17465-21cut19:15:191
141879996,2cyclictest28699-21tail20:45:191
141859996,2cyclictest21824-21wget19:25:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional