You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-03 - 15:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Sep 03, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759829569,10sleep00-21swapper/007:06:120
2756528976,9sleep20-21swapper/207:05:462
2785528664,18sleep30-21swapper/307:09:323
2762628565,16sleep10-21swapper/107:06:331
263302570,0sleep30-21swapper/310:10:273
2797399311,4cyclictest0-21swapper/011:55:010
2797399301,6cyclictest0-21swapper/010:29:340
2797399283,6cyclictest24581-21tune2fs12:05:160
27973992725,2cyclictest0-21swapper/008:41:090
27973992725,2cyclictest0-21swapper/008:41:090
27977992524,1cyclictest0-21swapper/111:51:301
2797799250,1cyclictest0-21swapper/112:00:131
2797399252,14cyclictest681-21ssh10:15:130
27983992421,2cyclictest16915-21diskstats10:05:153
2798399241,1cyclictest0-21swapper/310:00:563
27981992421,2cyclictest13569-21users11:00:282
27981992320,2cyclictest22571-21apache_processe09:30:162
27981992320,2cyclictest18069-21diskstats10:25:152
27977992320,2cyclictest16902-21df_inode10:05:151
27973992320,2cyclictest4386-21ntp_states10:55:200
27973992320,2cyclictest27689-21diskstats10:50:150
27983992219,2cyclictest16426-21ntp_states09:45:203
2798199221,20cyclictest16297-21ssh10:05:132
27977992219,2cyclictest26584-21df_inode10:30:181
2797399226,10cyclictest26527-21apt-get08:15:000
27973992219,2cyclictest16529-21/usr/sbin/munin12:20:190
2797399220,2cyclictest0-21swapper/009:41:370
27983992118,2cyclictest25945-21apache_processe10:30:163
27983992118,2cyclictest24988-21apache_accesses12:25:103
27977992119,1cyclictest4879-21df11:15:141
27977992119,1cyclictest21141-21/usr/sbin/munin09:10:131
27977992118,2cyclictest11012-21perl10:40:161
27973992118,2cyclictest24823-21apache_volume10:10:170
27983992017,2cyclictest28332-21/usr/sbin/munin11:10:143
27983992017,2cyclictest11392-21users12:35:303
27981992018,1cyclictest2367-21df10:35:152
2798199200,12cyclictest0-21swapper/210:40:132
27977992018,1cyclictest7139-21ntp_states11:35:201
27977992018,1cyclictest7139-21ntp_states11:35:191
2797799201,18cyclictest14045-21timerwakeupswit07:45:251
27973992017,2cyclictest22725-21ntp_states11:25:200
27973992013,6cyclictest12880-21ntp_states11:00:230
27983991919,0cyclictest0-21swapper/311:04:443
27983991916,2cyclictest2516-21/usr/sbin/munin10:35:163
27977991916,2cyclictest14191-21/usr/sbin/munin11:40:141
27977991916,2cyclictest10252-21hddtemp_smartct12:35:181
2797799190,1cyclictest20500-21diskmemload10:17:021
2797799190,1cyclictest0-21swapper/111:24:091
2797399191,6cyclictest20500-21diskmemload10:35:000
27973991910,8cyclictest606-21wc08:25:210
27983991817,1cyclictest0-21swapper/310:21:153
27983991815,2cyclictest9782-21sensors_temp12:15:243
2798399180,18cyclictest0-21swapper/309:36:103
2798199189,4cyclictest225-21systemd-journal10:19:312
27981991816,1cyclictest31743-21df_inode11:50:192
27981991815,2cyclictest29014-21df_inode11:10:152
27977991815,2cyclictest1847-21apache_processe10:35:121
2797799180,17cyclictest0-21swapper/111:31:281
2797399189,8cyclictest17147-21tr09:00:190
2797399189,2cyclictest23721-21cut09:50:180
2797399183,3cyclictest20660-21latency_hist09:10:010
27973991812,5cyclictest10760-21df10:40:140
27973991811,6cyclictest17052-21chrt12:39:550
2797399180,9cyclictest26582-21ssh09:32:010
2797399180,8cyclictest0-21swapper/011:42:260
2798399170,17cyclictest0-21swapper/310:27:033
2797399179,7cyclictest29471-21wc07:10:220
2797399178,2cyclictest9582-21grep08:45:150
2797399178,2cyclictest16571-21tr10:05:130
2797399177,2cyclictest22564-21sh11:45:090
2797399172,2cyclictest9225-21aten2_r7power_p10:20:140
27973991714,2cyclictest29690-21ntp_states11:10:190
2797399170,16cyclictest31631-21wc11:30:290
2797399170,10cyclictest1992-21sort07:20:240
2798399169,5cyclictest24222-21apache210:45:233
27983991614,1cyclictest14038-21irqstats11:20:173
2798399160,0cyclictest0-21swapper/311:44:463
2798199162,13cyclictest0-21swapper/210:00:392
2797799160,16cyclictest0-21swapper/110:10:411
2797399168,2cyclictest28661-21ssh09:14:370
2797399168,2cyclictest22951-21ssh12:04:450
2797399168,2cyclictest2169-21ssh09:36:450
2797399168,2cyclictest17408-21cat07:55:120
2797399168,2cyclictest12027-21ssh11:57:480
27973991610,5cyclictest934-21ssh09:55:220
27983991514,1cyclictest0-21swapper/310:19:423
27983991512,2cyclictest25608-21users09:50:303
27983991510,3cyclictest7666-21wc11:35:253
27983991510,3cyclictest7666-21wc11:35:253
27981991512,2cyclictest22992-21cat09:10:252
27981991512,2cyclictest19229-21df_inode10:45:162
27981991512,2cyclictest1504-21munin-node11:45:102
2798199150,3cyclictest0-21swapper/209:25:092
27977991511,3cyclictest4927-21ls08:35:171
2797799151,13cyclictest29126-21/usr/sbin/munin07:10:191
2797799150,15cyclictest0-21swapper/108:03:581
2797399158,2cyclictest8730-21ssh10:39:150
2797399158,2cyclictest1994-21ssh09:17:250
2797399158,1cyclictest9001-21gzip07:35:210
2797399158,1cyclictest3485-21sh12:11:440
2797399158,1cyclictest27281-21ssh12:25:370
2797399158,1cyclictest19901-21apache_processe11:05:110
2797399157,2cyclictest913-21ssh12:30:120
2797399157,2cyclictest5998-21egrep09:20:140
2797399157,2cyclictest5670-21chrt11:15:200
2797399157,2cyclictest20233-21ssh10:45:250
2798399148,1cyclictest20995-21kworker/3:010:44:463
27983991413,1cyclictest0-21swapper/312:11:343
27983991411,2cyclictest3941-21hddtemp_smartct10:55:163
27983991410,3cyclictest2893-21wc12:30:303
2798399140,0cyclictest0-21swapper/309:33:483
27981991411,2cyclictest6928-21apache_processe11:55:082
27981991410,3cyclictest32613-21grep09:55:162
27981991410,3cyclictest27338-21chrt09:51:512
2798199140,3cyclictest0-21swapper/211:15:122
2798199140,1cyclictest0-21swapper/212:05:132
2797799143,3cyclictest225-21systemd-journal10:59:591
27977991413,1cyclictest0-21swapper/112:26:401
2797399147,2cyclictest16378-21ssh11:21:290
2797399147,2cyclictest11072-21ssh10:01:200
2797399147,1cyclictest9762-21ssh12:15:220
2797399147,1cyclictest5737-21ssh11:35:090
2797399147,1cyclictest5737-21ssh11:35:090
2797399147,1cyclictest313602sleep008:23:130
2797399146,2cyclictest14445-21tr09:25:140
2797399146,2cyclictest12563-21wc08:50:240
2798399139,3cyclictest22525-21grep08:05:153
2798399131,11cyclictest29373-21awk11:30:003
27983991310,2cyclictest30004-21grep09:15:133
27983991310,2cyclictest25126-21grep08:10:153
27983991310,2cyclictest20010-21sed09:05:293
27983991310,2cyclictest15615-21wc09:25:223
27983991310,2cyclictest13897-21tr08:55:133
2798199139,3cyclictest9468-21cut08:45:162
2798199138,4cyclictest6418-21cat08:40:002
27981991311,1cyclictest22276-21hddtemp_smartct11:25:162
27981991310,2cyclictest6096-21sed07:30:172
27981991310,2cyclictest6093-21grep11:35:142
27981991310,2cyclictest6093-21grep11:35:142
27981991310,2cyclictest5270-21cron09:20:002
27981991310,2cyclictest24116-21sort08:10:002
27981991310,2cyclictest16123-21cat12:20:002
27981991310,2cyclictest10205-21grep12:35:182
2797799139,3cyclictest28147-21wc10:50:181
2797799138,4cyclictest31919-21egrep08:25:141
2797799134,5cyclictest16004-21/usr/sbin/munin09:00:111
27977991312,1cyclictest0-21swapper/109:41:301
2797799131,1cyclictest24011-21ssh11:07:021
27977991311,1cyclictest0-21swapper/111:49:121
27977991310,2cyclictest2972-21grep12:30:311
27977991310,2cyclictest23344-21grep09:30:201
27977991310,2cyclictest16962-21turbostat.cron10:25:011
2797799130,13cyclictest0-21swapper/109:27:271
2797399137,1cyclictest4621-21unixbench_singl07:25:270
2797399137,1cyclictest13082-21fschecks_count07:45:160
2797399137,1cyclictest11401-21sendmail_mailqu07:40:220
27973991311,1cyclictest0-21swapper/009:47:530
27973991310,2cyclictest28093-21grep08:15:190
2798399129,2cyclictest8018-21wc11:55:153
2798399129,2cyclictest7053-21cut09:20:203
2798399129,2cyclictest31134-21ssh11:50:173
2798399129,2cyclictest11734-21tail07:40:263
2798399128,3cyclictest1008-21wc09:55:223
2798399128,2cyclictest13236-21grep07:45:173
2798399120,11cyclictest16732-21grep09:00:163
2798199129,2cyclictest8341-21cut07:35:172
2798199129,2cyclictest8105-21uname09:40:212
2798199129,2cyclictest5464-21ssh09:39:212
2798199129,2cyclictest512-21wc12:30:002
2798199129,2cyclictest4663-21grep07:25:272
2798199129,2cyclictest25350-21df10:10:202
2798199129,2cyclictest17376-21tail12:20:162
2798199129,2cyclictest13884-21grep08:55:142
2798199129,2cyclictest1105-21grep12:30:182
2798199128,2cyclictest28472-21cut07:10:152
2798199128,2cyclictest14290-21grep11:40:132
2798199125,5cyclictest5002-21kworker/u8:210:59:582
27981991211,1cyclictest0-21swapper/210:30:492
2798199120,11cyclictest2000-21ssh10:54:342
2798199120,11cyclictest0-21swapper/210:21:562
2797799129,2cyclictest8717-21ssh10:00:161
2797799129,2cyclictest7191-21ls09:20:211
2797799129,2cyclictest6539-21grep07:30:211
2797799129,2cyclictest6011-21ssh09:58:571
2797799129,2cyclictest29485-21apt-config09:15:011
2797799129,2cyclictest23582-21grep08:05:251
2797799129,2cyclictest22807-21grep11:25:211
2797799128,4cyclictest26642-21sed08:15:001
2797799127,4cyclictest12381-21ssh12:17:091
27977991210,1cyclictest11398-21ssh11:00:071
2797799120,11cyclictest32280-21ssh12:10:151
2797799120,11cyclictest19425-21users10:25:251
2797799120,11cyclictest0-21swapper/108:20:141
2797399129,2cyclictest2860-21grep08:30:190
2797399128,3cyclictest24079-21sh08:10:000
2797399128,3cyclictest14389-21grep08:55:160
2798399119,1cyclictest5157-21grep11:15:163
2798399118,2cyclictest4373-21grep08:35:143
2798399118,2cyclictest362-21cat07:20:013
2798399118,2cyclictest30229-21cut11:30:183
2798399118,2cyclictest29470-21grep08:20:143
2798399118,2cyclictest29315-21head07:10:203
2798399118,2cyclictest28214-21ssh12:07:173
2798399118,2cyclictest27376-21sh10:50:133
2798399118,2cyclictest2669-21taskset07:24:583
2798399118,2cyclictest23659-21latency11:45:183
2798399118,2cyclictest18732-21ssh12:20:313
2798399114,1cyclictest20500-21diskmemload09:40:203
2798399110,10cyclictest16459-21ssh12:00:193
2798199119,1cyclictest11479-21tr08:50:152
2798199118,2cyclictest7392-21grep09:20:222
2798199118,2cyclictest31389-21taskset08:23:352
2798199118,2cyclictest30253-21cut11:30:182
2798199118,2cyclictest28989-21taskset08:18:312
2798199118,2cyclictest18487-21ssh11:23:072
2798199118,2cyclictest17195-21ssh09:45:292
2798199118,2cyclictest15416-21ssh12:00:122
2798199118,2cyclictest15079-21grep07:50:152
2798199118,2cyclictest1047-21wc12:10:232
2797799119,1cyclictest17263-21tune2fs12:20:151
2797799119,1cyclictest1232-21grep07:20:151
2797799118,2cyclictest7762-21expr11:55:141
2797799118,2cyclictest7282-21cut08:40:171
2797799118,2cyclictest7282-21cut08:40:171
2797799118,2cyclictest3177-21tr07:25:151
2797799118,2cyclictest31705-21head09:35:161
2797799118,2cyclictest31022-21ssh11:10:551
2797799118,2cyclictest30283-21sh12:08:531
2797799118,2cyclictest27750-21grep08:15:161
2797799118,2cyclictest24697-21cat09:50:211
2797799118,2cyclictest20503-21grep10:45:271
2797799118,2cyclictest20484-21cron09:10:011
2797799118,2cyclictest17663-21expr07:55:121
27977991110,1cyclictest38150irq/131-snd_hda09:45:251
2798399108,1cyclictest6001-21grep07:30:163
2798399108,1cyclictest5021-21taskset07:29:463
2798399108,1cyclictest31905-21cut08:25:143
2798399108,1cyclictest27602-21cut08:15:163
2798399108,1cyclictest213712sleep308:00:483
2798399108,1cyclictest1791-21basename08:30:133
2798399108,1cyclictest17523-21expr07:55:123
2798399108,1cyclictest166792sleep307:51:473
2798399107,2cyclictest8858-21cat08:45:113
2798399107,2cyclictest7656-21taskset07:35:073
2798399107,2cyclictest21056-21sh09:10:053
2798399107,2cyclictest19648-21date11:05:003
2798399102,4cyclictest225-21systemd-journal08:54:123
2798199109,1cyclictest32-21ksoftirqd/208:31:422
2798199108,1cyclictest6969-21expr08:40:152
2798199108,1cyclictest6969-21expr08:40:152
2798199108,1cyclictest32565-21which08:25:172
2798199108,1cyclictest25852sleep207:23:562
2798199108,1cyclictest25765-21sendmail_mailtr08:10:202
2798199108,1cyclictest20725-21grep11:05:162
2798199108,1cyclictest19527-21ntpq09:05:232
2798199108,1cyclictest17615-21expr07:55:122
2798199107,2cyclictest31304-21head07:15:162
2798199107,2cyclictest19801-21sort08:00:152
2798199107,2cyclictest15998-21basename09:00:112
2798199107,2cyclictest13281-21idleruntime07:45:162
2798199107,2cyclictest11402-21sendmail_mailqu07:40:222
2797799108,1cyclictest9142-21cut08:45:131
2797799108,1cyclictest8077-21expr07:35:141
2797799108,1cyclictest2488-21head08:30:161
2797799108,1cyclictest15333-21grep08:55:261
2797799107,2cyclictest30837-21tr07:15:141
2797799100,9cyclictest14867-21wget07:50:141
2797399107,2cyclictest6254-21uname07:30:190
2797399107,2cyclictest5863-21taskset08:35:380
2797399107,2cyclictest31212-21grep07:15:160
279839996,2cyclictest8155-21grep08:40:273
279839996,2cyclictest8155-21grep08:40:273
279779997,1cyclictest11518-21expr08:50:151
279779993,5cyclictest10309-21grep07:40:141
279739997,1cyclictest20778-21open_files08:00:210
279739997,1cyclictest15257-21ls07:50:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional