You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-08 - 00:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue Apr 07, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3243529678,13sleep20-21swapper/207:06:222
3240128860,9sleep30-21swapper/307:05:553
3243328370,9sleep00-21swapper/007:06:210
3236727758,6sleep10-21swapper/107:05:301
342993531,3cyclictest19661-21kworker/3:112:31:233
34299350,34cyclictest1070-21kworker/3:010:51:133
34299290,28cyclictest19661-21kworker/3:112:17:143
34099280,27cyclictest0-21swapper/211:52:212
34099270,27cyclictest0-21swapper/212:18:012
342992623,2cyclictest19661-21kworker/3:111:34:343
34299261,2cyclictest18654-21df11:00:153
34099260,25cyclictest0-21swapper/211:21:492
336992524,1cyclictest0-21swapper/111:31:451
342992421,3cyclictest1070-21kworker/3:010:38:433
34299241,22cyclictest1070-21kworker/3:010:11:243
34299241,22cyclictest1070-21kworker/3:010:11:233
34299240,23cyclictest19661-21kworker/3:112:01:433
33699240,24cyclictest0-21swapper/112:18:381
33699240,23cyclictest0-21swapper/109:55:461
331992421,2cyclictest1143-21irqstats12:25:190
327652240,0sleep00-21swapper/010:10:410
327652240,0sleep00-21swapper/010:10:400
342992320,2cyclictest27197-21df_inode11:05:153
342992320,2cyclictest27122-21munin-run09:29:593
342992320,2cyclictest24712-21df_inode12:20:153
342992320,2cyclictest21175-21apache_processe11:40:133
342992320,2cyclictest20584-21apache_processe09:45:123
342992320,2cyclictest20456-21hddtemp_smartct11:20:163
342992320,2cyclictest17565-21df12:35:153
342992320,2cyclictest12781-21df_inode09:40:163
340992320,2cyclictest13504-21perl11:35:152
336992320,2cyclictest7786-21diskstats12:10:151
336992320,2cyclictest4520-21hddtemp_smartct09:35:181
336992320,2cyclictest32512-21df_inode10:30:141
336992320,2cyclictest31023-21apache_volume12:05:121
336992320,2cyclictest25392-21ntp_states12:20:211
336992320,2cyclictest22488-21ntp_states11:40:201
336992320,2cyclictest21754-21memory09:45:191
331992321,1cyclictest3645-21apache_processe09:35:130
331992320,2cyclictest7868-21ntp_states10:15:220
331992320,2cyclictest3340-21memory09:15:190
331992320,2cyclictest32335-21ntp_states12:05:210
331992320,2cyclictest29145-21munin-run11:45:000
331992320,2cyclictest27591-21apache_accesses09:30:100
331992320,2cyclictest27216-21diskstats11:05:140
207222230,0sleep10-21swapper/112:36:441
342992219,2cyclictest3650-21apache_processe09:35:123
342992219,2cyclictest31881-21apache_volume10:30:123
342992219,2cyclictest30051-21users11:25:273
342992219,2cyclictest29616-21users09:30:263
342992219,2cyclictest23148-21memory10:05:183
342992219,2cyclictest16173-21users11:55:263
342992219,2cyclictest14983-21apache_volume10:20:133
34299221,0cyclictest0-21swapper/310:02:313
34299220,22cyclictest19661-21kworker/3:111:52:043
340992220,1cyclictest30362-21ntp_states09:50:212
340992220,1cyclictest28146-21df09:30:142
340992220,1cyclictest26762-21df_inode09:10:162
340992219,2cyclictest20217-21irqstats09:25:182
336992220,1cyclictest24878-21apache_accesses10:45:131
336992219,2cyclictest29017-21apache_accesses09:50:121
331992220,1cyclictest29643-21apache_processe11:45:130
331992220,1cyclictest28029-21apache_accesses11:25:120
331992219,2cyclictest9213-21diskstats12:30:160
331992219,2cyclictest6347-21ntp_states09:55:210
331992219,2cyclictest23038-21irqstats10:05:190
331992219,2cyclictest20889-21ntp_states11:20:190
331992219,2cyclictest13546-21apache_processe10:00:110
331992219,2cyclictest10988-21ntp_states10:55:210
342992117,3cyclictest39-21ksoftirqd/312:05:233
342992111,8cyclictest3333-21runrttasks09:15:183
340992118,2cyclictest30403-21apache_processe10:10:122
340992118,2cyclictest30403-21apache_processe10:10:112
33699213,10cyclictest0-21swapper/111:55:141
336992119,1cyclictest30966-21ntp_states11:45:221
336992118,2cyclictest9195-21df_inode12:30:151
336992118,2cyclictest6528-21hddtemp_smartct11:50:161
336992118,2cyclictest19152-21apache_accesses09:25:111
33699210,20cyclictest0-21swapper/108:35:161
331992119,1cyclictest26762-21df_inode09:10:160
331992118,2cyclictest16660-21irqstats12:15:190
342992017,2cyclictest30068-21irqstats09:50:193
34299200,1cyclictest0-21swapper/312:14:533
340992017,2cyclictest23941-21ntp_states12:00:212
340992017,2cyclictest21487-21hddtemp_smartct09:45:172
336992017,2cyclictest32510-21apache_accesses12:25:121
331992018,2cyclictest0-21swapper/010:39:340
331992018,1cyclictest23425-21apache_volume10:25:110
331992018,1cyclictest1114-21apache_processe10:50:110
149912200,0sleep30-21swapper/311:16:513
342991916,2cyclictest25552-21diskstats10:45:163
33699190,19cyclictest0-21swapper/110:42:361
342991815,2cyclictest2575-21apache_accesses11:10:103
342991815,2cyclictest23413-21apache_processe10:25:093
340991815,2cyclictest2583-21apache_processe11:10:112
34099180,18cyclictest0-21swapper/210:05:252
33699182,15cyclictest25645-21diskmemload09:12:161
331991816,1cyclictest4686-21df_inode11:30:150
331991816,1cyclictest29671-21df_inode09:50:150
331991815,2cyclictest22600-21apache_processe12:00:120
342991714,2cyclictest17079-21df_inode10:40:163
34099170,16cyclictest0-21swapper/209:17:372
336991716,1cyclictest0-21swapper/109:33:251
336991714,2cyclictest23419-21apache_volume10:25:101
33199173,7cyclictest8774-21munin-run08:35:010
34299160,16cyclictest0-21swapper/310:16:103
340991614,1cyclictest32-21ksoftirqd/210:46:192
34099161,12cyclictest0-21swapper/212:20:162
33699162,1cyclictest141rcu_preempt11:08:321
33699160,2cyclictest0-21swapper/110:38:101
342991512,2cyclictest14056-21ntp_states11:35:193
340991513,1cyclictest0-21swapper/210:52:442
34099151,1cyclictest141rcu_preempt10:00:372
34099150,15cyclictest0-21swapper/212:34:262
34099150,15cyclictest0-21swapper/211:29:422
34099150,14cyclictest0-21swapper/210:22:332
34099150,14cyclictest0-21swapper/209:20:552
34099150,14cyclictest0-21swapper/208:30:112
336991514,1cyclictest0-21swapper/110:57:511
342991410,3cyclictest5920-21sed08:25:273
342991410,3cyclictest17555-21cron11:00:003
34099141,8cyclictest0-21swapper/211:00:142
340991411,2cyclictest18598-21wc07:45:222
340991410,3cyclictest29794-21grep11:45:142
34099140,11cyclictest0-21swapper/209:36:422
33699143,11cyclictest25645-21diskmemload11:24:581
33699140,1cyclictest0-21swapper/109:17:211
33199140,2cyclictest0-21swapper/009:26:440
33199140,13cyclictest0-21swapper/010:43:050
34299139,3cyclictest9576-21grep08:35:163
34299139,3cyclictest8306-21grep08:30:243
34299139,3cyclictest28046-21sed09:10:253
34299139,3cyclictest20459-21sed08:55:263
34299139,3cyclictest12711-21sed09:20:273
342991310,2cyclictest22341-21grep07:55:123
342991310,2cyclictest13808-21grep07:35:213
34299130,12cyclictest8145-21wc12:30:013
34099139,3cyclictest6618-21grep07:20:202
34099139,3cyclictest21896-21wc07:55:002
34099139,3cyclictest15069-21cron12:15:002
34099139,3cyclictest13520-21cron08:45:012
340991310,2cyclictest9662-21apache_volume10:55:142
340991310,2cyclictest8789-21sort07:25:192
340991310,2cyclictest28077-21sed08:05:222
340991310,2cyclictest24140-21tune2fs09:05:142
34099130,2cyclictest0-21swapper/210:40:172
34099130,13cyclictest25645-21diskmemload12:25:162
34099130,13cyclictest0-21swapper/212:38:232
33699139,3cyclictest20132-21grep09:44:591
336991311,1cyclictest0-21swapper/111:16:131
336991310,2cyclictest8778-21perf08:35:001
336991310,2cyclictest31278-21sed10:10:171
336991310,2cyclictest31278-21sed10:10:161
336991310,2cyclictest13748-21wc08:45:011
336991310,2cyclictest13366-21diskstats11:35:141
33699130,9cyclictest0-21swapper/110:00:181
33699130,13cyclictest0-21swapper/110:53:281
33699130,12cyclictest0-21swapper/108:55:131
33199138,4cyclictest322-21cyclictest11:38:510
33199134,3cyclictest0-21swapper/012:37:290
331991311,1cyclictest12022-21hddtemp_smartct11:15:180
331991310,2cyclictest20441-21grep07:50:160
331991310,2cyclictest16042-21grep11:55:270
33199130,12cyclictest14114-21grep08:45:140
33199130,12cyclictest13551-21awk08:45:010
34299129,2cyclictest16595-21sed07:40:263
34299129,2cyclictest1322-21head07:10:163
34299123,7cyclictest537-21snmpd11:47:123
34099129,2cyclictest7384-21head10:15:172
34099129,2cyclictest7121-21apache207:30:122
34099129,2cyclictest4054-21sh11:30:132
34099129,2cyclictest32558-21sort10:30:162
34099129,2cyclictest31743-21cut08:15:142
34099129,2cyclictest26202-21head08:00:272
34099129,2cyclictest24674-21grep10:25:172
34099129,2cyclictest20370-21sed08:55:262
34099129,2cyclictest17050-21cut08:50:172
34099129,2cyclictest17050-21cut08:50:162
34099129,2cyclictest14672-21cron07:40:002
34099129,2cyclictest13038-21grep11:15:272
34099125,2cyclictest0-21swapper/210:37:582
34099124,4cyclictest225-21systemd-journal08:40:012
34099120,12cyclictest0-21swapper/211:57:292
33699129,2cyclictest27461-21sort08:05:171
33699129,2cyclictest25159-21cut08:00:171
33699129,2cyclictest22280-21tr09:00:191
33699129,2cyclictest20183-21chrt10:22:521
33699129,2cyclictest20118-21grep11:00:261
33699127,4cyclictest23722-21tr09:05:121
33699126,5cyclictest8538-21grep07:25:181
33699122,1cyclictest25645-21diskmemload10:06:201
33699121,3cyclictest225-21systemd-journal08:18:441
33699120,9cyclictest0-21swapper/112:00:271
33699120,11cyclictest0-21swapper/111:29:501
33199129,2cyclictest29970-21head08:10:160
33199129,2cyclictest25475-21ls08:00:200
33199129,2cyclictest24122-21grep09:05:140
33199129,2cyclictest20878-21grep09:45:150
33199129,2cyclictest20077-21munin-run09:44:590
33199129,2cyclictest11681-21cat07:30:250
33199123,3cyclictest0-21swapper/010:46:200
34299118,2cyclictest8328-21seq09:56:223
34299118,2cyclictest3067-21tr07:15:133
34299118,2cyclictest25281-21taskset09:05:383
34299118,2cyclictest19820-21tr07:50:133
34299118,2cyclictest18214-21cat07:45:193
34299118,2cyclictest17489-21cut08:50:203
34299118,2cyclictest17489-21cut08:50:193
34299118,2cyclictest1479-21taskset08:19:033
34099118,2cyclictest7481-21sh09:55:432
34099118,2cyclictest4986-21sed08:25:172
34099118,2cyclictest3534-21grep08:20:272
34099118,2cyclictest29333-21wget08:10:142
34099118,2cyclictest28184-21ssh11:05:232
34099118,2cyclictest24704-21ssh11:41:362
34099118,2cyclictest23271-21basename09:05:002
34099118,2cyclictest2091-21ssh12:06:342
34099118,2cyclictest1809-21sed07:10:212
34099118,2cyclictest12921-21grep09:40:172
33699118,2cyclictest905-21cat07:10:131
33699118,2cyclictest7909-21ssh10:15:221
33699118,2cyclictest6146-21taskset08:27:471
33699118,2cyclictest23741-21grep07:55:231
33699118,2cyclictest17041-21grep08:50:171
33699118,2cyclictest17041-21grep08:50:161
33699117,3cyclictest3420-21grep11:10:171
33699117,3cyclictest17285-21ssh09:23:571
33699113,3cyclictest21833-21turbostat.cron07:55:001
33699111,9cyclictest0-21swapper/108:45:141
33699110,7cyclictest38150irq/131-snd_hda07:30:131
33199118,2cyclictest9283-21cut08:35:140
33199118,2cyclictest6469-21cat07:20:180
33199118,2cyclictest5904-21ssh10:33:510
33199118,2cyclictest2866-21grep11:10:140
33199118,2cyclictest25299-21ssh11:04:250
33199118,2cyclictest2063-21cpuspeed_turbos08:20:150
33199118,2cyclictest18960-21grep07:45:270
33199118,2cyclictest17068-21cron07:45:000
33199118,2cyclictest1670-21cat07:10:180
33199118,2cyclictest16237-21cat10:20:190
33199118,2cyclictest13909-21ssh12:14:050
33199118,2cyclictest11609-21sh09:20:170
33199117,3cyclictest24254-21grep12:20:140
33199116,2cyclictest225-21systemd-journal11:52:420
33199112,7cyclictest12570-21apache_accesses07:35:100
34299108,1cyclictest96452sleep307:26:283
34299108,1cyclictest5767-21expr07:20:143
34299108,1cyclictest26370-21taskset08:02:193
34299108,1cyclictest12733-21sendmail_mailqu08:40:203
34299108,1cyclictest10344-21egrep07:30:143
34299108,1cyclictest0-21swapper/309:00:143
34299107,2cyclictest27817-21wc08:05:203
34299107,2cyclictest2188-21expr08:20:153
34299107,2cyclictest15634-21grep08:45:263
34299104,4cyclictest29854-21fschecks_count08:10:173
34099108,1cyclictest15890-21taskset08:47:152
34099107,2cyclictest4215-21sed07:15:212
34099105,2cyclictest225-21systemd-journal07:44:382
33699108,1cyclictest13565-21sort07:35:181
33699107,2cyclictest5099-21cron07:20:011
33699107,2cyclictest29277-21cut08:10:131
33699107,2cyclictest2118-21cut08:20:151
33699107,2cyclictest17702-21expr07:45:151
33699107,2cyclictest15631-21cut07:40:161
33199108,2cyclictest21590-21expr09:00:150
33199108,1cyclictest8244-21expr07:25:150
33199108,1cyclictest3133-21grep07:15:150
33199107,2cyclictest31882-21tr08:15:150
33199107,2cyclictest27689-21basename08:05:190
3409997,1cyclictest23056-21sed07:55:162
3369997,1cyclictest7264-21taskset07:21:191
3319997,1cyclictest4225-21apache_volume08:25:130
3319997,1cyclictest22184-21basename07:55:110
3319997,1cyclictest16384-21taskset08:50:130
3319997,1cyclictest16384-21taskset08:50:120
3319996,2cyclictest18860-21tr08:55:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional