You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-16 - 00:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Jul 15, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2169529582,9sleep30-21swapper/307:06:203
2196929480,10sleep10-21swapper/107:09:531
2169727966,9sleep00-21swapper/007:06:220
2191327666,6sleep20-21swapper/207:09:102
246192640,2sleep124626-21turbostat.cron07:15:001
206242570,0sleep00-21swapper/009:40:560
2207899270,27cyclictest0-21swapper/212:24:112
2207299271,25cyclictest20877-21ssh10:10:150
2207299260,3cyclictest0-21swapper/011:00:160
2208199250,24cyclictest0-21swapper/312:10:173
2207899250,23cyclictest0-21swapper/210:41:122
2207899241,23cyclictest0-21swapper/211:23:162
2207299240,3cyclictest0-21swapper/011:22:100
2207899230,22cyclictest0-21swapper/210:48:232
2207899230,22cyclictest0-21swapper/210:48:222
22081992219,2cyclictest94-21kswapd009:59:393
2207899220,20cyclictest0-21swapper/211:55:242
2207299221,2cyclictest14825-21diskmemload12:15:400
2207899210,21cyclictest0-21swapper/210:35:182
2207299214,1cyclictest0-21swapper/011:50:330
2207299212,1cyclictest12580-21ssh09:35:180
22072992121,0cyclictest0-21swapper/010:54:370
2207299211,2cyclictest14186-21grep12:35:160
2207299210,0cyclictest0-21swapper/011:34:100
2207899200,18cyclictest0-21swapper/212:30:122
2207299202,1cyclictest3345-21aten2_r7power_p09:55:170
2207299201,4cyclictest9944-21apt-get12:10:180
2207299200,1cyclictest0-21swapper/011:35:380
2207899192,14cyclictest32-21ksoftirqd/211:30:142
2207899190,17cyclictest0-21swapper/210:29:232
22075991916,1cyclictest0-21swapper/111:50:191
2207299195,1cyclictest14382-21perf11:50:000
2207299192,3cyclictest26641-21latency_hist09:20:000
2207299192,3cyclictest14825-21diskmemload10:21:190
2207299191,4cyclictest20331-21grep10:35:290
2207299191,2cyclictest10975-21grep09:00:200
2207299190,8cyclictest0-21swapper/010:19:510
2207299190,4cyclictest15920-21fschecks_count09:10:170
2207299190,2cyclictest0-21swapper/009:48:500
2207299190,16cyclictest0-21swapper/007:25:160
310212180,0chrt0-21swapper/209:50:202
22081991817,1cyclictest0-21swapper/310:08:293
2208199180,18cyclictest0-21swapper/312:17:103
2208199180,18cyclictest0-21swapper/311:54:493
2208199180,18cyclictest0-21swapper/311:40:293
2208199180,18cyclictest0-21swapper/310:42:493
2208199180,18cyclictest0-21swapper/309:18:303
2207899184,8cyclictest224-21systemd-journal09:13:352
2207599180,18cyclictest0-21swapper/111:20:561
2207299182,2cyclictest0-21swapper/010:27:220
2207299181,17cyclictest0-21swapper/011:25:230
2208199172,14cyclictest14997-21sensors12:35:223
22081991717,0cyclictest0-21swapper/310:52:303
22081991716,1cyclictest0-21swapper/312:24:293
2208199170,13cyclictest0-21swapper/311:13:483
2207899171,15cyclictest0-21swapper/209:22:342
2207899170,1cyclictest0-21swapper/212:05:562
2207899170,17cyclictest0-21swapper/211:48:372
2207899170,17cyclictest0-21swapper/210:58:112
2207899170,15cyclictest0-21swapper/210:24:182
2207299176,10cyclictest0-21swapper/010:00:190
2207299173,13cyclictest14825-21diskmemload11:55:260
2207299172,3cyclictest27969-21cron08:30:010
2207299171,3cyclictest0-21swapper/012:08:460
2207299170,3cyclictest0-21swapper/010:45:240
2207299170,3cyclictest0-21swapper/010:45:230
22081991615,1cyclictest0-21swapper/312:33:293
22081991615,1cyclictest0-21swapper/312:04:293
22081991615,1cyclictest0-21swapper/311:57:303
22081991615,1cyclictest0-21swapper/311:25:103
22081991615,1cyclictest0-21swapper/310:22:293
22081991615,1cyclictest0-21swapper/309:27:103
2207899160,16cyclictest0-21swapper/210:32:112
2207899160,16cyclictest0-21swapper/209:38:042
2207899160,16cyclictest0-21swapper/209:19:532
2207299168,6cyclictest7646-21ntp_states09:30:210
2207299160,15cyclictest0-21swapper/011:44:040
22081991515,0cyclictest0-21swapper/311:46:303
22081991514,1cyclictest0-21swapper/309:46:293
22081991514,1cyclictest0-21swapper/309:24:493
22081991511,3cyclictest39-21ksoftirqd/307:20:163
2208199150,13cyclictest0-21swapper/310:19:253
22078991512,2cyclictest9881-21/usr/sbin/munin12:10:162
2207899151,13cyclictest7817-21latency_hist11:45:002
2207899150,15cyclictest0-21swapper/212:35:222
2207899150,15cyclictest0-21swapper/212:16:112
2207899150,15cyclictest0-21swapper/212:02:352
2207299154,1cyclictest0-21swapper/011:06:570
2207299152,5cyclictest15624-21grep08:00:270
2207299151,3cyclictest14825-21diskmemload10:40:150
2207299150,1cyclictest0-21swapper/009:51:150
2208199149,3cyclictest18597-21grep10:35:173
22081991411,2cyclictest7215-21head09:30:183
22081991411,2cyclictest29691-21wc08:30:273
22081991410,3cyclictest2965-21ssh12:05:143
22081991410,3cyclictest20312-21tail08:10:263
22078991414,0cyclictest0-21swapper/209:27:372
2207899141,1cyclictest14299-21munin-run10:05:002
22078991410,3cyclictest24812-21cat07:15:002
22078991410,2cyclictest23237-21awk08:20:012
2207899140,14cyclictest0-21swapper/210:50:162
2207599141,2cyclictest0-21swapper/110:35:151
22075991410,3cyclictest10647-21cut12:10:201
2207299148,5cyclictest6851-21grep07:45:160
2207299148,5cyclictest6211-21taskset07:41:330
2207299148,5cyclictest5502-21sh09:29:150
2207299148,5cyclictest4892-21rm11:16:500
2207299148,5cyclictest15365-21expr10:05:160
2207299148,5cyclictest1290-21ssh12:03:510
2207299147,6cyclictest9174-21tr07:50:150
2207299147,6cyclictest16776-21df_abs08:05:200
2207299146,7cyclictest23931-21expr08:20:170
2207299143,2cyclictest14825-21diskmemload09:22:070
2207299142,3cyclictest9623-21unixbench_singl08:55:270
2207299141,11cyclictest0-21swapper/012:30:150
2207299140,8cyclictest0-21swapper/008:10:130
2207299140,3cyclictest13459-21head09:05:190
2207299140,13cyclictest17912-21ssh10:34:520
2207299140,13cyclictest12590-21ssh10:56:170
2208199134,5cyclictest26370-21idleruntime-cro10:15:003
22081991312,1cyclictest0-21swapper/309:38:103
22081991310,2cyclictest30979-21grep09:50:203
22081991310,2cyclictest21957-21grep11:30:203
2207899139,3cyclictest17004-21wc11:00:212
2207899133,7cyclictest14810-21cron09:10:012
2207899132,1cyclictest31588-21apt-get12:25:142
22078991312,1cyclictest0-21swapper/209:58:322
22078991310,2cyclictest8861-21cron07:50:002
22078991310,2cyclictest26341-21cat07:15:282
2207899130,13cyclictest0-21swapper/211:12:232
2207899130,11cyclictest27527-21tr07:20:152
2207599139,2cyclictest32277-21cron10:20:001
22075991310,2cyclictest3645-21grep11:15:271
22075991310,2cyclictest30574-21ssh09:50:161
22075991310,2cyclictest23217-21sendmail08:20:001
22075991310,2cyclictest1568-21grep08:40:191
22075991310,2cyclictest1228-21grep07:30:261
22075991310,2cyclictest10408-21wc07:50:301
2207599130,8cyclictest0-21swapper/109:55:201
2207599130,12cyclictest0-21swapper/112:15:151
2207599130,11cyclictest24238-21head08:20:191
2207299138,4cyclictest25201-21cat07:15:190
2207299138,4cyclictest23246-21head07:10:180
2207299137,5cyclictest5016-21taskset12:29:100
2207299137,5cyclictest25350-21grep12:20:200
2207299137,5cyclictest22419-21find08:15:230
2207299136,6cyclictest29006-21expr08:30:170
2207299132,3cyclictest29411-21munin-run07:25:000
2207299130,7cyclictest0-21swapper/008:45:000
2208199129,2cyclictest980-21grep12:25:283
2208199129,2cyclictest9663-21ssh10:00:223
2208199129,2cyclictest9622-21grep08:55:283
2208199129,2cyclictest7318-21grep10:25:173
2208199129,2cyclictest476-21cron08:40:003
2208199129,2cyclictest16514-21tr08:05:203
2208199129,2cyclictest10803-21sed07:50:343
2208199124,7cyclictest14807-21cron09:10:013
2208199124,4cyclictest224-21systemd-journal10:50:013
2208199124,4cyclictest224-21systemd-journal10:50:003
2207899129,2cyclictest32281-21sh10:20:002
2207899129,2cyclictest26581-21grep08:25:202
2207899129,2cyclictest2591-21sort07:35:162
2207899129,2cyclictest23759-21tr08:20:182
2207899129,2cyclictest15744-21wc11:25:272
2207899129,2cyclictest1251-21grep07:30:272
2207899128,3cyclictest23388-21sh09:44:592
2207899120,11cyclictest0-21swapper/207:55:172
2207599129,2cyclictest6809-21tr10:25:151
2207599129,2cyclictest3319-21grep08:45:151
2207599129,2cyclictest25718-21wc12:20:231
2207599129,2cyclictest22653-21head11:30:271
2207599129,2cyclictest21974-21ssh11:05:061
2207599129,2cyclictest14377-21basename11:50:001
2207599129,2cyclictest12859-21tr10:30:361
2207599129,2cyclictest10874-21df09:00:181
2207599128,3cyclictest2404-21grep09:25:291
2207599128,3cyclictest19524-21cat09:40:201
2207599126,5cyclictest21588-21apache210:40:141
2207599124,6cyclictest4929-21df07:40:151
2207599124,6cyclictest0-21swapper/107:45:151
2207599120,5cyclictest0-21swapper/109:11:271
2207599120,2cyclictest0-21swapper/109:23:361
2207599120,1cyclictest141rcu_preempt10:46:531
2207599120,1cyclictest141rcu_preempt10:46:531
2207599120,11cyclictest19409-21head08:10:171
2207299127,4cyclictest2456-21expr07:35:160
2207299126,5cyclictest7583-21sshd08:54:110
2207299126,5cyclictest11838-21cat07:55:180
2207299125,6cyclictest473-21cron08:40:010
2207299125,6cyclictest30943-21ssh11:11:200
2207299122,2cyclictest224-21systemd-journal07:31:360
2207299121,2cyclictest224-21systemd-journal08:50:000
2208199118,2cyclictest6098-21grep07:40:253
2208199118,2cyclictest32047-21taskset07:30:153
2208199118,2cyclictest2924-21grep07:35:193
2208199118,2cyclictest27330-21rm11:35:093
2208199118,2cyclictest26572-21cut08:25:203
2208199118,2cyclictest24985-21ssh11:06:393
2208199118,2cyclictest23638-21awk07:10:223
2208199118,2cyclictest23101-21taskset08:18:463
2208199118,2cyclictest20143-21ssh11:03:143
2208199118,2cyclictest14655-21ssh10:31:093
2208199118,2cyclictest11841-21sed10:55:293
2208199118,2cyclictest10876-21grep09:00:183
2208199118,2cyclictest10377-21ssh11:21:193
2208199117,3cyclictest2449-21grep08:40:273
2208199115,4cyclictest20264-21vmstat09:40:283
2208199110,8cyclictest0-21swapper/307:58:593
2208199110,10cyclictest3076-21sed11:15:203
2207899118,2cyclictest9735-21taskset08:56:162
2207899118,2cyclictest4504-21wc08:45:232
2207899118,2cyclictest29407-21sh08:30:222
2207899118,2cyclictest25865-21ssh10:14:272
2207899118,2cyclictest25098-21ssh09:45:242
2207899118,2cyclictest2188-21sh11:15:152
2207899118,2cyclictest18053-21ssh10:07:002
2207899118,2cyclictest15427-21tr11:50:212
2207899118,2cyclictest14239-21expr08:00:162
2207899118,2cyclictest12495-21cat09:05:002
2207899118,2cyclictest10880-21ssh09:33:462
2207899117,3cyclictest6261-21grep08:50:192
2207899110,0cyclictest0-21swapper/211:06:032
2207599118,2cyclictest9931-21sh10:55:091
2207599118,2cyclictest9290-21ssh10:00:191
2207599118,2cyclictest6977-21ssh10:51:281
2207599118,2cyclictest6320-21head08:50:191
2207599118,2cyclictest5957-21ssh11:43:021
2207599118,2cyclictest32181-21chrt11:12:541
2207599118,2cyclictest31911-21expr12:25:151
2207599118,2cyclictest30995-21which07:25:261
2207599118,2cyclictest29915-21grep08:30:261
2207599118,2cyclictest29893-21ssh11:36:011
2207599118,2cyclictest28125-21awk12:00:001
2207599118,2cyclictest26863-21cat08:25:241
2207599118,2cyclictest24786-21if_enp2s009:45:211
2207599118,2cyclictest24041-21seq10:12:111
2207599118,2cyclictest22469-21wc09:15:281
2207599118,2cyclictest2222-21expr07:35:141
2207599118,2cyclictest16401-21expr11:00:161
2207599118,2cyclictest15035-21cut11:25:201
2207599118,2cyclictest14979-21sh10:05:131
2207599118,2cyclictest13339-21ssh12:35:071
2207599116,2cyclictest493-21systemd-logind12:09:211
2207599115,2cyclictest224-21systemd-journal12:02:351
2207599114,6cyclictest6109-21dump-pmu-power10:25:011
2207599110,5cyclictest0-21swapper/109:05:151
2207599110,10cyclictest0-21swapper/112:30:071
2208199108,1cyclictest30371-21tr07:25:193
2208199108,1cyclictest26594-21head07:15:293
2208199108,1cyclictest25527-21chrt08:24:113
2208199107,2cyclictest7512-21tail07:45:203
2208199107,2cyclictest7191-21tail08:50:273
2208199107,2cyclictest3949-21grep08:45:173
2208199107,2cyclictest15346-21grep09:10:133
2207899108,1cyclictest4839-21cat07:40:142
2207899108,1cyclictest30358-21grep07:25:182
2207899107,2cyclictest31061-21tr08:35:192
2207899107,2cyclictest28687-21ssh11:35:232
2207899107,2cyclictest2496-21grep08:40:282
2207899107,2cyclictest19600-21cut08:10:182
2207899107,2cyclictest16909-21cut08:05:222
2207899100,9cyclictest111162sleep207:53:312
2207599108,1cyclictest7893-21taskset08:55:081
2207599108,1cyclictest31768-21kernelversion08:35:231
2207599108,1cyclictest157822sleep108:02:081
2207599108,1cyclictest11884-21expr07:55:191
2207599107,2cyclictest7262-21grep09:30:181
2207599107,2cyclictest29103-21chrt07:21:211
2207599107,2cyclictest25512-21tune2fs07:15:201
2207599107,2cyclictest16540-21expr08:05:201
2207599107,2cyclictest13081-21tune2fs09:35:211
220819990,8cyclictest0-21swapper/308:00:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional