You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-01 - 16:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Aug 01, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
306982238162,25sleep20-21swapper/207:08:342
307582230162,56sleep00-21swapper/007:09:210
304742204169,23sleep10-21swapper/107:05:391
307762203167,24sleep30-21swapper/307:09:363
3125421610,5sleep23090899cyclictest12:00:252
2582521430,3sleep23090899cyclictest09:20:252
2470021350,4sleep33091399cyclictest09:20:143
309089910697,3cyclictest0-21swapper/211:29:322
189522570,5sleep03089599cyclictest10:20:180
26262530,2sleep10-21swapper/110:55:271
279802450,1sleep10-21swapper/111:55:151
143262450,1sleep20-21swapper/210:10:172
3089599379,13cyclictest7666-21munin-run11:10:010
3089599368,13cyclictest21813-21munin-run09:15:010
3089599354,15cyclictest1689-21munin-run12:10:010
30903993119,4cyclictest0-21swapper/111:30:031
3089599302,14cyclictest23389-21munin-run08:05:010
3089599301,14cyclictest5802-21munin-run07:25:010
3089599288,18cyclictest28419-21perf09:30:010
3089599272,18cyclictest0-21swapper/009:20:160
3089599260,16cyclictest7190-21latency12:20:180
3089599260,14cyclictest573-21/usr/sbin/munin09:40:220
121712260,2sleep10-21swapper/112:30:261
3091399255,4cyclictest27291-21ls09:25:203
3091399251,6cyclictest0-21swapper/311:30:033
3091399251,5cyclictest0-21swapper/310:31:223
3091399251,4cyclictest29505-21unixbench_multi08:15:253
30913992511,11cyclictest827-21runrttasks11:53:253
30913992511,11cyclictest827-21runrttasks11:53:253
30913992510,3cyclictest3900-21awk12:15:013
3090899251,5cyclictest808-21snmpd10:23:072
3090399252,15cyclictest16542-21sshd07:47:031
3090399251,11cyclictest0-21swapper/109:28:521
30903992510,6cyclictest0-21swapper/111:25:261
3089599252,12cyclictest12925-21grep11:20:160
3089599252,11cyclictest22654-21/usr/sbin/munin10:30:150
3089599251,14cyclictest29244-21/usr/sbin/munin10:45:150
3089599251,12cyclictest13005-21munin-run08:55:010
3089599250,16cyclictest0-21swapper/010:05:200
3089599250,14cyclictest2896-21taskset12:10:210
30908992418,3cyclictest11635-21grep08:50:162
3090899241,6cyclictest0-21swapper/210:05:052
3090399241,4cyclictest8663-21fschecks_count07:30:161
3089599242,14cyclictest0-21swapper/009:37:220
3089599241,10cyclictest23286-21acpi11:45:140
3089599240,18cyclictest0-21swapper/009:33:140
3089599240,18cyclictest0-21swapper/009:33:140
113002241,3sleep211348-21smart_sda11:15:232
3091399238,3cyclictest7896-21/usr/sbin/munin11:10:163
3091399231,11cyclictest4116-21/usr/sbin/munin12:15:273
3091399231,11cyclictest30206-21/usr/sbin/munin08:20:233
3091399231,11cyclictest10704-21/usr/sbin/munin12:30:173
3091399231,10cyclictest20682-21sshd11:38:033
3090899238,5cyclictest0-21swapper/212:15:232
3090899236,8cyclictest5622-21processes08:35:222
3090399232,11cyclictest20232-21munin-run10:25:011
3090399231,19cyclictest8230-21df11:10:161
3090399231,16cyclictest0-21swapper/108:24:311
3090399231,16cyclictest0-21swapper/107:11:411
3090399231,11cyclictest0-21swapper/111:50:271
3090399231,11cyclictest0-21swapper/111:50:271
3089599233,7cyclictest0-21swapper/009:46:290
3089599231,9cyclictest1062-21sendmail07:25:170
30895992315,5cyclictest31913-21/usr/sbin/munin07:10:180
3089599230,9cyclictest12558-21cat07:40:020
3089599230,8cyclictest15652-21cstates09:00:140
3089599230,8cyclictest0-21swapper/012:33:330
3089599230,11cyclictest0-21swapper/007:50:140
116322230,7sleep3401ktimersoftd/308:50:163
3091399225,4cyclictest19610-21diskmemload10:10:203
3091399224,4cyclictest2459-21uname07:15:173
30913992217,2cyclictest39-21rcuc/309:00:013
3091399221,11cyclictest819-21lldpd10:38:203
3090899223,7cyclictest17171-40dnf09:02:052
3090899223,2cyclictest0-21swapper/209:46:272
3090399222,11cyclictest16408-21diskstats10:15:141
30903992214,2cyclictest0-21swapper/107:24:511
3090399221,3cyclictest32559-21cpuspeed_turbos08:25:131
3090399221,3cyclictest27783-21munin-run08:15:011
3090399221,18cyclictest22077-21netstat11:40:201
3090399221,16cyclictest0-21swapper/111:05:171
3089599225,10cyclictest87350irq/126-enp1s0-11:15:150
3089599223,11cyclictest87650irq/129-enp1s0-12:25:260
3089599222,7cyclictest0-21swapper/012:17:290
3089599222,7cyclictest0-21swapper/010:15:210
3089599221,13cyclictest14481-21/usr/sbin/munin11:25:270
3089599221,11cyclictest87350irq/126-enp1s0-10:10:200
3089599220,9cyclictest6559-21unixbench_multi09:50:260
3089599220,10cyclictest25217-21forks10:35:180
3091399215,13cyclictest0-21swapper/307:47:033
3091399211,10cyclictest839-21lldpd10:43:373
3091399211,10cyclictest839-21lldpd10:43:373
3090899212,4cyclictest0-21swapper/209:41:232
3090899211,6cyclictest0-21swapper/209:28:532
3090399212,11cyclictest24001-21ls10:30:251
3090399211,3cyclictest13637-21munin-run10:10:011
30903992113,3cyclictest0-21swapper/109:02:231
30903992113,2cyclictest0-21swapper/107:40:151
3090399211,14cyclictest0-21swapper/110:45:461
3090399211,14cyclictest0-21swapper/110:35:471
3090399211,14cyclictest0-21swapper/109:47:241
3090399211,14cyclictest0-21swapper/109:05:301
3090399211,11cyclictest13234-21/usr/sbin/munin08:55:251
30903992110,5cyclictest22040-21/usr/sbin/munin09:15:141
3090399210,15cyclictest0-21swapper/112:17:081
3090399210,14cyclictest21-21rcuc/111:20:161
3089599215,4cyclictest20861-21sshd07:56:030
3089599214,10cyclictest0-21swapper/008:15:030
3089599211,8cyclictest26688-21gzip11:50:230
3089599211,8cyclictest26688-21gzip11:50:230
30895992114,4cyclictest89102sleep009:56:030
30895992113,5cyclictest13952-21cat08:55:180
3089599211,11cyclictest19610-21diskmemload11:40:250
3089599210,19cyclictest0-21swapper/009:16:270
3091399207,12cyclictest41-21ksoftirqd/312:05:243
3091399205,8cyclictest87350irq/126-enp1s0-09:30:233
3091399205,8cyclictest87350irq/126-enp1s0-09:30:233
3091399204,3cyclictest41-21ksoftirqd/309:45:163
3091399201,7cyclictest0-21swapper/311:45:223
3091399201,10cyclictest27371-21kworker/u8:111:59:093
30913992010,3cyclictest10093-21cat11:15:123
3090899208,4cyclictest0-21swapper/210:04:592
3090899207,5cyclictest7613-21switchtime12:20:252
3090899201,6cyclictest0-21swapper/210:40:522
3090899201,6cyclictest0-21swapper/210:40:522
3090399209,5cyclictest11422-21kworker/u8:010:10:321
3090399209,3cyclictest0-21swapper/112:10:071
3090399206,7cyclictest30596-21turbostat09:35:011
3090399206,7cyclictest30596-21turbostat09:35:011
3090399206,6cyclictest0-21swapper/109:23:341
3090399206,3cyclictest87550irq/128-enp1s0-08:40:161
3090399201,9cyclictest0-21swapper/112:25:141
3090399201,6cyclictest11195-21sshd10:02:031
3090399201,3cyclictest3609-21grep11:00:121
30903992012,2cyclictest0-21swapper/108:02:081
3090399201,14cyclictest0-21swapper/112:23:161
3090399201,14cyclictest0-21swapper/108:30:291
3090399201,14cyclictest0-21swapper/107:27:011
3090399201,14cyclictest0-21swapper/107:17:161
3090399201,13cyclictest0-21swapper/109:12:541
3090399201,11cyclictest31537-21/usr/sbin/munin09:35:171
3090399201,11cyclictest13014-21cpuspeed_turbos12:35:131
3090399201,10cyclictest9649-21irqstats08:45:171
3090399201,10cyclictest8539-21switchtime09:55:271
3090399200,15cyclictest0-21swapper/110:50:271
3090399200,14cyclictest0-21swapper/107:55:391
3090399200,14cyclictest0-21swapper/107:38:591
3089599208,3cyclictest20458-21/usr/sbin/munin10:25:150
3089599202,10cyclictest1628-21/usr/sbin/munin07:15:140
30895992014,4cyclictest8897-21tail07:30:170
3089599200,9cyclictest0-21swapper/010:50:140
3089599200,8cyclictest29921-21kworker/u8:011:03:050
3089599200,8cyclictest0-21swapper/008:22:460
3089599200,12cyclictest87650irq/129-enp1s0-10:55:190
3089599200,12cyclictest28907-21chrt10:43:460
3089599200,12cyclictest28907-21chrt10:43:460
3089599200,11cyclictest28717-21ntpq11:55:220
3089599200,11cyclictest0-21swapper/012:35:200
299372201,3sleep130196-21df12:00:131
3091399198,3cyclictest839-21lldpd10:21:073
3091399197,3cyclictest0-21swapper/310:04:433
3091399195,8cyclictest31708-40dnf12:03:253
3091399194,3cyclictest31485-21fschecks_count07:10:153
3091399194,3cyclictest0-21swapper/311:00:233
3091399193,4cyclictest7680-21fschecks_count09:55:173
3091399193,4cyclictest27217-21timerwakeupswit08:10:273
3091399191,6cyclictest0-21swapper/310:45:183
3091399191,6cyclictest0-21swapper/309:37:373
3091399191,5cyclictest0-21swapper/309:53:553
3091399191,4cyclictest12288-21/usr/sbin/munin11:20:183
3090899197,3cyclictest1602-21grep08:25:262
3090899195,5cyclictest1034-21crond10:55:012
3090899195,4cyclictest1462-21gltestperf07:15:042
3090899193,3cyclictest27404-21/usr/sbin/munin08:10:282
3090899192,15cyclictest495-21jbd2/dm-0-808:15:222
3090899191,4cyclictest0-21swapper/211:55:182
30908991914,3cyclictest2574-21cut12:10:172
3090899191,3cyclictest17931-21memory07:50:202
3090899191,12cyclictest0-21swapper/210:46:112
3090399196,5cyclictest0-21swapper/109:44:381
30903991912,2cyclictest0-21swapper/107:51:261
3090399191,16cyclictest0-21swapper/111:45:051
3090399191,10cyclictest11048-21cat11:15:191
3090399190,14cyclictest0-21swapper/108:36:181
3090399190,14cyclictest0-21swapper/108:15:281
3089599199,6cyclictest19610-21diskmemload11:10:270
3089599195,5cyclictest0-21swapper/011:38:130
3089599192,5cyclictest23617-21/usr/sbin/munin08:05:160
3089599192,10cyclictest19610-21diskmemload12:00:550
3089599191,11cyclictest12622-21/usr/sbin/munin07:40:280
3089599191,10cyclictest9807-21diskstats10:00:150
3089599191,10cyclictest8838-21/usr/sbin/munin08:45:170
3089599191,10cyclictest17642-21/usr/sbin/munin09:05:220
3089599190,9cyclictest0-21swapper/008:35:200
3089599190,8cyclictest0-21swapper/011:30:150
3089599190,17cyclictest0-21swapper/008:25:250
3091399185,3cyclictest12050irq/123-ahci[0011:07:043
3091399184,8cyclictest799-21munin-node09:02:063
3091399181,6cyclictest0-21swapper/308:32:533
3091399181,5cyclictest4443-21/usr/sbin/munin08:35:133
3091399181,5cyclictest21412-21/usr/sbin/munin08:00:263
30913991812,4cyclictest10024-21tail12:25:283
30913991812,3cyclictest10422-21users08:45:253
3091399181,11cyclictest0-21swapper/310:05:563
3091399180,9cyclictest39-21rcuc/311:30:013
3090899186,3cyclictest12050irq/123-ahci[0008:35:012
3090899185,5cyclictest23037-21dump-pmu-power11:49:562
3090899181,7cyclictest24619-21latency_hist10:35:012
3090899181,7cyclictest0-21swapper/211:22:472
3090899181,6cyclictest31516-21hwlatdetect09:35:172
3090899181,4cyclictest0-21swapper/211:30:562
30908991813,3cyclictest31436-21df07:10:142
3090899181,2cyclictest0-21swapper/212:33:582
3090899181,15cyclictest0-21swapper/211:52:532
3090899181,15cyclictest0-21swapper/211:52:532
3090899180,5cyclictest0-21swapper/210:38:272
3090399185,4cyclictest0-21swapper/109:51:551
3090399183,9cyclictest0-21swapper/110:25:041
3090399183,3cyclictest0-21swapper/108:09:331
3090399181,16cyclictest0-21swapper/112:06:131
3090399180,17cyclictest0-21swapper/110:40:011
3090399180,17cyclictest0-21swapper/110:40:011
3090399180,16cyclictest0-21swapper/108:51:101
3090399180,12cyclictest19557-21kworker/u8:211:35:251
30895991813,3cyclictest6798-21tr08:40:130
3091399177,3cyclictest0-21swapper/308:06:273
3091399175,4cyclictest0-21swapper/309:18:083
3091399172,8cyclictest41-21ksoftirqd/311:41:483
3091399172,4cyclictest17467-21fschecks_count07:50:163
3091399171,5cyclictest16697-21gltestperf10:15:163
3091399171,15cyclictest0-21swapper/307:55:533
3091399171,15cyclictest0-21swapper/307:21:163
3091399171,14cyclictest0-21swapper/307:42:443
3091399171,14cyclictest0-21swapper/307:26:363
3091399171,13cyclictest8228-21/usr/sbin/munin07:30:173
3091399171,13cyclictest31452-21/usr/sbin/munin10:50:153
3091399171,13cyclictest19845-21/usr/sbin/munin09:10:213
30913991711,3cyclictest1272-21/usr/sbin/munin10:55:213
3091399171,13cyclictest10423-21/usr/sbin/munin07:35:243
30913991710,3cyclictest7225-21gltestperf08:40:163
3091399170,6cyclictest0-21swapper/312:20:153
3090899174,4cyclictest14317-21sed12:35:252
3090899174,4cyclictest14273-21ntp_kernel_pll_08:55:222
3090899171,8cyclictest0-21swapper/207:26:112
3090899171,6cyclictest30550-21df_inode08:20:132
3090899171,6cyclictest21174-21tr11:40:132
3090899171,6cyclictest0-21swapper/212:27:282
3090899171,6cyclictest0-21swapper/210:55:352
3090899171,6cyclictest0-21swapper/210:25:262
3090899171,6cyclictest0-21swapper/210:15:372
3090899171,6cyclictest0-21swapper/209:16:452
3090899171,5cyclictest9159-21smart_sda11:10:272
3090899171,3cyclictest87350irq/126-enp1s0-11:36:452
30908991711,3cyclictest21179-21threads09:10:242
3090899171,10cyclictest0-21swapper/209:56:132
3089599176,3cyclictest14817-21/usr/sbin/munin07:45:210
30895991713,3cyclictest0-21swapper/008:30:180
3091399161,8cyclictest0-21swapper/309:05:143
3091399161,7cyclictest0-21swapper/308:25:143
3091399161,6cyclictest19610-21diskmemload09:40:143
30913991610,3cyclictest20458-21/usr/sbin/munin10:25:193
30913991610,3cyclictest13104-21expr12:35:133
3090899163,6cyclictest0-21swapper/211:07:352
3090899161,7cyclictest0-21swapper/207:20:112
3090899161,6cyclictest8925-21/usr/sbin/munin07:30:182
3090899161,6cyclictest5877-21irqstats09:50:182
3090899161,6cyclictest0-21swapper/208:00:242
3090899161,5cyclictest7703-21ntp_states08:40:222
3090899161,3cyclictest0-21swapper/209:31:322
3090899161,3cyclictest0-21swapper/209:31:322
30908991612,3cyclictest0-21swapper/209:05:542
3090899161,14cyclictest0-21swapper/208:47:072
3090899161,14cyclictest0-21swapper/207:58:372
3090899161,13cyclictest4148-21grep11:00:162
30908991611,3cyclictest32076-21/usr/sbin/munin12:05:192
30908991611,3cyclictest25628-21sed08:10:012
30908991611,3cyclictest12622-21/usr/sbin/munin07:40:202
30908991611,3cyclictest10423-21/usr/sbin/munin07:35:252
3089599167,6cyclictest25999-21cstates08:10:140
30908991510,3cyclictest14804-21seq07:45:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional