You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 23:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue Jul 07, 2026 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
720429343,46sleep10-21swapper/107:05:501
737028961,9sleep30-21swapper/307:08:013
727028169,8sleep00-21swapper/007:06:410
726728068,8sleep20-21swapper/207:06:402
174342600,0sleep00-21swapper/011:52:530
761999290,29cyclictest0-21swapper/209:16:332
7624992420,3cyclictest30156-21kworker/3:110:09:333
761999240,24cyclictest0-21swapper/212:33:572
761999240,1cyclictest0-21swapper/210:08:292
761699241,1cyclictest141rcu_preempt10:14:131
761699240,1cyclictest0-21swapper/112:18:521
7616992321,1cyclictest18750-21dpkg11:35:001
761699230,1cyclictest369-21diskmemload12:30:351
761999220,22cyclictest0-21swapper/210:41:122
761699225,16cyclictest369-21diskmemload11:13:341
761699221,19cyclictest7447-21head12:25:161
214172220,0sleep00-21swapper/011:55:160
7624992119,1cyclictest30156-21kworker/3:110:48:433
7624992117,3cyclictest30156-21kworker/3:112:24:533
762499210,21cyclictest0-21swapper/310:19:493
761499215,8cyclictest377-21munin-run09:10:000
762499202,1cyclictest369-21diskmemload11:55:523
7624992018,2cyclictest0-21swapper/310:59:493
7619992018,2cyclictest32-21ksoftirqd/210:25:332
761999200,20cyclictest0-21swapper/210:58:242
761999200,18cyclictest0-21swapper/212:00:052
761699200,19cyclictest0-21swapper/110:45:331
7624991916,2cyclictest39-21ksoftirqd/310:44:243
762499191,14cyclictest30156-21kworker/3:111:02:273
762499190,1cyclictest0-21swapper/310:22:553
761999190,19cyclictest0-21swapper/211:58:232
761999190,19cyclictest0-21swapper/211:38:162
7616991914,1cyclictest25-21ksoftirqd/111:03:031
761699190,17cyclictest0-21swapper/110:05:121
761499195,7cyclictest13024-21basename12:10:000
7614991918,1cyclictest0-21swapper/012:17:190
761499190,19cyclictest0-21swapper/012:13:530
761499190,18cyclictest0-21swapper/010:51:380
7624991814,4cyclictest39-21ksoftirqd/310:13:513
761999180,16cyclictest0-21swapper/212:08:522
761499188,5cyclictest22160-21perf07:40:010
761499187,6cyclictest31319-21perf10:45:000
7616991714,2cyclictest19832-21ssh10:18:431
761699170,2cyclictest0-21swapper/107:15:141
761699170,1cyclictest0-21swapper/109:28:211
761699170,15cyclictest0-21swapper/109:39:091
761699170,15cyclictest0-21swapper/109:39:081
761499176,6cyclictest347-21ls11:05:000
761499176,5cyclictest23314-21idleruntime-cro08:50:000
7614991717,0cyclictest0-21swapper/010:39:210
7624991614,2cyclictest39-21ksoftirqd/309:23:073
7624991612,3cyclictest24531-21ssh12:35:193
761999166,5cyclictest225-21systemd-journal10:14:592
761999161,14cyclictest0-21swapper/211:10:132
761999160,16cyclictest0-21swapper/208:47:082
761999160,13cyclictest0-21swapper/210:52:242
761699163,10cyclictest369-21diskmemload12:36:451
7616991615,1cyclictest0-21swapper/109:55:541
761499160,8cyclictest0-21swapper/010:04:360
761499160,4cyclictest0-21swapper/012:36:550
761499160,0cyclictest0-21swapper/010:07:000
762499152,1cyclictest141rcu_preempt10:27:563
7624991512,3cyclictest30156-21kworker/3:109:59:233
7624991511,3cyclictest30156-21kworker/3:112:30:433
761999154,5cyclictest537-21snmpd10:24:082
761999152,12cyclictest0-21swapper/209:28:402
761699152,12cyclictest369-21diskmemload09:30:371
7616991514,1cyclictest0-21swapper/111:23:091
761699151,2cyclictest225-21systemd-journal11:20:001
761699150,15cyclictest0-21swapper/110:28:191
761699150,14cyclictest0-21swapper/110:34:231
761499153,6cyclictest6153-21basename12:25:000
761499151,8cyclictest17770-21kworker/u8:110:20:280
762499149,2cyclictest225-21systemd-journal10:36:163
7624991412,1cyclictest0-21swapper/311:20:003
7624991410,3cyclictest26208-21tr08:55:143
7624991410,3cyclictest16882-21grep07:25:253
7624991410,2cyclictest18353-21tr07:30:213
762499140,13cyclictest4684-21sed11:45:173
7619991411,2cyclictest12550-21readlink07:20:002
761999140,13cyclictest0-21swapper/209:32:582
761999140,12cyclictest936-21cut11:05:142
761699149,3cyclictest1853-21ls09:10:181
7616991413,1cyclictest241rcuc/108:15:001
761699141,1cyclictest369-21diskmemload09:43:181
761499144,6cyclictest6588-21taskset09:51:010
761499141,7cyclictest2136-21awk09:30:000
7614991413,1cyclictest0-21swapper/009:47:070
7614991410,2cyclictest31635-21cron08:00:010
761499140,12cyclictest20178-21grep09:40:160
286862140,1sleep30-21swapper/311:40:173
762499139,3cyclictest8440-21sort10:50:153
762499139,3cyclictest14252-21cut12:10:153
762499138,2cyclictest2491-21sh11:05:263
762499134,5cyclictest17310-21sendmail09:20:003
7624991311,2cyclictest30156-21kworker/3:109:45:143
7624991311,2cyclictest30156-21kworker/3:109:28:133
7624991311,1cyclictest0-21swapper/311:20:133
7624991310,2cyclictest2832-21grep08:05:193
7624991310,2cyclictest21956-21head08:45:203
761999139,3cyclictest818-21grep08:00:202
761999139,3cyclictest27216-21uname08:55:212
761999139,3cyclictest12531-21tr11:50:152
761999139,2cyclictest30466-21grep09:05:012
761999133,8cyclictest13947-21apache209:54:412
761999131,11cyclictest1845-21sh11:25:002
7619991311,1cyclictest14169-21wc09:55:262
7619991310,2cyclictest30849-21grep12:20:152
7619991310,2cyclictest2623-21grep11:25:142
7619991310,2cyclictest25962-21cat11:00:192
7619991310,2cyclictest21566-21grep12:15:002
7619991310,2cyclictest18043-21grep08:35:272
7619991310,2cyclictest15917-21cut07:25:152
761699139,3cyclictest4142-21tr11:45:141
761699139,2cyclictest369-21diskmemload09:19:441
761699132,3cyclictest225-21systemd-journal08:08:421
761699131,9cyclictest369-21diskmemload12:00:101
7616991310,2cyclictest9964-21perf07:15:001
7616991310,2cyclictest30641-21cat09:05:011
761699130,1cyclictest141rcu_preempt08:40:141
761699130,1cyclictest0-21swapper/111:54:011
761499139,3cyclictest19299-21grep07:30:290
761499138,3cyclictest386-21tr08:00:170
761499134,5cyclictest2883-21df11:25:150
7614991310,2cyclictest9521-21tr08:20:160
7614991310,2cyclictest1686-21grep11:05:170
7614991310,2cyclictest15914-21grep12:30:190
7614991310,2cyclictest10981-21sed07:15:180
761499130,2cyclictest0-21swapper/010:15:140
761499130,13cyclictest0-21swapper/009:24:400
762499129,2cyclictest32636-21grep08:00:153
762499129,2cyclictest29448-21strings09:00:193
762499129,2cyclictest24249-21sort08:50:173
762499129,2cyclictest23953-21grep12:15:293
762499129,2cyclictest13591-21grep07:20:163
762499129,2cyclictest11296-21grep09:35:153
762499129,2cyclictest11296-21grep09:35:153
762499128,3cyclictest20302-21tail09:40:163
762499128,3cyclictest10306-21ssh11:10:193
762499127,3cyclictest10875-21grep08:20:283
762499124,4cyclictest225-21systemd-journal11:34:593
7624991210,2cyclictest30156-21kworker/3:109:32:433
7624991210,1cyclictest13665-21wc11:50:223
762499120,9cyclictest8248-21sensors12:25:233
762499120,8cyclictest12116-21sshd12:09:193
762499120,0cyclictest0-21swapper/312:03:413
761999129,2cyclictest30155-21grep07:55:162
761999129,2cyclictest28799-21sed07:50:242
761999129,2cyclictest25276-21chrt12:16:262
761999129,2cyclictest13586-21ssh10:15:122
761999128,3cyclictest20366-21ssh10:38:042
7619991210,1cyclictest32143-21tr10:45:142
7619991210,1cyclictest26007-21grep07:45:232
7619991210,1cyclictest26007-21grep07:45:222
761999120,12cyclictest0-21swapper/211:15:102
761999120,11cyclictest8445-21grep10:30:272
761999120,11cyclictest4600-21ssh11:45:172
761699129,2cyclictest24340-21sed08:50:171
761699129,2cyclictest22527-21wc08:45:241
761699129,2cyclictest21113-21cut10:00:161
761699129,2cyclictest18864-21head09:20:171
761699129,2cyclictest15582-21sed10:35:161
761699128,3cyclictest8756-21taskset08:17:271
761699126,5cyclictest449-21irqbalance12:11:211
761699121,2cyclictest0-21swapper/111:55:131
7616991210,1cyclictest25-21ksoftirqd/112:05:231
7616991210,1cyclictest241rcuc/109:55:001
761699120,1cyclictest0-21swapper/108:25:141
761499129,2cyclictest8390-21sed10:30:280
761499129,2cyclictest6575-21ssh09:13:150
761499129,2cyclictest5476-21grep10:10:150
761499129,2cyclictest27809-21grep07:50:150
761499129,2cyclictest26547-21grep08:55:160
761499129,2cyclictest20492-21grep08:40:270
761499129,2cyclictest19905-21sed11:15:270
761499129,2cyclictest19028-21apt-key11:35:000
761499129,2cyclictest16986-21grep07:25:250
761499129,2cyclictest16194-21cat08:35:000
761499129,2cyclictest11898-21tr08:25:150
761499129,2cyclictest11285-21tr09:35:160
761499129,2cyclictest11285-21tr09:35:150
761499124,7cyclictest13947-21apache211:31:140
761499123,2cyclictest28254-21grep11:20:270
761499120,2cyclictest0-21swapper/012:04:420
169942120,0chrt0-21swapper/211:33:422
762499118,2cyclictest9105-21sh10:30:543
762499118,2cyclictest7410-21ssh09:51:393
762499118,2cyclictest5220-21ssh11:26:053
762499118,2cyclictest31643-21sh08:00:003
762499118,2cyclictest31509-21head09:05:173
762499118,2cyclictest28018-21cat07:50:163
762499118,2cyclictest25280-21grep07:45:163
762499118,2cyclictest25280-21grep07:45:163
762499118,2cyclictest1663-21fschecks_time09:10:173
7624991110,1cyclictest1370-21apache210:00:493
762499110,10cyclictest8538-21sed08:15:243
762499110,10cyclictest22463-21ssh11:36:153
761999118,2cyclictest9545-21sed07:10:272
761999118,2cyclictest9234-21ssh12:25:532
761999118,2cyclictest6472-21taskset08:13:462
761999118,2cyclictest31801-21ssh11:41:582
761999118,2cyclictest27345-21ssh12:36:522
761999118,2cyclictest22916-21ssh09:41:322
761999118,2cyclictest22854-21ssh09:22:382
761999118,2cyclictest20861-21cut10:00:162
761999118,2cyclictest20540-21head08:40:282
761999118,2cyclictest188032sleep207:30:232
761999118,2cyclictest16156-21cron08:34:592
761999118,2cyclictest13453-21chrt08:26:332
761999117,3cyclictest11881-21sed09:35:172
761999117,3cyclictest11881-21sed09:35:172
761999115,2cyclictest225-21systemd-journal09:10:162
761999115,2cyclictest225-21systemd-journal08:55:002
761999111,1cyclictest2309-21ssh09:49:132
761699119,1cyclictest23395-21ssh10:40:121
761699118,2cyclictest31808-21sed12:20:201
761699118,2cyclictest28429-21ssh11:40:151
761699118,2cyclictest27405-21grep07:50:131
761699118,2cyclictest26364-21expr08:55:141
761699118,2cyclictest2626-21ssh09:49:291
761699118,2cyclictest2249-21head11:05:221
761699118,2cyclictest21486-21rm11:35:291
761699118,2cyclictest17303-21sed10:55:191
761699118,2cyclictest14899-21cat08:30:191
761699118,2cyclictest10465-21ssh10:50:591
761699116,4cyclictest32721-21taskset09:08:461
761499119,2cyclictest0-21swapper/009:16:030
761499118,2cyclictest9547-21tr11:10:160
761499118,2cyclictest6947-21grep08:15:120
761499118,2cyclictest6318-21taskset08:11:480
761499118,2cyclictest5533-21ssh11:45:260
761499118,2cyclictest4695-21ssh09:30:290
761499118,2cyclictest32352-21ssh10:25:260
761499118,2cyclictest31752-21ssh10:45:110
761499118,2cyclictest30512-21ssh11:40:580
761499118,2cyclictest21077-21ssh10:57:300
761499118,2cyclictest16690-21tr08:35:150
761499118,2cyclictest13207-21cat07:20:150
761499118,2cyclictest12261-21apt-get09:55:120
761499118,2cyclictest10271-21ssh12:26:410
761499118,2cyclictest0-21swapper/007:40:130
761499117,2cyclictest28379-21grep09:00:130
762499108,1cyclictest9640-21taskset07:11:063
762499108,1cyclictest4854-21cat08:10:143
762499108,1cyclictest219682sleep307:38:103
762499108,1cyclictest15397-21readlink08:30:243
762499107,2cyclictest19235-21ls08:40:163
762499107,2cyclictest16618-21egrep08:35:143
762499107,2cyclictest11728-21wget08:25:153
762499107,2cyclictest10187-21chrt07:15:063
761999108,1cyclictest6841-21apache_accesses08:15:102
761999108,1cyclictest30805-21cut09:05:142
761999108,1cyclictest23035-21fschecks_count07:40:162
761999108,1cyclictest147912sleep207:22:262
761999107,2cyclictest20727-21grep07:35:172
761999103,6cyclictest3223-21ntp_kernel_err08:05:212
761699108,1cyclictest22422-21aten2_r7power_c07:40:131
761699108,1cyclictest21760-21sh10:20:001
761699108,1cyclictest15624-21expr07:25:131
761699108,1cyclictest15562sleep108:01:361
761699108,1cyclictest13222-21expr07:20:151
761699107,2cyclictest24886-21cut07:45:151
761699107,2cyclictest24886-21cut07:45:141
761699107,2cyclictest19312-21grep07:30:291
761699107,2cyclictest16703-21tr08:35:151
761699107,2cyclictest10191-21sed08:20:221
761699106,3cyclictest3771-21grep11:25:211
761699100,9cyclictest31279-21chrt07:55:281
761699100,9cyclictest20823-21hostname07:35:181
761499108,1cyclictest2939-21cat08:05:190
761499108,1cyclictest23574-21apache_volume08:50:130
761499107,2cyclictest8446-21grep07:10:170
761499107,2cyclictest25083-21cstates07:45:150
761499107,2cyclictest25083-21cstates07:45:150
76249997,1cyclictest244832sleep307:44:393
76199997,1cyclictest9469-21ls08:20:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional