You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-01 - 12:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Aug 01, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2763229381,8sleep20-21swapper/219:06:562
2770829078,8sleep00-21swapper/019:07:550
2779628568,13sleep10-21swapper/119:09:041
2767128572,9sleep30-21swapper/319:07:253
170062570,0sleep00-21swapper/000:00:330
2795699260,25cyclictest0-21swapper/222:44:292
2795699260,25cyclictest0-21swapper/222:44:292
27960992517,7cyclictest0-21swapper/323:59:553
2795499240,1cyclictest0-21swapper/121:43:351
27954992319,3cyclictest9492-21/usr/sbin/munin22:20:211
2796099221,20cyclictest1666-21ssh23:51:383
2795499220,22cyclictest0-21swapper/123:16:121
2795699210,21cyclictest0-21swapper/200:06:362
2795499210,0cyclictest0-21swapper/122:18:091
27948992019,1cyclictest0-21swapper/023:45:130
27948992018,1cyclictest0-21swapper/000:18:430
2796099190,15cyclictest0-21swapper/323:42:173
2795699194,2cyclictest20746-21diskmemload22:52:172
2794899190,4cyclictest0-21swapper/021:12:260
2795699182,2cyclictest141rcu_preempt22:39:482
2794899180,3cyclictest0-21swapper/021:27:560
2796099171,13cyclictest8448-21kworker/u8:221:31:003
2795499171,2cyclictest0-21swapper/123:22:321
2795499170,1cyclictest0-21swapper/100:04:031
27948991715,2cyclictest13-21ksoftirqd/021:05:260
2795699160,1cyclictest0-21swapper/221:50:362
2795699160,16cyclictest0-21swapper/221:43:242
27954991614,2cyclictest0-21swapper/121:46:031
2795499160,1cyclictest0-21swapper/123:53:351
2795499160,1cyclictest0-21swapper/121:22:571
2794899162,11cyclictest9869-21sensors22:00:250
27948991615,1cyclictest0-21swapper/023:19:080
27948991614,1cyclictest0-21swapper/022:58:210
2794899160,1cyclictest141rcu_preempt23:44:420
2795699152,10cyclictest16260-21apache_accesses21:00:132
27956991514,1cyclictest0-21swapper/221:27:572
27956991513,1cyclictest0-21swapper/200:18:052
2795699150,1cyclictest0-21swapper/219:55:122
2795699150,15cyclictest0-21swapper/221:56:512
27954991514,1cyclictest0-21swapper/121:08:021
27954991513,2cyclictest0-21swapper/123:13:061
2794899150,2cyclictest0-21swapper/021:46:020
243732150,0sleep10-21swapper/100:06:081
27960991411,2cyclictest39-21ksoftirqd/322:49:123
27960991410,3cyclictest10354-21tr19:40:143
27960991410,2cyclictest601-21grep19:20:153
2796099140,12cyclictest26984-21tr20:15:143
2795699141,8cyclictest0-21swapper/223:08:162
27956991414,0cyclictest0-21swapper/221:22:302
27956991413,1cyclictest0-21swapper/223:44:582
27956991413,1cyclictest0-21swapper/200:26:262
27956991412,1cyclictest0-21swapper/221:33:462
2795699140,2cyclictest30961-21ntp_states21:15:222
2795699140,14cyclictest0-21swapper/200:38:152
2795499141,12cyclictest0-21swapper/122:52:221
27954991410,3cyclictest11171-21grep22:40:151
27954991410,3cyclictest11171-21grep22:40:151
2795499140,1cyclictest20746-21diskmemload21:14:041
2795499140,14cyclictest0-21swapper/123:07:571
2795499140,14cyclictest0-21swapper/122:35:071
2795499140,14cyclictest0-21swapper/121:53:521
2794899141,10cyclictest19949-21ssh23:05:130
2794899140,12cyclictest0-21swapper/022:15:560
2796099139,3cyclictest8587-21grep22:00:163
2796099139,3cyclictest5438-21grep23:35:143
2796099139,3cyclictest2382-21sort20:30:163
2796099139,3cyclictest18671-21head22:25:163
2796099139,3cyclictest10399-21head22:20:163
2796099138,4cyclictest23666-21apt-get21:50:143
2796099135,5cyclictest0-21swapper/323:05:133
27960991311,1cyclictest20667-21wc20:00:203
27960991310,2cyclictest7037-21cut21:40:153
27960991310,2cyclictest4837-21sed20:35:193
27960991310,2cyclictest4375-21sh23:15:163
27960991310,2cyclictest23215-21grep21:10:283
27960991310,2cyclictest22476-21grep23:45:143
2795699137,3cyclictest13302-21munin-run23:40:002
2795699133,9cyclictest7543-21apache223:54:352
27956991312,1cyclictest0-21swapper/223:45:132
27956991310,2cyclictest6675-21turbostat.cron00:34:552
27956991310,2cyclictest23135-21sshd23:25:332
27956991310,2cyclictest22237-21grep20:05:172
27956991310,2cyclictest16047-21cron22:05:002
2795699130,8cyclictest0-21swapper/222:57:312
2795699130,1cyclictest0-21swapper/200:03:472
2795699130,13cyclictest0-21swapper/223:58:062
2795499139,3cyclictest4372-21cat22:55:191
2795499139,3cyclictest15112-21tr19:50:191
2795499132,9cyclictest15704-21/usr/sbin/munin21:25:221
2795499131,1cyclictest141rcu_preempt21:35:161
27954991311,2cyclictest25-21ksoftirqd/122:09:431
27954991310,2cyclictest32035-21grep00:30:161
27954991310,2cyclictest30878-21ssh23:30:231
27954991310,2cyclictest25677-21sh22:10:151
27954991310,2cyclictest17837-21grep19:55:151
27954991310,2cyclictest12400-21head23:00:171
2795499130,2cyclictest0-21swapper/122:02:161
2795499130,0cyclictest0-21swapper/123:41:201
2794899136,2cyclictest32224-21apt-get21:55:140
2794899132,7cyclictest537-21snmpd21:17:150
2794899131,7cyclictest0-21swapper/019:30:110
27948991311,1cyclictest0-21swapper/023:51:450
27948991310,2cyclictest25422-21cut22:10:140
2794899130,8cyclictest5046-21perf19:30:000
2794899130,2cyclictest0-21swapper/000:39:180
2794899130,12cyclictest29997-21ssh00:10:010
2794899130,10cyclictest0-21swapper/023:03:010
2796099129,2cyclictest6517-21tr00:15:143
2796099129,2cyclictest39-21ksoftirqd/319:35:123
2796099129,2cyclictest32233-21ntp_kernel_pll_21:35:213
2796099129,2cyclictest3153-21tr19:25:163
2796099129,2cyclictest31451-21wc21:15:273
2796099129,2cyclictest29848-21grep19:10:253
2796099129,2cyclictest29710-21tr23:30:143
2796099129,2cyclictest27904-21ssh22:50:163
2796099129,2cyclictest21640-21rm23:25:163
2796099129,2cyclictest1766-21grep21:55:273
2796099129,2cyclictest16812-21grep21:00:153
2796099129,2cyclictest15356-21cut21:45:133
2796099129,2cyclictest14540-21tr21:25:143
2796099129,2cyclictest12322-21cut23:00:173
2796099128,3cyclictest11618-21wc22:40:183
2796099128,3cyclictest11618-21wc22:40:183
2796099128,2cyclictest6709-21tr20:40:143
2796099128,2cyclictest1656-21sed22:15:163
27960991212,0cyclictest0-21swapper/300:30:343
2796099121,11cyclictest0-21swapper/322:34:403
27960991211,1cyclictest0-21swapper/322:06:133
2795699129,2cyclictest8448-21head19:35:172
2795699129,2cyclictest5018-21tr20:35:212
2795699129,2cyclictest3639-21grep19:25:192
2795699129,2cyclictest31250-21sed23:30:272
2795699129,2cyclictest1739-21ls19:20:222
2795699129,2cyclictest17113-21grep22:05:142
2795699129,2cyclictest16481-21ls00:20:292
2795699122,9cyclictest30208-21apt-get00:10:092
27956991210,1cyclictest27051-21grep22:30:162
2795699120,5cyclictest0-21swapper/222:27:432
2795699120,11cyclictest26001-21grep22:10:162
2795699120,11cyclictest0-21swapper/223:00:132
2795699120,10cyclictest0-21swapper/220:20:092
2795499129,2cyclictest8140-21grep20:40:261
2795499129,2cyclictest7466-21cat19:35:001
2795499129,2cyclictest359-21grep21:55:161
2795499129,2cyclictest32301-21grep20:25:191
2795499129,2cyclictest27370-21cat22:30:181
2795499129,2cyclictest15964-21cron21:00:001
2795499129,2cyclictest13420-21uname19:45:191
2795499127,4cyclictest30517-21grep21:15:181
2795499122,9cyclictest12155-21munin-node23:45:541
27954991210,1cyclictest193222sleep100:22:341
2795499120,0cyclictest0-21swapper/123:35:291
2794899129,2cyclictest8474-21wc21:40:230
2794899129,2cyclictest3878-21wc22:35:240
2794899129,2cyclictest28647-21tr23:10:150
2794899129,2cyclictest10381-21grep22:20:160
2794899128,3cyclictest0-21swapper/021:36:360
2794899128,2cyclictest22598-21ssh21:30:130
27948991210,1cyclictest2674-21cron22:55:000
27948991210,1cyclictest22446-21cron00:25:000
27948991210,1cyclictest21730-21cut23:25:160
2796099118,2cyclictest8485-21cat00:35:193
2796099118,2cyclictest6037-21tail19:30:153
2796099118,2cyclictest5710-21cron00:15:003
2796099118,2cyclictest5503-21ssh21:20:103
2796099118,2cyclictest3494-21tr22:55:143
2796099118,2cyclictest336-21wc19:20:003
2796099118,2cyclictest3310-21grep22:35:193
2796099118,2cyclictest30306-21grep23:10:283
2796099118,2cyclictest25143-21grep20:10:243
2796099118,2cyclictest25077-21ssh22:10:103
2796099118,2cyclictest24797-21wc00:25:283
2796099118,2cyclictest15494-21sed00:20:203
2796099118,2cyclictest14351-21taskset19:46:563
2796099118,2cyclictest13593-21cron20:55:003
2796099118,2cyclictest12377-21ssh23:20:033
2796099117,3cyclictest20746-21diskmemload00:05:023
2796099111,7cyclictest16305-21ps00:00:223
2796099110,10cyclictest20419-21taskset21:05:583
2795699119,1cyclictest5058-21grep23:15:182
2795699119,1cyclictest32228-21grep19:15:262
2795699119,1cyclictest27408-21grep20:15:162
2795699118,2cyclictest8662-21basename20:45:002
2795699118,2cyclictest6034-21grep19:30:152
2795699118,2cyclictest30942-21ssh21:35:122
2795699118,2cyclictest24650-21grep20:10:212
2795699118,2cyclictest23138-21ssh22:47:112
2795699118,2cyclictest22085-21head21:10:172
2795699118,2cyclictest17389-21sh22:25:002
2795699118,2cyclictest1675-21ssh23:13:252
2795699118,2cyclictest16145-21sh23:21:382
2795699118,2cyclictest16144-21sed21:45:172
2795699118,2cyclictest14539-21grep20:55:162
2795699115,2cyclictest1564-21expr22:15:162
2795499119,1cyclictest12252-21sed20:50:171
2795499119,1cyclictest0-21swapper/120:15:161
2795499118,2cyclictest9716-21grep20:45:171
2795499118,2cyclictest9692-21ssh23:56:221
2795499118,2cyclictest7829-21df00:35:141
2795499118,2cyclictest7660-21ssh00:15:231
2795499118,2cyclictest4226-21gzip19:25:231
2795499118,2cyclictest30351-21cat19:15:001
2795499118,2cyclictest30217-21taskset00:10:091
2795499118,2cyclictest2383-21uniq20:30:161
2795499118,2cyclictest22959-21ssh00:25:131
2795499118,2cyclictest22870-21tr21:30:141
2795499118,2cyclictest22795-21ls23:25:251
2795499118,2cyclictest17929-21grep22:25:121
2795499117,3cyclictest25-21ksoftirqd/119:39:591
2795499110,10cyclictest5280-21cut20:35:221
2794899119,1cyclictest6787-21wc23:35:220
2794899119,1cyclictest4775-21grep20:35:170
2794899119,1cyclictest2793-21tail20:30:190
2794899118,2cyclictest8278-21sort23:55:270
2794899118,2cyclictest6817-21sleep00:35:000
2794899118,2cyclictest5870-21tr21:20:140
2794899118,2cyclictest31209-21chrt20:24:190
2794899118,2cyclictest28255-21grep19:10:130
2794899118,2cyclictest26719-21expr22:30:140
2794899118,2cyclictest26361-21ssh00:26:320
2794899118,2cyclictest23417-21ssh22:47:260
2794899118,2cyclictest21450-21chrt20:01:190
2794899118,2cyclictest20569-21wc23:25:010
2794899118,2cyclictest18833-21if_enp2s022:25:170
2794899118,2cyclictest13752-21wc19:45:220
2794899118,2cyclictest12724-21grep20:50:220
2794899118,2cyclictest1248-21basename20:30:000
2794899118,2cyclictest11312-21expr22:40:160
2794899118,2cyclictest11312-21expr22:40:160
2794899110,1cyclictest0-21swapper/022:08:090
2794899110,10cyclictest13970-21grep20:55:140
2796099108,1cyclictest32509-21cat20:25:203
2796099108,1cyclictest29165-21apt-get20:20:093
2796099108,1cyclictest19143-21taskset19:57:283
2796099108,1cyclictest148222sleep319:50:043
2796099107,2cyclictest39-21ksoftirqd/320:45:153
2796099107,2cyclictest23732-21awk20:05:303
2796099107,2cyclictest15084-21head20:55:223
2795699108,2cyclictest13300-21grep19:45:182
2795699108,1cyclictest133622sleep220:52:052
2795699108,1cyclictest11416-21cat20:50:002
2795699107,2cyclictest20963-21cat21:10:002
2795699107,2cyclictest10460-21expr19:40:142
2795699100,9cyclictest1593-21dpkg20:30:002
2795699100,3cyclictest225-21systemd-journal19:55:002
2795499108,1cyclictest30979-21expr19:15:151
2795499108,1cyclictest20119-21df20:00:151
2795499108,1cyclictest19499-21fschecks_count22:45:151
2795499108,1cyclictest16519-21cut21:00:151
2795499108,1cyclictest10072-21seq19:40:041
2795499107,2cyclictest30280-21ntpq20:20:161
2795499107,2cyclictest22119-21grep20:05:161
2795499107,2cyclictest1634-21grep19:20:211
2795499106,3cyclictest25119-21head20:10:231
2794899108,1cyclictest9827-21sh19:39:590
2794899108,1cyclictest286432sleep020:17:050
2794899108,1cyclictest17975-21grep19:55:160
2794899107,2cyclictest6527-21cut20:40:130
2794899107,2cyclictest30027-21ssh23:30:160
2794899107,2cyclictest24537-21cut21:50:190
2794899107,2cyclictest23034-21ssh00:05:060
2794899107,2cyclictest22618-21cut20:05:210
2794899107,2cyclictest16490-21grep19:50:310
2794899101,5cyclictest0-21swapper/020:10:180
2794899100,9cyclictest162722sleep021:00:130
279569998,1cyclictest29308-21ntpq19:10:192
279569998,1cyclictest19930-21ls20:00:152
279569997,1cyclictest31790-21cut20:25:152
279489997,1cyclictest30663-21hostname19:15:140
279489997,1cyclictest1351-21sed19:20:180
279489997,1cyclictest112242sleep019:40:200
279489996,2cyclictest9893-21grep20:45:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional