You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-30 - 08:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon Mar 30, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
333828846,38sleep20-21swapper/219:09:122
323428573,8sleep00-21swapper/019:07:480
126728471,9sleep30-21swapper/319:05:073
305227959,16sleep10-21swapper/119:05:291
109962610,0sleep20-21swapper/223:52:372
349699251,23cyclictest0-21swapper/321:39:233
3490992322,1cyclictest0-21swapper/100:25:131
139972230,0sleep00-21swapper/022:17:420
349699220,21cyclictest0-21swapper/319:45:143
277282220,0sleep00-21swapper/021:47:040
3496992020,0cyclictest0-21swapper/322:31:353
349699201,18cyclictest0-21swapper/322:47:003
349699200,4cyclictest0-21swapper/321:58:483
349399200,1cyclictest0-21swapper/200:35:172
349399200,19cyclictest29170-21diskmemload21:48:482
349099200,3cyclictest0-21swapper/100:18:461
3493991918,0cyclictest0-21swapper/222:55:372
349099193,7cyclictest0-21swapper/100:05:141
3490991911,2cyclictest15919-21rm22:00:011
349099190,19cyclictest0-21swapper/122:33:231
3496991817,1cyclictest0-21swapper/322:07:343
3496991817,1cyclictest0-21swapper/321:43:013
349099182,6cyclictest0-21swapper/120:25:121
3490991817,1cyclictest21733-21kworker/1:000:03:561
3490991814,3cyclictest20910-21ssh22:40:431
3490991814,3cyclictest0-21swapper/123:10:131
30982180,0sleep00-21swapper/021:32:260
349099174,2cyclictest0-21swapper/120:15:141
349099170,1cyclictest0-21swapper/122:49:431
52962160,0sleep00-21swapper/000:27:520
3496991613,2cyclictest0-21swapper/322:54:363
349699161,15cyclictest0-21swapper/300:27:373
349399163,8cyclictest0-21swapper/222:40:132
349399160,16cyclictest0-21swapper/222:17:532
3490991616,0cyclictest0-21swapper/122:08:081
3490991615,1cyclictest0-21swapper/121:19:551
3490991614,1cyclictest0-21swapper/123:42:041
3490991614,1cyclictest0-21swapper/122:02:271
349099160,16cyclictest0-21swapper/100:23:141
348699162,8cyclictest2689-21idleruntime-cro20:15:000
348699160,9cyclictest0-21swapper/021:20:190
349699150,14cyclictest0-21swapper/300:20:563
349699150,0cyclictest0-21swapper/323:02:343
349099153,8cyclictest16039-21rt-features23:55:221
3490991513,1cyclictest5232-21ssh23:30:131
349099151,1cyclictest32728-21ssh21:11:151
3490991511,3cyclictest38150irq/131-snd_hda21:20:181
349099150,15cyclictest0-21swapper/123:16:401
349099150,15cyclictest0-21swapper/123:16:401
348699156,8cyclictest15356-21apt-get19:35:000
349699142,11cyclictest24042-21ntp_states21:25:253
349699142,11cyclictest24042-21ntp_states21:25:253
3496991410,2cyclictest12971-21cron19:29:593
349399144,8cyclictest0-21swapper/220:55:132
3493991411,2cyclictest28814-21cat23:25:012
3493991411,2cyclictest18957-21tail19:40:192
349399140,0cyclictest0-21swapper/222:01:132
349099146,2cyclictest225-21systemd-journal22:59:591
3490991411,2cyclictest3066-21tr22:50:131
3490991410,2cyclictest25488-21cat22:25:001
349099140,14cyclictest0-21swapper/123:09:591
349099140,12cyclictest0-21swapper/100:30:361
348699144,5cyclictest0-21swapper/021:10:000
3486991411,2cyclictest15333-21cron00:15:000
349699139,3cyclictest23534-21grep19:50:143
3496991311,2cyclictest39-21ksoftirqd/323:52:543
349699131,10cyclictest14265-21ssh21:20:133
3496991310,2cyclictest6807-21grep19:15:173
3496991310,2cyclictest30128-21expr21:10:153
3496991310,2cyclictest29161-21cron21:09:593
3496991310,2cyclictest25328-21cron22:25:003
3496991310,2cyclictest17228-21turbostat.cron20:44:593
3496991310,2cyclictest10149-21grep19:20:253
349699130,9cyclictest0-21swapper/300:30:133
349699130,13cyclictest0-21swapper/322:13:063
349699130,12cyclictest6790-21ls21:15:193
349699130,12cyclictest29960-21grep23:25:163
349399139,3cyclictest29587-21sh23:25:142
349399139,3cyclictest26326-21sed21:00:272
3493991311,1cyclictest0-21swapper/220:00:222
3493991310,2cyclictest8892-21tr20:25:192
3493991310,2cyclictest32631-21ntpq00:05:222
3493991310,2cyclictest2671-21ls20:15:002
3493991310,2cyclictest2607-21cut22:30:162
3493991310,2cyclictest13730-21tr20:35:212
3493991310,2cyclictest13269-21grep23:15:172
3493991310,2cyclictest13269-21grep23:15:172
349399130,13cyclictest0-21swapper/221:57:172
349099139,3cyclictest7445-21uname23:50:191
349099139,3cyclictest7201-21grep19:15:191
349099139,2cyclictest13314-21cut19:30:121
349099133,5cyclictest19948-21apache219:35:121
349099133,5cyclictest19948-21apache219:35:121
3490991313,0cyclictest0-21swapper/121:27:321
3490991313,0cyclictest0-21swapper/121:27:311
3490991312,1cyclictest0-21swapper/123:35:011
3490991310,2cyclictest7544-21sort21:35:151
3490991310,2cyclictest32156-21sed23:45:261
3490991310,2cyclictest17218-21sh20:45:001
349099130,1cyclictest0-21swapper/100:13:591
349099130,13cyclictest0-21swapper/122:25:461
349099130,13cyclictest0-21swapper/121:33:151
349099130,12cyclictest23982-21cut21:45:151
349099130,12cyclictest19339-21uname19:40:221
348699139,3cyclictest8662-21wc21:35:250
348699139,3cyclictest29810-21basename23:45:010
348699137,2cyclictest225-21systemd-journal23:17:040
348699137,2cyclictest225-21systemd-journal23:17:030
3486991310,2cyclictest5274-21awk23:10:190
3486991310,2cyclictest24986-21egrep22:05:130
3486991310,2cyclictest24463-21sed19:50:210
3486991310,2cyclictest19098-21sed20:45:240
3486991310,2cyclictest12415-21wc19:25:270
348699130,1cyclictest0-21swapper/022:30:140
349699129,2cyclictest4589-21ssh21:52:503
349699129,2cyclictest3977-21ssh23:10:113
349699129,2cyclictest32250-21ntpq21:30:203
349699129,2cyclictest29271-21taskset20:00:233
349699129,2cyclictest25910-21grep21:00:233
349699129,2cyclictest25842-21tr19:55:133
349699129,2cyclictest23826-21tail00:00:153
349699129,2cyclictest16955-21tr22:20:003
349699129,2cyclictest15444-21grep23:35:253
349699129,2cyclictest15372-21cat00:14:593
349699129,2cyclictest14000-21sed19:30:153
349699128,3cyclictest20408-21grep22:40:253
349699126,4cyclictest16969-21ssh22:00:143
349699122,1cyclictest225-21systemd-journal22:40:003
349699120,2cyclictest0-21swapper/323:15:173
349699120,2cyclictest0-21swapper/323:15:163
349399129,2cyclictest7998-21cron00:30:002
349399129,2cyclictest5928-21grep21:15:152
349399129,2cyclictest5393-21tail19:10:262
349399129,2cyclictest4043-21grep20:15:212
349399129,2cyclictest22164-21basename21:25:002
349399129,2cyclictest21010-21grep23:00:202
349399129,2cyclictest18513-21cat20:45:192
349399129,2cyclictest17943-21idleruntime-cro19:40:002
349399129,2cyclictest17943-21idleruntime-cro19:40:002
349399129,2cyclictest16941-21grep21:40:252
349399129,2cyclictest11832-21cat19:25:192
349399129,2cyclictest10212-21grep19:20:252
349399123,6cyclictest15371-21awk19:35:002
349399122,5cyclictest0-21swapper/223:55:142
3493991210,1cyclictest27156-21sh22:25:212
349399120,10cyclictest0-21swapper/200:20:112
349099129,2cyclictest30121-21sed23:25:161
349099129,2cyclictest23498-21grep19:50:141
349099129,2cyclictest21752-21cut19:45:191
349099129,2cyclictest18603-21wc00:35:261
349099129,2cyclictest18161-21cut20:45:161
349099129,2cyclictest16461-21grep21:40:191
349099120,3cyclictest0-21swapper/121:52:311
348699129,2cyclictest8575-21cut21:55:170
348699129,2cyclictest30090-21grep23:25:150
348699129,2cyclictest28717-21cut23:05:160
348699129,2cyclictest27199-21tr22:45:160
348699129,2cyclictest21487-21wc20:50:250
348699129,2cyclictest16314-21wc21:40:170
348699129,2cyclictest11099-21ssh22:55:090
348699129,2cyclictest10334-21chrt21:17:230
348699128,3cyclictest6472-21tr23:50:130
348699128,3cyclictest19171-21grep22:20:250
348699128,2cyclictest10756-21wc22:55:000
348699127,3cyclictest26105-21wc21:00:260
348699127,3cyclictest14015-21ls20:35:250
348699124,3cyclictest7001-21/usr/sbin/munin00:10:000
348699122,9cyclictest1-21systemd00:17:570
348699122,5cyclictest28805-21perf23:25:000
348699120,11cyclictest1533-21grep22:10:170
349699118,2cyclictest8929-21ssh23:31:493
349699118,2cyclictest6469-21grep20:20:203
349699118,2cyclictest4112-21expr19:10:153
349699118,2cyclictest27711-21ssh23:05:053
349699118,2cyclictest27341-21sh22:25:233
349699118,2cyclictest26993-21ssh21:46:293
349699118,2cyclictest25351-21ssh23:22:193
349699118,2cyclictest22876-21ls23:40:183
349699118,2cyclictest19107-21ssh00:35:373
349699118,2cyclictest18888-21grep19:40:183
349699118,2cyclictest18151-21wc19:40:013
349699118,2cyclictest18151-21wc19:40:003
349699118,2cyclictest18084-21grep20:45:163
349699118,2cyclictest15633-21ssh00:15:043
349699118,2cyclictest13719-21ls20:35:213
349699118,2cyclictest11292-21egrep22:55:123
349699118,2cyclictest10165-21grep20:30:153
349699117,3cyclictest31674-21grep00:05:163
349699111,6cyclictest22296-21apt-get20:55:133
349699110,10cyclictest22430-21cron00:00:003
349399119,1cyclictest22810-21head23:40:172
349399118,2cyclictest7348-21expr21:35:132
349399118,2cyclictest6877-21sed19:15:162
349399118,2cyclictest6057-21sed23:30:172
349399118,2cyclictest3855-21ssh21:33:012
349399118,2cyclictest338-21sed21:50:172
349399118,2cyclictest32394-21rm22:09:482
349399118,2cyclictest32180-21sed23:45:262
349399118,2cyclictest31892-21ssh23:07:002
349399118,2cyclictest3053-21tr22:50:132
349399118,2cyclictest27326-21grep19:55:242
349399118,2cyclictest26399-21sh21:26:492
349399118,2cyclictest26399-21sh21:26:482
349399118,2cyclictest21908-21ssh22:22:182
349399118,2cyclictest21396-21expr19:45:162
349399118,2cyclictest14388-21chrt00:33:282
349399118,2cyclictest10183-21hostname22:35:142
349399117,3cyclictest16764-21grep20:40:252
349399116,4cyclictest31903-21sh21:10:372
349399116,4cyclictest28961-21taskset00:03:332
349399116,2cyclictest3742-21ssh22:11:112
349399110,1cyclictest0-21swapper/200:15:082
349399110,10cyclictest13842-21cut23:35:132
349399110,10cyclictest10892-21ssh00:11:312
349099119,1cyclictest11605-21grep19:25:181
349099118,2cyclictest5417-21grep20:20:131
349099118,2cyclictest32507-21apt-get20:10:011
349099118,2cyclictest29233-21sed20:00:231
349099118,2cyclictest29086-21chrt21:09:061
349099118,2cyclictest22030-21turbostat.cron20:55:001
349099118,2cyclictest21791-21ssh23:00:371
349099118,2cyclictest20812-21ssh23:20:081
349099118,2cyclictest1929-21tr20:10:231
349099118,2cyclictest11887-21ssh22:16:031
349099118,2cyclictest11858-21wc22:35:251
349099118,2cyclictest10503-21taskset19:23:471
349099117,3cyclictest25399-21grep21:00:181
348699118,2cyclictest7890-21chrt19:16:020
348699118,2cyclictest5882-21cut23:30:160
348699118,2cyclictest5537-21taskset19:11:210
348699118,2cyclictest29991-21cut21:10:140
348699118,2cyclictest27576-21ssh00:02:280
348699118,2cyclictest27540-21ssh00:21:300
348699118,2cyclictest27115-21nfs4_client22:25:210
348699118,2cyclictest23117-21expr21:25:190
348699118,2cyclictest23117-21expr21:25:190
348699118,2cyclictest20909-21tr19:45:150
348699118,2cyclictest20903-21sh00:37:020
348699118,2cyclictest19472-21dpkg23:00:110
348699118,2cyclictest18481-21ssh22:40:130
348699118,2cyclictest1775-21ssh21:50:380
348699118,2cyclictest15588-21ssh23:55:190
348699118,2cyclictest14331-21tail23:35:170
348699118,2cyclictest12193-21ssh00:31:440
348699118,2cyclictest11824-21wc22:35:250
348699118,2cyclictest1115-21ssh23:46:250
349699108,1cyclictest7755-21apache_volume20:25:123
349699108,1cyclictest20315-21expr20:50:153
349699107,2cyclictest3193-21tr20:15:153
349699107,2cyclictest30035-21sh23:45:023
349699100,9cyclictest322362sleep320:06:453
349399108,1cyclictest3568-21cat23:10:022
349399108,1cyclictest26752-21ssh22:45:082
349399108,1cyclictest24740-21cat19:50:252
349399108,1cyclictest10334-21cat20:30:152
349399107,2cyclictest6163-21grep20:20:172
349399107,2cyclictest27188-21cut21:05:142
349399107,2cyclictest22017-21cron20:55:002
349399100,9cyclictest30374-21cat20:05:092
349099108,1cyclictest989-21grep22:10:131
349099108,1cyclictest5818-21chrt19:14:591
349099108,1cyclictest12287-21cat20:34:591
349099107,2cyclictest26048-21cat19:55:121
349099107,2cyclictest22979-21cut20:55:151
349099107,2cyclictest13127-21cut20:35:161
348699108,1cyclictest77172sleep020:25:060
348699108,1cyclictest6256-21cat20:20:170
348699108,1cyclictest17684-21ssh22:00:200
348699108,1cyclictest169042sleep020:41:050
348699107,2cyclictest4146-21cut20:15:230
348699107,2cyclictest30668-21cut20:05:130
348699107,2cyclictest26322-21grep19:55:150
348699107,2cyclictest18301-21cut19:40:160
348699100,9cyclictest29937-21chrt20:02:420
34969997,1cyclictest1692-21ntpq20:10:213
34869997,1cyclictest23817-21cat20:55:240
34869997,1cyclictest17390-21sed19:35:250
34869997,1cyclictest17390-21sed19:35:250
34869997,1cyclictest113682sleep020:30:230
34869996,2cyclictest9241-21tr19:20:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional