You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-21 - 05:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue Jul 21, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4627210087,9sleep00-21swapper/019:07:500
355929043,42sleep30-21swapper/319:05:213
452928676,5sleep10-21swapper/119:06:341
466728471,9sleep20-21swapper/219:08:212
488699270,24cyclictest0-21swapper/223:43:272
4886992625,1cyclictest0-21swapper/222:27:432
4883992524,1cyclictest0-21swapper/121:19:171
488899242,20cyclictest31288-21kworker/3:223:27:423
488899241,22cyclictest0-21swapper/319:35:133
488899232,19cyclictest31288-21kworker/3:223:23:073
488699230,22cyclictest0-21swapper/200:22:222
488399230,23cyclictest0-21swapper/122:34:271
488399230,1cyclictest0-21swapper/100:16:101
488199231,5cyclictest12661-21df_inode22:35:160
488899221,19cyclictest9027-21kworker/3:000:39:153
488699223,11cyclictest31954-21apt-get21:30:132
488699210,17cyclictest0-21swapper/200:35:082
488899200,18cyclictest0-21swapper/321:43:323
488399200,1cyclictest0-21swapper/120:15:151
488199206,6cyclictest9617-21munin-run19:19:590
488699190,1cyclictest0-21swapper/223:47:122
488699190,1cyclictest0-21swapper/223:02:042
488699190,19cyclictest0-21swapper/222:55:002
488399190,19cyclictest0-21swapper/122:27:401
488199190,17cyclictest0-21swapper/021:22:300
4886991816,1cyclictest0-21swapper/221:54:302
488699180,18cyclictest0-21swapper/223:26:572
4883991814,3cyclictest38150irq/131-snd_hda23:05:281
488399180,18cyclictest0-21swapper/121:40:551
488399180,16cyclictest0-21swapper/121:28:091
488199186,7cyclictest401-21ssh23:45:140
488199180,2cyclictest0-21swapper/023:23:360
488199180,1cyclictest0-21swapper/023:16:410
4888991714,2cyclictest0-21swapper/300:16:483
488699173,10cyclictest9770-21forks23:50:182
488699170,1cyclictest141rcu_preempt21:19:482
488699170,17cyclictest0-21swapper/200:09:172
488699170,15cyclictest0-21swapper/222:05:102
488399175,10cyclictest0-21swapper/122:20:141
488399172,1cyclictest30102-21diskmemload22:42:171
4888991612,3cyclictest7960-21sshd21:35:143
4888991610,5cyclictest19699-21apache223:12:283
488199160,1cyclictest0-21swapper/000:22:210
4888991510,4cyclictest12838-21apache200:20:413
488699159,5cyclictest32-21ksoftirqd/220:50:132
488699153,11cyclictest30102-21diskmemload21:56:162
4886991513,1cyclictest0-21swapper/223:59:092
488699151,13cyclictest0-21swapper/221:13:502
488699151,13cyclictest0-21swapper/219:39:332
4886991511,2cyclictest0-21swapper/222:11:092
488699150,1cyclictest0-21swapper/223:19:232
488699150,15cyclictest0-21swapper/223:32:512
488699150,14cyclictest0-21swapper/221:35:102
488399158,5cyclictest26493-21cut22:05:141
488399153,9cyclictest30102-21diskmemload22:49:511
4883991514,1cyclictest0-21swapper/123:30:131
488199157,7cyclictest31463-21/usr/sbin/munin23:25:130
488199150,1cyclictest0-21swapper/021:18:190
48742150,4chrt39-21ksoftirqd/300:06:093
488899149,3cyclictest15639-21head19:30:183
488899147,5cyclictest14031-21munin-node22:10:143
488899144,8cyclictest11999-21apt-get19:25:003
4888991410,3cyclictest5628-21sed20:15:273
4888991410,3cyclictest14001-21grep22:55:173
488899140,11cyclictest26001-21sensors23:40:243
488699141,9cyclictest0-21swapper/220:30:122
4886991413,1cyclictest0-21swapper/222:19:132
488699140,14cyclictest0-21swapper/223:09:162
488699140,14cyclictest0-21swapper/222:00:362
488699140,14cyclictest0-21swapper/200:27:292
488699140,13cyclictest0-21swapper/223:13:302
488399149,2cyclictest225-21systemd-journal20:10:011
488399142,1cyclictest0-21swapper/100:35:141
4883991411,2cyclictest10622-21tr21:55:231
4883991410,3cyclictest29939-21grep20:00:171
4883991410,3cyclictest26279-21ssh21:45:221
4883991410,3cyclictest26279-21ssh21:45:221
4883991410,2cyclictest27599-21cut00:20:161
488399140,13cyclictest0-21swapper/122:04:121
488399140,0cyclictest0-21swapper/123:02:091
488199146,1cyclictest0-21swapper/019:15:000
488199143,3cyclictest0-21swapper/020:35:160
4881991413,1cyclictest0-21swapper/022:41:380
488199141,2cyclictest0-21swapper/021:10:480
4881991412,2cyclictest0-21swapper/000:18:290
488199140,13cyclictest17242-21ssh22:56:530
488199140,13cyclictest0-21swapper/022:20:140
488899139,3cyclictest4824-21sh00:25:203
488899139,3cyclictest2355-21grep23:45:283
488899139,3cyclictest12103-21ls22:15:223
488899139,2cyclictest455-21dbus-daemon00:10:413
488899132,4cyclictest0-21swapper/322:26:183
488899132,10cyclictest0-21swapper/321:17:433
4888991310,2cyclictest3625-21cron20:15:013
4888991310,2cyclictest32589-21head21:10:273
4888991310,2cyclictest30102-21diskmemload23:07:203
4888991310,2cyclictest29458-21grep20:00:153
4888991310,2cyclictest15294-21sed20:35:283
488899130,1cyclictest0-21swapper/322:03:253
488699132,10cyclictest0-21swapper/200:32:312
4886991312,1cyclictest0-21swapper/200:00:232
488699131,11cyclictest16766-21tail20:40:162
4886991311,1cyclictest0-21swapper/222:32:272
488699130,1cyclictest0-21swapper/223:21:272
488699130,13cyclictest0-21swapper/223:38:412
488699130,13cyclictest0-21swapper/223:38:412
488699130,12cyclictest0-21swapper/222:40:262
488699130,12cyclictest0-21swapper/221:26:312
488399139,4cyclictest0-21swapper/123:56:591
488399139,3cyclictest7520-21ls23:10:221
488399139,3cyclictest17743-21grep20:40:261
488399139,2cyclictest23496-21grep20:55:151
488399131,2cyclictest25035-21sed23:20:281
4883991310,2cyclictest4985-21cut22:50:131
4883991310,2cyclictest22565-21cron23:19:591
4883991310,2cyclictest19504-21wc19:40:011
488399130,13cyclictest0-21swapper/123:26:551
488199135,2cyclictest7427-21perf21:34:590
488199131,7cyclictest30960-21/usr/sbin/munin23:05:180
4881991311,1cyclictest17082-21uniq21:40:190
4881991310,2cyclictest9009-21ssh00:09:180
4881991310,2cyclictest3171-21grep20:10:250
4881991310,2cyclictest27041-21cut19:55:150
4881991310,2cyclictest22444-21grep00:35:270
488199130,3cyclictest0-21swapper/020:55:140
488899129,2cyclictest25050-21grep19:50:183
488899129,2cyclictest25050-21grep19:50:183
488899129,2cyclictest15446-21sed23:15:183
488899128,3cyclictest27711-21cron21:05:003
488899125,6cyclictest0-21swapper/321:48:383
488899125,6cyclictest0-21swapper/321:48:373
4888991210,1cyclictest25980-21ssh00:00:133
488899120,1cyclictest0-21swapper/321:28:143
488699129,2cyclictest9304-21sort20:25:172
488699129,2cyclictest4131-21tr20:15:152
488699129,2cyclictest20821-21tr19:40:212
488699129,2cyclictest12078-21cat19:25:002
488699129,2cyclictest11274-21tail00:10:182
488699128,3cyclictest3593-21taskset22:49:232
488699125,3cyclictest225-21systemd-journal20:05:012
4886991211,1cyclictest0-21swapper/222:59:532
4886991211,1cyclictest0-21swapper/222:21:332
4886991210,1cyclictest7561-21wc20:20:222
4886991210,1cyclictest2557-21wc20:10:192
4886991210,1cyclictest16217-21wc21:20:212
488699120,9cyclictest0-21swapper/219:25:002
488699120,11cyclictest26343-21grep21:00:172
488399129,2cyclictest23161-21cut19:45:201
488399129,2cyclictest10278-21rm00:10:061
488399127,4cyclictest14720-21ssh21:39:101
488399127,4cyclictest10121-21ssh00:10:001
488399127,2cyclictest2637-21sh21:12:191
488399124,4cyclictest225-21systemd-journal21:05:001
488399121,10cyclictest30102-21diskmemload22:57:491
488399120,5cyclictest0-21swapper/122:10:151
488399120,2cyclictest0-21swapper/122:19:221
488399120,12cyclictest0-21swapper/100:25:141
488199129,2cyclictest8825-21cut23:30:210
488199129,2cyclictest27716-21cut22:25:140
488199129,2cyclictest26527-21cron19:55:010
488199129,2cyclictest26527-21cron19:55:000
488199129,2cyclictest26188-21cut00:00:140
488199129,2cyclictest23870-21apt23:40:010
488199129,2cyclictest23870-21apt23:40:000
488199129,2cyclictest16961-21sh23:55:010
488199129,2cyclictest13502-21ssh00:30:220
488199129,2cyclictest11488-21cron22:35:000
488199128,2cyclictest19166-21tail20:45:160
488199126,5cyclictest30102-21diskmemload00:12:570
488199126,1cyclictest2356-21apt-get22:10:150
4881991210,1cyclictest6463-21tr23:10:150
488899119,1cyclictest16292-21grep21:20:213
488899118,2cyclictest9620-21cat19:20:003
488899118,2cyclictest9385-21grep23:50:163
488899118,2cyclictest7433-21ssh23:30:123
488899118,2cyclictest3457-21sh21:50:593
488899118,2cyclictest32229-21grep21:30:153
488899118,2cyclictest30896-21ssh22:45:263
488899118,2cyclictest30878-21ssh22:07:243
488899118,2cyclictest25341-21ssh23:01:343
488899118,2cyclictest22765-21expr19:45:173
488899118,2cyclictest20552-21cat19:40:193
488899118,2cyclictest20175-21taskset22:20:183
488899118,2cyclictest17564-21ssh23:55:123
488899118,2cyclictest16373-21cat23:35:153
488899118,2cyclictest16373-21cat23:35:143
488899118,2cyclictest14388-21taskset19:27:353
488899118,2cyclictest14145-21ssh21:57:423
488899118,2cyclictest12966-21head00:30:173
488899117,3cyclictest3887-21grep22:30:153
488899115,2cyclictest225-21systemd-journal22:45:003
488899113,4cyclictest225-21systemd-journal22:55:003
488899110,1cyclictest0-21swapper/322:35:123
488699119,1cyclictest8599-21grep19:15:202
488699119,1cyclictest28904-21cron20:00:002
488699119,1cyclictest22339-21tr19:45:152
488699118,2cyclictest31312-21taskset21:48:562
488699118,2cyclictest31312-21taskset21:48:562
488699118,2cyclictest24322-21grep20:55:202
488699118,2cyclictest20729-21cut20:50:002
488699116,3cyclictest30102-21diskmemload00:15:082
488699115,3cyclictest12631-21df_abs22:35:152
488699114,6cyclictest0-21swapper/219:30:132
488699110,4cyclictest225-21systemd-journal20:36:122
488699110,10cyclictest24415-21basename19:50:132
488699110,10cyclictest24415-21basename19:50:122
488399118,2cyclictest9880-21cut20:25:221
488399118,2cyclictest9282-21chrt19:16:041
488399118,2cyclictest896-21expr23:45:161
488399118,2cyclictest6796-21sed19:10:261
488399118,2cyclictest29687-21grep21:05:271
488399118,2cyclictest28388-21ssh00:00:481
488399118,2cyclictest26836-21grep19:55:131
488399118,2cyclictest24493-21wget23:40:131
488399118,2cyclictest22893-21ssh23:39:171
488399118,2cyclictest22893-21ssh23:39:161
488399118,2cyclictest21289-21grep19:40:261
488399118,2cyclictest19130-21head20:45:151
488399118,2cyclictest15359-21cut21:20:161
488399118,2cyclictest15097-21cut19:30:151
488399118,2cyclictest14139-21awk00:30:271
488399118,2cyclictest12528-21cat20:30:231
488399117,3cyclictest9255-21aten2_r7power_v23:50:161
488399117,3cyclictest21390-21grep20:50:161
488399117,3cyclictest11952-21ssh22:35:101
4883991111,0cyclictest0-21swapper/121:32:461
488399110,10cyclictest15850-21sed20:40:001
488399110,10cyclictest1228-21expr21:50:151
488199119,2cyclictest9851-21ssh00:28:410
488199119,1cyclictest5625-21grep20:15:280
488199119,1cyclictest32304-21grep20:05:180
488199119,1cyclictest29240-21cut22:45:150
488199119,1cyclictest25228-21grep23:40:160
488199119,1cyclictest16089-21grep19:30:230
488199119,1cyclictest10616-21grep19:20:180
488199118,2cyclictest32461-21ssh22:08:390
488199118,2cyclictest28243-21ssh23:03:500
488199118,2cyclictest22782-21taskset23:57:580
488199118,2cyclictest21100-21tr20:50:160
488199118,2cyclictest18675-21tail22:00:180
488199118,2cyclictest14007-21rm22:16:270
488199118,2cyclictest0-21swapper/021:51:340
488199112,3cyclictest24810-21ssh21:45:140
488199112,3cyclictest24810-21ssh21:45:140
488199111,4cyclictest225-21systemd-journal21:35:140
4881991110,1cyclictest5239-21ssh22:50:140
279842111,2sleep327990-21apt-get21:05:133
488899108,1cyclictest6302-21apache_volume20:20:123
488899108,1cyclictest5465-21cat19:10:153
488899108,1cyclictest24187-21cat20:55:203
488899108,1cyclictest19901-21cat20:45:253
488899108,1cyclictest130682sleep320:33:593
488899108,1cyclictest10605-21chrt20:27:513
488899108,1cyclictest1028-21taskset20:07:183
488899107,2cyclictest21117-21grep20:50:163
488899107,2cyclictest17945-21taskset20:42:353
488899105,2cyclictest225-21systemd-journal19:57:103
488699107,2cyclictest334-21awk20:05:222
488699107,2cyclictest17021-21ssh21:40:172
488699102,7cyclictest0-21swapper/221:05:162
488699102,2cyclictest0-21swapper/219:10:132
488399108,1cyclictest3502-21chrt20:13:271
488399108,1cyclictest143872sleep119:27:341
488399108,1cyclictest10231-21expr19:20:151
488399107,2cyclictest7605-21cut20:20:221
488399107,2cyclictest25564-21ntpq19:50:221
488399107,2cyclictest25564-21ntpq19:50:211
488199108,2cyclictest26646-21cat21:00:190
488199108,1cyclictest28049-21basename21:05:150
488199108,1cyclictest23276-21ssh21:25:020
488199108,1cyclictest11979-21uname20:30:180
488199107,2cyclictest14050-21cat19:25:290
488199105,2cyclictest225-21systemd-journal20:25:140
488199104,6cyclictest31321-21munin-run20:05:010
488199103,3cyclictest0-21swapper/021:57:180
48819998,1cyclictest23818-21taskset19:45:450
48819997,2cyclictest6673-21expr20:20:140
48819997,2cyclictest17604-21expr19:35:160
48819997,1cyclictest195852sleep019:40:070
48819996,2cyclictest16259-21grep20:40:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional