You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-29 - 16:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Mar 29, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3218529667,8sleep20-21swapper/207:05:202
62528976,9sleep00-21swapper/007:05:370
83728267,8sleep30-21swapper/307:08:263
87228164,13sleep10-21swapper/107:08:521
164202600,1sleep216411-21ssh12:15:182
1056994339,3cyclictest25166-21kworker/3:012:21:033
104999281,2cyclictest0-21swapper/112:35:121
105299270,26cyclictest0-21swapper/210:02:142
1056992622,3cyclictest16219-21kworker/3:210:25:133
105699240,23cyclictest16219-21kworker/3:211:34:433
105299240,18cyclictest0-21swapper/212:31:172
1056992320,2cyclictest25166-21kworker/3:012:17:033
105299230,0cyclictest0-21swapper/210:35:032
104699230,15cyclictest0-21swapper/009:55:120
105699221,21cyclictest16219-21kworker/3:210:40:323
105699220,21cyclictest16219-21kworker/3:211:29:033
1052992222,0cyclictest0-21swapper/209:34:442
105299220,22cyclictest0-21swapper/211:57:092
105299220,19cyclictest0-21swapper/209:16:002
1056992119,2cyclictest25166-21kworker/3:012:07:433
1056992119,2cyclictest16219-21kworker/3:211:46:133
105699210,21cyclictest16219-21kworker/3:211:53:023
105699210,21cyclictest16219-21kworker/3:210:53:133
1052992120,1cyclictest0-21swapper/209:39:222
105299210,3cyclictest0-21swapper/209:52:292
105299210,1cyclictest26066-21diskmemload12:12:132
1056992018,2cyclictest25166-21kworker/3:012:04:533
1056992018,2cyclictest16219-21kworker/3:211:00:533
1056992018,2cyclictest16219-21kworker/3:210:35:033
105699200,19cyclictest16219-21kworker/3:210:30:333
1052992012,2cyclictest32-21ksoftirqd/208:50:142
104999200,19cyclictest0-21swapper/109:15:121
104699200,1cyclictest0-21swapper/011:23:550
1056991918,1cyclictest16219-21kworker/3:210:49:333
1056991917,2cyclictest16219-21kworker/3:210:19:533
1056991916,3cyclictest16219-21kworker/3:210:08:433
105699191,3cyclictest0-21swapper/311:15:063
105699191,18cyclictest16219-21kworker/3:211:24:233
1052991912,2cyclictest23172-21wc09:45:262
105299190,16cyclictest0-21swapper/209:57:132
105299190,14cyclictest0-21swapper/210:40:102
105299190,14cyclictest0-21swapper/207:45:142
1049991917,1cyclictest0-21swapper/109:51:201
104999190,19cyclictest0-21swapper/110:17:231
104999190,18cyclictest0-21swapper/111:39:501
104699195,7cyclictest11745-21grep08:40:000
1056991816,2cyclictest25166-21kworker/3:012:25:233
1056991816,2cyclictest16219-21kworker/3:211:40:333
1052991810,2cyclictest24079-21grep12:20:142
104999180,18cyclictest0-21swapper/111:19:201
104999180,18cyclictest0-21swapper/107:55:251
104699180,15cyclictest0-21swapper/012:05:260
105699173,7cyclictest16619-21ssh12:35:133
1056991715,2cyclictest16219-21kworker/3:211:12:433
105299170,17cyclictest0-21swapper/212:29:262
105299170,16cyclictest0-21swapper/209:40:452
1049991716,1cyclictest0-21swapper/111:54:011
1049991714,2cyclictest38150irq/131-snd_hda12:21:571
104999170,1cyclictest0-21swapper/110:57:501
104999170,17cyclictest0-21swapper/112:26:391
104699170,1cyclictest0-21swapper/011:25:220
1056991615,1cyclictest16219-21kworker/3:209:40:213
105699160,1cyclictest0-21swapper/311:59:273
105299164,1cyclictest141rcu_preempt10:31:502
105299160,16cyclictest0-21swapper/212:36:392
105299160,13cyclictest0-21swapper/211:00:162
104999160,1cyclictest0-21swapper/111:44:441
104699166,5cyclictest9964-21apache_volume10:55:120
104699160,10cyclictest20886-21basename09:45:010
292342150,0chrt0-21swapper/309:30:183
105299150,2cyclictest0-21swapper/210:10:142
1049991513,1cyclictest0-21swapper/111:27:461
104999151,1cyclictest27124-21chrt12:02:271
104999150,1cyclictest0-21swapper/110:50:371
104999150,14cyclictest0-21swapper/110:05:381
104699154,6cyclictest19900-21awk09:25:140
1046991515,0cyclictest0-21swapper/010:06:420
104699150,14cyclictest0-21swapper/011:38:580
142602140,0sleep30-21swapper/311:35:253
105699149,3cyclictest32508-21tr10:10:213
105699141,6cyclictest0-21swapper/312:30:023
1056991413,1cyclictest0-21swapper/312:10:413
1052991413,1cyclictest0-21swapper/210:57:512
1052991411,2cyclictest2740-21tr08:20:142
1052991410,3cyclictest26066-21diskmemload11:49:262
1052991410,3cyclictest20465-21sort11:20:162
105299140,14cyclictest0-21swapper/211:06:162
105299140,14cyclictest0-21swapper/209:21:242
104999148,5cyclictest22007-21basename12:00:001
104999142,5cyclictest0-21swapper/109:30:161
1049991413,1cyclictest25-21ksoftirqd/110:12:321
1049991410,3cyclictest38150irq/131-snd_hda07:45:151
104999140,1cyclictest0-21swapper/109:20:201
104999140,13cyclictest0-21swapper/110:42:151
104699142,5cyclictest0-21swapper/011:10:010
1046991411,2cyclictest24545-21cron12:40:000
1046991410,3cyclictest30842-21ls11:45:200
105699138,4cyclictest11677-21cat10:55:263
105699135,6cyclictest0-21swapper/310:20:123
105699132,6cyclictest27750-21apt-get08:05:153
105699131,11cyclictest0-21swapper/309:35:363
1056991310,2cyclictest5800-21systemctl07:20:003
1056991310,2cyclictest27217-21grep11:05:163
1056991310,2cyclictest25072-21cron08:00:003
1056991310,2cyclictest19166-21sh09:25:003
105299139,3cyclictest5786-21sh07:20:002
105299139,3cyclictest24911-21grep10:25:162
1052991310,2cyclictest9545-21uname07:25:182
1052991310,2cyclictest9545-21uname07:25:182
1052991310,2cyclictest6949-21cron08:30:002
1052991310,2cyclictest26616-21cat10:45:192
105299130,13cyclictest0-21swapper/211:44:292
104999139,3cyclictest21360-21ssh09:45:121
104999139,3cyclictest19824-21basename11:20:141
104999131,11cyclictest19408-21ssh10:02:511
1049991310,2cyclictest32538-21basename08:15:131
1049991310,2cyclictest19894-21grep08:55:171
1049991310,1cyclictest4594-21apt08:24:591
104699139,3cyclictest7318-21grep10:15:130
104699135,7cyclictest19167-21munin-run11:20:000
104699134,4cyclictest23776-21munin-node12:20:100
1046991311,2cyclictest13-21ksoftirqd/009:51:020
1046991310,2cyclictest9521-21cat10:55:000
1046991310,2cyclictest28224-21wc09:10:220
1046991310,2cyclictest27663-21cat08:05:000
1046991310,2cyclictest25250-21cat08:00:010
1046991310,2cyclictest24409-21tr10:25:140
1046991310,2cyclictest18247-21tr07:45:160
1046991310,2cyclictest12141-21perf11:35:010
105699129,2cyclictest5426-21cat09:55:013
105699129,2cyclictest21827-21tr09:00:143
105699129,2cyclictest21377-21head07:50:153
105699129,2cyclictest15830-21grep07:40:133
105699128,3cyclictest2514-21grep07:10:213
105699128,3cyclictest22254-21cron10:05:003
105699128,3cyclictest2186-21cron08:20:003
105699128,2cyclictest18719-21grep07:45:183
105299129,2cyclictest8846-21grep08:30:262
105299129,2cyclictest6562-21grep11:50:172
105299129,2cyclictest32220-21cron08:15:002
105299129,2cyclictest31091-21awk12:05:142
105299129,2cyclictest24702-21sed09:05:172
105299129,2cyclictest24117-21taskset12:00:232
105299129,2cyclictest2133-21grep10:50:152
105299129,2cyclictest15316-21perf07:40:002
105299124,5cyclictest20261-21ssh09:25:142
105299123,5cyclictest225-21systemd-journal07:45:002
105299121,1cyclictest141rcu_preempt09:11:052
104999129,2cyclictest3060-21seq10:31:061
104999129,2cyclictest14848-21sh12:15:001
104999129,2cyclictest14091-21cron08:45:001
104999129,2cyclictest0-21swapper/111:30:071
104999128,3cyclictest20381-21ssh09:25:151
104999128,3cyclictest16081-21cron12:35:001
104999127,4cyclictest4923-21grep08:25:141
104999127,4cyclictest31500-21grep11:45:261
104999124,4cyclictest225-21systemd-journal10:50:001
104999123,8cyclictest0-21swapper/109:40:161
104999122,9cyclictest26066-21diskmemload11:05:021
1049991211,1cyclictest0-21swapper/112:06:001
1049991210,1cyclictest0-21swapper/111:03:561
104999120,1cyclictest0-21swapper/111:11:251
104699129,2cyclictest9629-21ls07:25:200
104699129,2cyclictest9629-21ls07:25:190
104699129,2cyclictest6815-21head07:20:160
104699129,2cyclictest4767-21tail11:10:260
104699129,2cyclictest29353-21wc09:50:000
104699129,2cyclictest25454-21sort09:05:260
104699129,2cyclictest23669-21sh09:05:000
104699128,2cyclictest5877-21cut11:50:140
104699128,2cyclictest25201-21wc10:45:000
104699120,6cyclictest1027-21perf10:50:000
104699120,1cyclictest0-21swapper/009:37:560
105699118,2cyclictest7997-21ssh09:55:333
105699118,2cyclictest7136-21cat07:20:193
105699118,2cyclictest5459-21ssh09:15:563
105699118,2cyclictest3216-21grep08:20:163
105699118,2cyclictest27240-21ssh09:10:163
105699118,2cyclictest25826-21chrt09:07:173
105699118,2cyclictest23263-21unixbench_multi09:45:263
105699118,2cyclictest19658-21df08:55:153
105699118,2cyclictest19419-21ssh09:25:023
105699118,2cyclictest16975-21tr08:50:143
105699118,2cyclictest16206-21seq08:46:443
105699118,1cyclictest89732sleep308:30:313
105299118,2cyclictest7932-21chrt07:22:032
105299118,2cyclictest5305-21ssh11:10:482
105299118,2cyclictest3248-21taskset07:13:152
105299118,2cyclictest30105-21sed11:25:272
105299118,2cyclictest28462-21head08:05:192
105299118,2cyclictest27468-21cron08:05:002
105299118,2cyclictest1744-21grep08:15:252
105299118,2cyclictest15979-21grep08:45:252
105299118,2cyclictest13368-21ssh11:15:322
105299118,2cyclictest11034-21sort07:30:152
105299118,2cyclictest10480-21ssh10:16:252
105299118,2cyclictest10340-21ssh11:33:382
105299117,2cyclictest21366-21grep07:50:152
105299115,3cyclictest0-21swapper/210:06:072
105299111,2cyclictest225-21systemd-journal11:40:002
105299110,10cyclictest19351-21expr08:55:132
104999119,1cyclictest3684-21cut07:15:131
104999118,2cyclictest7150-21rm09:55:201
104999118,2cyclictest4587-21ssh09:35:161
104999118,2cyclictest27435-21taskset08:04:361
104999118,2cyclictest27343-21grep09:10:161
104999118,2cyclictest26047-21sh09:10:001
104999118,2cyclictest19731-21ssh12:17:171
104999118,2cyclictest17471-21grep08:50:171
104999118,2cyclictest15915-21grep10:20:131
104999118,2cyclictest12044-21head07:30:221
104999118,2cyclictest10362-21sed08:35:181
104999117,3cyclictest9128-21ssh10:35:141
104999117,3cyclictest14450-21wc07:35:241
104999110,10cyclictest7356-21cut08:30:151
104699119,1cyclictest22871-21expr12:00:130
104699119,1cyclictest14987-21grep10:00:150
104699118,2cyclictest7980-21sed08:30:170
104699118,2cyclictest4063-21ssh09:15:190
104699118,2cyclictest3105-21sort08:20:160
104699118,2cyclictest3102-21ssh10:12:140
104699118,2cyclictest29586-21head09:30:210
104699118,2cyclictest29068-21grep11:45:000
104699118,2cyclictest28495-21sed08:05:190
104699118,2cyclictest23222-21ssh11:02:360
104699118,2cyclictest2035-21taskset08:18:080
104699118,2cyclictest2017-21head07:10:150
104699118,2cyclictest18911-21sh10:21:220
104699118,2cyclictest16868-21fgrep12:15:220
104699118,2cyclictest16851-21sh11:56:030
104699118,2cyclictest13944-21sed07:35:190
104699118,2cyclictest13763-21ssh09:20:480
104699118,2cyclictest12977-21taskset10:37:150
104699118,2cyclictest12927-21uname08:40:190
104699118,2cyclictest1247-21ls12:25:220
104699118,2cyclictest12247-21cat07:30:250
104699116,5cyclictest31055-21munin-node12:35:000
104699110,10cyclictest6832-21ssh12:10:080
105699108,1cyclictest9955-21expr08:35:163
105699108,1cyclictest5264-21expr08:25:153
105699108,1cyclictest26156-21expr08:00:173
105699108,1cyclictest15192-21taskset07:38:523
105699108,1cyclictest13034-21cat08:40:193
105699107,2cyclictest10301-21seq07:27:023
105699107,2cyclictest10301-21seq07:27:023
105699100,9cyclictest300712sleep308:10:003
105299108,1cyclictest235682sleep209:03:482
105299107,2cyclictest24077-21grep07:55:192
105299107,2cyclictest15988-21ssh10:20:132
105299107,2cyclictest12847-21if_enp0s31f608:40:182
104999108,1cyclictest28095-21expr08:05:161
104999107,2cyclictest7424-21awk07:20:221
104999107,2cyclictest23924-21ssh10:25:041
104999107,2cyclictest22541-21taskset07:53:011
104999107,2cyclictest16037-21expr07:40:131
104999107,2cyclictest14347-21bc08:45:091
104999100,9cyclictest9445-21sed07:25:171
104999100,9cyclictest9445-21sed07:25:171
104999100,9cyclictest31693-21sort08:10:261
104999100,9cyclictest2301-21cat07:10:181
104699108,1cyclictest6854-21chrt08:28:480
104699108,1cyclictest173392sleep007:40:360
104699108,1cyclictest16708-21taskset08:50:110
104699108,1cyclictest15161-21hwlatdetect08:45:160
104699108,1cyclictest0-21swapper/008:55:200
104699107,2cyclictest3857-21egrep07:15:140
104699107,2cyclictest30477-21expr08:10:140
104699107,2cyclictest21010-21expr07:50:130
104699100,4cyclictest0-21swapper/010:30:010
10569997,1cyclictest11775-21expr07:30:193
10529997,1cyclictest9599-21taskset08:35:142
10499997,1cyclictest21817-21cut09:00:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional