You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-26 - 17:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Fri Jun 26, 2026 12:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
495799260,25cyclictest15605-21latency_hist10:50:001
495699260,25cyclictest4877-21sh09:42:370
4960992523,1cyclictest0-21swapper/312:07:463
496099250,24cyclictest31463-21diskmemload11:52:543
496099250,24cyclictest25328-21if_enp1s010:00:443
495799250,24cyclictest31463-21diskmemload10:07:561
495799250,1cyclictest1365-21ntp_states09:10:461
495699253,21cyclictest62550irq/127-eno110:00:230
495699250,24cyclictest31463-21diskmemload11:28:220
495699250,24cyclictest27748-21ssh11:58:060
495699250,24cyclictest0-21swapper/008:35:400
495699250,1cyclictest31463-21diskmemload09:18:220
4960992422,1cyclictest4022-21apt12:35:013
496099242,21cyclictest31463-21diskmemload10:15:443
496099242,21cyclictest24309-21sh11:55:393
496099240,23cyclictest30374-21timerwakeupswit11:30:503
496099240,23cyclictest0-21swapper/312:29:133
496099240,23cyclictest0-21swapper/312:21:303
496099240,23cyclictest0-21swapper/309:20:443
4958992421,0cyclictest0-21swapper/209:15:512
495899241,1cyclictest0-21swapper/210:20:402
495899240,3cyclictest27758-21cron09:35:012
495899240,23cyclictest2681-21munin-run08:10:012
495899240,23cyclictest0-21swapper/210:45:322
495899240,23cyclictest0-21swapper/210:35:472
495799240,23cyclictest31463-21diskmemload12:11:031
495799240,0cyclictest0-21swapper/111:53:561
495799240,0cyclictest0-21swapper/111:13:231
495799240,0cyclictest0-21swapper/110:50:311
4956992423,1cyclictest31463-21diskmemload10:30:320
4956992422,1cyclictest23859-21sh12:23:140
495699240,3cyclictest12918-21rm10:46:250
495699240,24cyclictest31463-21diskmemload11:47:000
495699240,24cyclictest31463-21diskmemload10:27:240
495699240,24cyclictest31463-21diskmemload09:38:530
495699240,23cyclictest19801-21ssh09:26:390
495699240,23cyclictest19801-21ssh09:26:390
495699240,1cyclictest0-21swapper/012:39:410
495699240,0cyclictest0-21swapper/010:16:250
4960992323,0cyclictest31463-21diskmemload09:48:493
4960992323,0cyclictest31463-21diskmemload09:32:263
4960992323,0cyclictest0-21swapper/312:17:103
4960992323,0cyclictest0-21swapper/311:46:573
4960992322,1cyclictest0-21swapper/311:02:493
4960992322,1cyclictest0-21swapper/309:56:063
4960992321,1cyclictest0-21swapper/310:55:063
4960992321,1cyclictest0-21swapper/309:12:173
4960992320,2cyclictest2499-21dump-pmu-power12:05:003
4960992320,2cyclictest12683-21sh10:45:563
496099230,23cyclictest0-21swapper/311:27:333
496099230,23cyclictest0-21swapper/311:10:393
496099230,23cyclictest0-21swapper/309:43:113
496099230,22cyclictest4596-21ssh11:36:573
496099230,22cyclictest4261-21runrttasks12:35:023
496099230,22cyclictest31463-21diskmemload10:40:483
496099230,22cyclictest31463-21diskmemload09:52:123
496099230,22cyclictest27001-21apt-get09:00:403
496099230,22cyclictest18255-21sh10:50:513
496099230,22cyclictest0-21swapper/310:20:413
496099230,0cyclictest0-21swapper/311:15:533
496099230,0cyclictest0-21swapper/309:35:523
496099230,0cyclictest0-21swapper/308:25:413
4958992323,0cyclictest31463-21diskmemload11:01:312
4958992323,0cyclictest0-21swapper/211:40:322
4958992323,0cyclictest0-21swapper/211:31:512
4958992323,0cyclictest0-21swapper/210:40:092
4958992323,0cyclictest0-21swapper/209:55:452
4958992323,0cyclictest0-21swapper/209:50:562
4958992323,0cyclictest0-21swapper/209:40:552
4958992322,1cyclictest12588-21apt-get08:30:402
4958992321,1cyclictest25929-21ssh10:30:192
4958992321,1cyclictest25451-21sh10:30:012
4958992321,1cyclictest0-21swapper/211:53:462
4958992320,3cyclictest0-21swapper/212:30:052
4958992320,2cyclictest31463-21diskmemload09:38:092
495899230,23cyclictest0-21swapper/210:56:332
495899230,22cyclictest6865-21grep11:10:432
495899230,22cyclictest31486-21apt-get09:10:012
495899230,22cyclictest31463-21diskmemload12:20:432
495899230,22cyclictest31463-21diskmemload11:27:332
495899230,22cyclictest31463-21diskmemload09:46:202
495899230,22cyclictest17326-21ssh09:25:412
495899230,22cyclictest17326-21ssh09:25:412
495899230,22cyclictest10157-21awk07:20:012
495899230,1cyclictest4667-21apt-get12:35:412
495899230,1cyclictest0-21swapper/212:15:392
495899230,1cyclictest0-21swapper/212:04:022
495899230,0cyclictest31463-21diskmemload11:57:472
495899230,0cyclictest31463-21diskmemload09:22:422
495899230,0cyclictest0-21swapper/212:26:292
495899230,0cyclictest0-21swapper/212:10:032
495899230,0cyclictest0-21swapper/210:00:392
4957992323,0cyclictest31463-21diskmemload10:26:461
4957992323,0cyclictest0-21swapper/112:28:461
4957992323,0cyclictest0-21swapper/112:01:091
4957992323,0cyclictest0-21swapper/111:33:221
4957992323,0cyclictest0-21swapper/111:20:061
4957992323,0cyclictest0-21swapper/110:16:461
4957992323,0cyclictest0-21swapper/109:31:061
4957992321,2cyclictest18761-21sh09:55:271
4957992321,1cyclictest1902-21ssh10:36:361
4957992321,1cyclictest15348-21sh11:46:181
4957992321,1cyclictest0-21swapper/112:20:151
4957992320,2cyclictest27440-21sh10:02:131
4957992320,2cyclictest1468-21ssh09:40:201
495799230,2cyclictest7792-21rm12:37:351
495799230,23cyclictest0-21swapper/110:21:571
495799230,23cyclictest0-21swapper/109:45:141
495799230,23cyclictest0-21swapper/109:36:311
495799230,22cyclictest31262-21ssh12:30:221
495799230,22cyclictest2677-21idleruntime-cro08:10:011
495799230,22cyclictest23805-21sh10:55:571
495799230,22cyclictest1873-21ntp_states11:05:461
495799230,22cyclictest14488-21sh11:16:471
495799230,0cyclictest0-21swapper/110:40:491
4956992323,0cyclictest30158-21ssh10:05:380
4956992323,0cyclictest0-21swapper/012:26:430
4956992323,0cyclictest0-21swapper/011:22:300
4956992323,0cyclictest0-21swapper/010:59:440
4956992323,0cyclictest0-21swapper/010:40:290
4956992323,0cyclictest0-21swapper/010:20:060
4956992323,0cyclictest0-21swapper/009:31:330
4956992321,1cyclictest9969-21ssh12:10:470
495699230,23cyclictest31463-21diskmemload12:31:390
495699230,23cyclictest0-21swapper/010:51:330
495699230,22cyclictest2667-21apt-get10:10:020
495699230,22cyclictest25418-21cron07:50:010
495699230,22cyclictest20735-21sed09:55:500
495699230,22cyclictest1913-21sh11:05:470
495699230,22cyclictest14987-21ssh09:51:070
495699230,0cyclictest0-21swapper/012:05:280
495699230,0cyclictest0-21swapper/011:00:400
4960992222,0cyclictest0-21swapper/310:25:223
4960992222,0cyclictest0-21swapper/309:25:203
4960992222,0cyclictest0-21swapper/309:25:203
4960992222,0cyclictest0-21swapper/307:40:403
4960992220,1cyclictest31847-21apt-get10:35:413
4960992220,1cyclictest17714-21apt-get07:35:023
4960992220,1cyclictest10449-21apt-get07:20:403
4960992220,1cyclictest0-21swapper/311:40:133
4960992220,1cyclictest0-21swapper/311:06:353
4960992220,1cyclictest0-21swapper/310:06:483
496099220,22cyclictest0-21swapper/311:21:273
496099220,21cyclictest31486-21apt09:10:003
496099220,21cyclictest31463-21diskmemload12:10:353
496099220,21cyclictest0-21swapper/309:16:183
496099220,21cyclictest0-21swapper/308:40:403
496099220,1cyclictest9884-21turbostat08:25:003
496099220,1cyclictest0-21swapper/310:13:093
4958992222,0cyclictest31463-21diskmemload11:05:232
4958992220,1cyclictest1965-21apt-get11:35:412
4958992220,1cyclictest0-21swapper/210:16:172
495899220,2cyclictest23020-21awk07:45:002
495899220,21cyclictest0-21swapper/211:45:192
495899220,21cyclictest0-21swapper/211:15:152
495899220,21cyclictest0-21swapper/210:10:152
495899220,1cyclictest31463-21diskmemload11:22:012
495899220,1cyclictest0-21swapper/212:05:392
495899220,1cyclictest0-21swapper/207:55:012
4957992220,2cyclictest28829-21ssh10:31:531
4957992220,1cyclictest22883-21taskset11:25:251
4957992220,1cyclictest19427-21sh09:26:101
4957992220,1cyclictest19427-21sh09:26:101
4957992220,1cyclictest0-21swapper/111:02:361
495799220,22cyclictest31463-21diskmemload09:22:591
495799220,22cyclictest0-21swapper/111:43:191
495799220,21cyclictest5695-21ssh09:15:371
495799220,21cyclictest5255-21ssh12:06:071
495799220,21cyclictest24586-21apt-get08:55:401
495799220,21cyclictest17107-21awk08:40:011
495799220,21cyclictest12840-21apt-get07:25:391
495799220,21cyclictest0-21swapper/110:10:241
495799220,21cyclictest0-21swapper/109:50:441
495799220,0cyclictest0-21swapper/111:55:451
4956992222,0cyclictest31463-21diskmemload09:11:110
4956992222,0cyclictest0-21swapper/011:42:190
4956992222,0cyclictest0-21swapper/009:45:400
495699220,22cyclictest0-21swapper/011:13:020
495699220,21cyclictest29101-21munin-run09:05:000
495699220,21cyclictest28433-21ssh11:30:220
495699220,21cyclictest23041-21sleep07:45:000
495699220,21cyclictest0-21swapper/011:35:440
495699220,1cyclictest14261-21sh11:16:290
495699220,1cyclictest12198-1kworker/u9:009:20:490
495699220,1cyclictest0-21swapper/011:50:560
495699220,1cyclictest0-21swapper/010:35:280
4960992119,1cyclictest8033-21apt-get07:15:443
4960992119,1cyclictest25730-21apt-get07:50:393
4960992119,1cyclictest0-21swapper/308:55:403
4958992119,1cyclictest10189-21apt-get08:25:402
495899210,21cyclictest0-21swapper/210:05:202
495899210,20cyclictest0-21swapper/209:05:392
495899210,0cyclictest0-21swapper/208:50:402
4957992119,1cyclictest31470-21sh09:10:001
495799210,21cyclictest0-21swapper/111:35:151
495799210,20cyclictest0-21swapper/108:45:411
495799210,0cyclictest0-21swapper/107:55:411
495799210,0cyclictest0-21swapper/107:50:391
495799210,0cyclictest0-21swapper/107:35:391
4956992121,0cyclictest0-21swapper/012:15:100
4956992120,1cyclictest0-21swapper/007:50:390
495699210,20cyclictest0-21swapper/008:00:400
495699210,20cyclictest0-21swapper/007:15:440
4960992020,0cyclictest0-21swapper/308:10:403
4960992020,0cyclictest0-21swapper/308:05:413
4960992019,0cyclictest12840-21apt-get07:25:393
4957992020,0cyclictest0-21swapper/109:00:401
4957992020,0cyclictest0-21swapper/107:45:411
4957992020,0cyclictest0-21swapper/107:40:401
4956992020,0cyclictest17400-21apt-get08:40:410
4956992020,0cyclictest0-21swapper/009:05:390
4956992020,0cyclictest0-21swapper/008:45:410
4956992020,0cyclictest0-21swapper/008:25:400
4956992020,0cyclictest0-21swapper/007:10:420
4960991919,0cyclictest0-21swapper/308:50:403
4960991917,1cyclictest7337-21perf07:15:013
4960991917,1cyclictest17739-21munin-run07:35:003
4960991916,2cyclictest32630-21dump-pmu-power08:05:013
4958991919,0cyclictest0-21swapper/208:55:402
4958991919,0cyclictest0-21swapper/208:20:402
4958991919,0cyclictest0-21swapper/207:45:412
4958991919,0cyclictest0-21swapper/207:30:412
4958991919,0cyclictest0-21swapper/207:10:422
4957991919,0cyclictest2977-21apt-get08:10:401
4957991919,0cyclictest0-21swapper/108:20:391
4957991917,1cyclictest7479-21grep08:20:001
4956991919,0cyclictest0-21swapper/007:20:400
4960991818,0cyclictest0-21swapper/308:35:403
4960991817,1cyclictest5377-21apt-get08:15:393
4958991818,0cyclictest0-21swapper/208:45:412
4958991818,0cyclictest0-21swapper/207:55:412
4957991817,1cyclictest0-21swapper/108:30:411
4957991817,1cyclictest0-21swapper/108:25:411
4957991816,1cyclictest15635-21apt-get07:30:411
495799180,17cyclictest30545-21apt-get08:00:401
495799180,17cyclictest19503-21munin-run08:45:011
4956991818,0cyclictest0-21swapper/008:15:400
4956991817,1cyclictest28130-21apt-get07:55:410
4960991717,0cyclictest0-21swapper/307:55:413
4960991716,0cyclictest19796-21apt-get08:45:413
4958991717,0cyclictest0-21swapper/208:40:402
4958991717,0cyclictest0-21swapper/207:35:022
495899170,14cyclictest0-21swapper/208:17:532
4957991717,0cyclictest0-21swapper/107:20:401
4957991717,0cyclictest0-21swapper/107:10:421
4956991717,0cyclictest0-21swapper/008:05:410
495699170,16cyclictest17714-21apt-get07:35:000
4958991614,1cyclictest14902-21cron07:30:002
495699160,1cyclictest25635-21ntp_states08:55:460
4958991515,0cyclictest0-21swapper/208:10:402
4958991513,1cyclictest0-21swapper/209:00:402
495899150,1cyclictest11352-21memory07:20:442
4957991514,0cyclictest0-21swapper/107:15:441
4956991515,0cyclictest0-21swapper/008:30:400
4956991515,0cyclictest0-21swapper/007:35:020
495699150,14cyclictest0-21swapper/008:25:000
496099140,14cyclictest0-21swapper/308:30:333
496099140,14cyclictest0-21swapper/307:45:443
4958991414,0cyclictest0-21swapper/208:36:052
4958991414,0cyclictest0-21swapper/208:00:462
495799140,0cyclictest0-21swapper/108:50:011
4956991414,0cyclictest0-21swapper/008:10:400
495699140,0cyclictest0-21swapper/008:51:520
495699140,0cyclictest0-21swapper/007:25:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional