You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-03 - 20:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Jun 03, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
327629776,16sleep30-21swapper/307:05:163
13132950,1sleep30-21swapper/310:10:013
478528867,6sleep10-21swapper/107:09:281
476928875,9sleep00-21swapper/007:09:150
462028875,9sleep20-21swapper/207:07:192
236772820,1sleep323670-21ssh11:20:203
157392690,0sleep10-21swapper/109:59:131
491799340,32cyclictest0-21swapper/211:26:032
4917993228,4cyclictest0-21swapper/210:40:332
491799321,30cyclictest20725-21taskset10:01:332
491799271,25cyclictest10392-21apache_volume10:15:132
491799270,1cyclictest0-21swapper/211:45:232
4917992624,2cyclictest0-21swapper/212:30:232
4917992622,4cyclictest0-21swapper/211:40:232
491799250,24cyclictest0-21swapper/211:10:122
491799250,0cyclictest0-21swapper/210:57:572
491399250,2cyclictest0-21swapper/110:50:021
491799240,24cyclictest0-21swapper/211:51:232
491799240,24cyclictest0-21swapper/210:25:512
491799240,23cyclictest0-21swapper/209:25:332
491999231,1cyclictest0-21swapper/309:05:133
491799231,22cyclictest30207-21ssh12:03:232
491799231,21cyclictest17351-21ssh09:20:332
491799230,23cyclictest0-21swapper/207:56:472
491799230,22cyclictest0-21swapper/211:55:332
4917992219,3cyclictest0-21swapper/210:05:032
4917992219,3cyclictest0-21swapper/209:10:432
4917992218,4cyclictest0-21swapper/211:17:332
4917992218,3cyclictest30136-21diskmemload12:26:132
491799220,22cyclictest0-21swapper/211:09:132
491799220,22cyclictest0-21swapper/210:11:232
491799220,22cyclictest0-21swapper/209:16:032
491799220,21cyclictest30136-21diskmemload10:30:532
4917992118,3cyclictest0-21swapper/212:16:132
4917992117,3cyclictest14564-21ssh12:13:222
491799211,19cyclictest1026-21sensors09:30:222
491799210,21cyclictest0-21swapper/212:20:332
491799210,21cyclictest0-21swapper/212:06:032
491799210,21cyclictest0-21swapper/211:30:132
491799210,21cyclictest0-21swapper/211:02:532
491799210,21cyclictest0-21swapper/210:20:032
491799210,21cyclictest0-21swapper/209:52:432
491399213,10cyclictest0-21swapper/108:00:131
491399213,10cyclictest0-21swapper/108:00:131
491399210,2cyclictest0-21swapper/109:36:391
491399210,20cyclictest0-21swapper/110:55:491
4908992119,1cyclictest0-21swapper/009:30:120
4917992017,3cyclictest0-21swapper/211:36:332
491399200,1cyclictest0-21swapper/112:15:141
490899200,1cyclictest0-21swapper/008:45:140
4917991916,3cyclictest0-21swapper/211:23:032
491799190,19cyclictest0-21swapper/210:51:132
491399190,19cyclictest0-21swapper/109:21:001
4908991916,1cyclictest0-21swapper/010:16:120
490899190,1cyclictest0-21swapper/011:49:420
4919991815,2cyclictest23319-21sh09:25:053
4919991814,3cyclictest7026-21kworker/u8:112:33:333
491999180,18cyclictest0-21swapper/312:27:063
491799180,17cyclictest24200-21latency_hist07:50:002
491799180,17cyclictest17108-21gltestperf09:40:172
491399172,13cyclictest30136-21diskmemload10:30:071
491399170,17cyclictest0-21swapper/111:22:311
490899170,13cyclictest0-21swapper/009:15:160
4913991614,2cyclictest25-21ksoftirqd/112:02:291
491399160,1cyclictest0-21swapper/111:51:031
491399160,15cyclictest13240-21grep10:35:251
490899162,8cyclictest22865-21sendmail-msp11:40:010
491999152,11cyclictest6066-21awk09:15:003
4919991510,3cyclictest24458-21ssh10:04:293
491999150,9cyclictest0-21swapper/309:49:523
491999150,14cyclictest0-21swapper/310:31:103
491799151,9cyclictest4723-21df_inode08:15:162
4917991511,2cyclictest1538-21grep08:10:002
4913991515,0cyclictest0-21swapper/110:15:421
490899158,4cyclictest0-21swapper/011:40:200
490899155,5cyclictest17138-21apache_volume12:15:120
490899153,11cyclictest3169-21python08:10:250
491999145,5cyclictest225-21systemd-journal11:26:313
491999144,9cyclictest26037-21idleruntime-cro12:39:593
4919991411,2cyclictest39-21ksoftirqd/310:36:113
4919991410,3cyclictest32386-21cat12:05:003
4919991410,3cyclictest22154-21grep07:45:143
4919991410,2cyclictest13827-21taskset10:55:193
4917991410,3cyclictest94-21kswapd009:37:312
491799140,7cyclictest17934-21fschecks_count07:35:172
491799140,13cyclictest0-21swapper/209:47:262
491399149,3cyclictest17023-21grep09:40:171
491399141,13cyclictest0-21swapper/110:12:281
4913991411,2cyclictest13766-21wc07:25:221
491399141,12cyclictest11366-21rm11:13:311
4913991411,2cyclictest10447-21wc12:10:271
4913991410,4cyclictest14301-21apache207:40:171
4913991410,3cyclictest11402-21grep08:30:151
491399140,14cyclictest0-21swapper/111:04:141
490899144,9cyclictest30136-21diskmemload12:13:250
490899144,4cyclictest6071-21dump-pmu-power09:15:000
4908991413,1cyclictest0-21swapper/012:06:190
490899140,14cyclictest0-21swapper/010:47:220
491999139,3cyclictest8227-21sed08:20:263
491999139,3cyclictest30325-21sed10:45:253
491999139,2cyclictest16625-21grep07:30:273
491999138,3cyclictest0-21swapper/310:14:433
4919991310,2cyclictest8340-21cron09:55:003
4919991310,2cyclictest24917-21cut07:50:153
4919991310,2cyclictest14854-21tr08:35:213
491999130,12cyclictest0-21swapper/310:20:203
491799139,3cyclictest11746-21grep07:20:272
491799138,3cyclictest18990-21df08:45:152
4917991310,2cyclictest24509-21wc08:55:222
4917991310,2cyclictest13183-21grep07:25:172
491799130,13cyclictest0-21swapper/208:00:152
491799130,13cyclictest0-21swapper/208:00:152
491799130,12cyclictest0-21swapper/208:41:412
491399139,2cyclictest17655-21awk12:35:001
4913991311,1cyclictest0-21swapper/110:24:421
4913991310,2cyclictest4844-21tune2fs08:15:171
4913991310,2cyclictest10881-21grep07:20:171
4913991310,2cyclictest0-21swapper/109:25:071
491399130,13cyclictest0-21swapper/112:36:011
490899139,3cyclictest6258-21cron08:20:000
490899139,2cyclictest19883-21cut10:40:130
490899132,9cyclictest18772-21apache_accesses10:20:110
4908991310,2cyclictest6798-21grep11:30:130
4908991310,2cyclictest29982-21ssh10:07:050
4908991310,2cyclictest25616-21wc12:00:180
4908991310,2cyclictest24115-21basename09:45:000
4908991310,2cyclictest18254-21tr12:35:140
490899130,12cyclictest0-21swapper/011:28:210
491999129,2cyclictest9305-21df12:10:163
491999129,2cyclictest32738-21head09:30:173
491999129,2cyclictest31787-21chrt11:45:133
491999129,2cyclictest23924-21grep11:40:153
491999129,2cyclictest23921-21grep08:55:163
491999129,2cyclictest21963-21head11:00:173
491999129,2cyclictest1585-21grep12:05:183
491999129,2cyclictest12014-21head08:30:183
491999129,2cyclictest10706-21tr10:15:153
491999128,3cyclictest8157-21smartctl11:30:243
491999126,2cyclictest225-21systemd-journal09:20:003
491999125,6cyclictest0-21swapper/309:40:003
491999121,2cyclictest0-21swapper/311:19:493
4919991210,1cyclictest30136-21diskmemload09:43:133
491999120,1cyclictest0-21swapper/311:56:023
491799129,2cyclictest26026-21ls12:39:592
491799129,2cyclictest25410-21grep07:50:182
491799129,2cyclictest22990-21grep08:55:002
491799129,2cyclictest16591-21grep07:30:262
491799129,2cyclictest12944-21sed08:30:272
491799129,2cyclictest12606-21grep10:35:192
491799128,3cyclictest28086-21sh10:45:012
491799128,3cyclictest15834-21sh08:40:002
491399129,2cyclictest9543-21grep08:25:171
491399129,2cyclictest3881-21chrt11:09:131
491399129,2cyclictest32423-21cron09:49:591
491399129,2cyclictest29648-21ssh10:45:201
491399129,2cyclictest28722-21sed07:55:261
491399129,2cyclictest2703-21cat10:30:001
491399129,2cyclictest24190-21dump-pmu-power07:50:001
491399129,2cyclictest23969-21ssh11:40:161
491399129,2cyclictest17510-21ssh11:35:531
491399129,2cyclictest1189-21grep11:45:271
491399127,2cyclictest13225-21ssh09:18:551
4913991210,1cyclictest25-21ksoftirqd/110:05:171
491399120,11cyclictest13743-21cut08:35:151
490899129,2cyclictest9169-21sed07:15:260
490899129,2cyclictest7268-21grep08:20:180
490899129,2cyclictest614-21grep08:05:220
490899129,2cyclictest4896-21cron11:10:000
490899129,2cyclictest27118-21grep07:55:140
490899129,2cyclictest21593-21grep08:50:180
490899129,2cyclictest19290-21sh10:40:000
490899129,2cyclictest16729-21grep11:55:160
490899128,2cyclictest9108-21tr09:55:150
490899127,4cyclictest0-21swapper/009:22:240
490899126,5cyclictest7267-21munin-run07:15:000
490899124,4cyclictest28007-21hddtemp_smartct10:25:180
4908991211,1cyclictest0-21swapper/012:26:180
491999118,2cyclictest9521-21sed11:50:253
491999118,2cyclictest8829-21gzip07:15:223
491999118,2cyclictest6819-21ssh11:10:223
491999118,2cyclictest5885-21head07:10:183
491999118,2cyclictest5021-21grep10:50:163
491999118,2cyclictest39-21ksoftirqd/311:35:113
491999118,2cyclictest31662-21ssh11:05:353
491999118,2cyclictest31031-21ssh12:23:143
491999118,2cyclictest27157-21cut10:25:143
491999118,2cyclictest23659-21ssh10:41:533
491999118,2cyclictest22841-21chrt08:53:233
491999118,2cyclictest2250-21expr08:10:153
491999118,2cyclictest19975-21tr07:40:153
491999118,2cyclictest19753-21proc_pri08:45:233
491999118,2cyclictest18050-21grep07:35:183
491999118,2cyclictest16881-21ssh12:15:013
491999118,2cyclictest15126-21acpi09:20:103
491999118,2cyclictest11577-21ssh09:55:583
491999118,2cyclictest11235-21grep07:20:213
491999117,3cyclictest30165-21head08:00:163
491999117,3cyclictest30165-21head08:00:163
491799118,2cyclictest9323-21taskset07:16:112
491799118,2cyclictest8742-21ssh09:55:082
491799118,2cyclictest5931-21tail07:10:182
491799118,2cyclictest2491-21expr08:10:172
491799118,2cyclictest21395-21head07:40:272
491799114,6cyclictest7133-21diskstats08:20:172
491799111,9cyclictest28025-21apt-get09:05:132
491399118,3cyclictest18884-21ssh11:56:041
491399118,2cyclictest9994-21ssh11:31:341
491399118,2cyclictest32717-21sh09:50:031
491399118,2cyclictest32596-21ssh12:05:041
491399118,2cyclictest32515-21tail11:25:281
491399118,2cyclictest2799-21grep12:25:231
491399118,2cyclictest26020-21expr12:20:141
491399118,2cyclictest22700-21chrt10:41:081
491399118,2cyclictest18998-21ssh11:17:551
491399118,2cyclictest18227-21cron08:45:001
491399118,2cyclictest1810-21cut08:10:141
491399118,2cyclictest1095-21sed08:05:271
491399116,2cyclictest225-21systemd-journal07:15:001
491399110,2cyclictest0-21swapper/109:11:011
491399110,10cyclictest21993-21ssh10:02:331
491399110,10cyclictest19419-21cron07:40:001
490899119,1cyclictest4912-21grep10:30:260
490899118,2cyclictest5724-21ssh10:11:560
490899118,2cyclictest5233-21sh09:52:340
490899118,2cyclictest5169-21ssh11:10:020
490899118,2cyclictest4615-21cut10:50:140
490899118,2cyclictest29904-21cat08:00:140
490899118,2cyclictest29904-21cat08:00:140
490899118,2cyclictest29245-21grep09:05:220
490899118,2cyclictest26267-21grep09:00:160
490899118,2cyclictest25729-21ssh11:02:280
490899118,2cyclictest23995-21ssh09:25:160
490899118,2cyclictest22839-21grep07:45:170
490899118,2cyclictest20337-21sort07:40:170
490899118,2cyclictest19618-21ssh10:00:420
490899118,2cyclictest16407-21grep08:40:140
490899118,2cyclictest12695-21sh10:55:120
490899118,2cyclictest12553-21ssh12:31:040
490899118,2cyclictest10269-21ssh09:35:490
490899117,2cyclictest25363-21turbostat.cron09:00:000
490899113,7cyclictest23672-21processes11:20:210
490899113,6cyclictest7628-21/usr/sbin/munin11:50:120
490899110,10cyclictest13497-21ssh11:15:010
491999108,1cyclictest5078-21uname08:15:193
491999108,1cyclictest32639-21basename08:05:183
491999108,1cyclictest27513-21expr07:55:153
491999108,1cyclictest16088-21taskset08:40:083
491999108,1cyclictest10704-21chrt08:25:543
491999107,2cyclictest12696-21grep07:25:153
491799107,2cyclictest26310-21grep09:00:172
491799100,5cyclictest8889-21taskset08:25:072
491399108,1cyclictest9488-21taskset07:18:191
491399108,1cyclictest29136-21nfs4_client09:05:201
491399108,1cyclictest26387-21sed07:50:271
491399108,1cyclictest2165-21ssh09:31:001
491399107,2cyclictest8052-21sed08:20:271
491399107,2cyclictest19857-21sendmail_mailtr08:45:231
491399100,9cyclictest27867-21sort09:05:001
491399100,9cyclictest22632-21taskset08:50:391
490899108,1cyclictest9381-21expr08:25:150
490899108,1cyclictest25175-21sed07:50:160
490899107,2cyclictest26336-21head12:20:160
490899107,2cyclictest24640-21grep09:45:120
490899107,2cyclictest14035-21expr08:35:150
490899107,2cyclictest13429-21taskset07:25:190
490899101,5cyclictest225-21systemd-journal07:40:000
490899100,9cyclictest16845-21seq07:32:300
49199996,2cyclictest26287-21cut09:00:163
49139997,1cyclictest16196-21grep07:30:211
49139996,2cyclictest24469-21head08:55:221
49089998,1cyclictest11409-21awk08:30:150
49089997,1cyclictest11620-21cat07:20:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional