You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-27 - 11:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed May 27, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1658529077,9sleep00-21swapper/019:06:330
1658728875,9sleep20-21swapper/219:06:352
1601528461,19sleep10-21swapper/119:05:261
1471228269,9sleep30-21swapper/319:05:033
212042490,0sleep00-21swapper/022:14:300
1693599290,28cyclictest0-21swapper/222:58:012
16939992826,1cyclictest381rcuc/321:15:143
1693099272,21cyclictest0-21swapper/123:00:001
1692599270,1cyclictest0-21swapper/022:35:390
1693099260,2cyclictest0-21swapper/100:15:121
1692599260,1cyclictest0-21swapper/022:58:010
1693599250,24cyclictest0-21swapper/221:27:362
1693599250,20cyclictest0-21swapper/222:44:102
1693599250,20cyclictest0-21swapper/222:44:102
1693099251,23cyclictest3968-21open_inodes20:55:231
1693999240,17cyclictest0-21swapper/320:20:193
1693099240,1cyclictest9745-21diskmemload23:54:501
16939992319,4cyclictest14378-21kworker/u8:020:46:183
1693599230,22cyclictest0-21swapper/200:12:112
1693999220,1cyclictest0-21swapper/323:09:253
1693599220,1cyclictest0-21swapper/200:35:162
16925992216,4cyclictest0-21swapper/023:17:570
16939992119,1cyclictest9745-21diskmemload00:25:173
16939992117,4cyclictest39-21ksoftirqd/322:27:003
1693599211,19cyclictest0-21swapper/222:08:172
1693599210,21cyclictest0-21swapper/200:24:372
1693599210,20cyclictest0-21swapper/223:37:282
1692599212,15cyclictest0-21swapper/000:37:510
16939992019,1cyclictest9745-21diskmemload00:19:563
16935992018,1cyclictest0-21swapper/220:30:202
1693599200,3cyclictest0-21swapper/221:40:152
1693599200,17cyclictest0-21swapper/200:28:032
1692599200,19cyclictest0-21swapper/023:34:470
16939991919,0cyclictest0-21swapper/300:12:103
16939991918,1cyclictest39-21ksoftirqd/323:23:263
16939991910,6cyclictest39-21ksoftirqd/323:26:203
1693999190,1cyclictest0-21swapper/321:51:553
1693099190,2cyclictest0-21swapper/123:19:491
16925991917,2cyclictest13-21ksoftirqd/023:52:450
16939991815,1cyclictest9745-21diskmemload23:34:393
16939991814,3cyclictest39-21ksoftirqd/300:20:153
16939991814,3cyclictest2527-21ls21:25:003
1693599180,17cyclictest0-21swapper/200:03:422
1693099182,14cyclictest9745-21diskmemload22:14:191
16930991816,1cyclictest335-21/usr/sbin/munin20:50:231
16930991816,1cyclictest335-21/usr/sbin/munin20:50:231
16930991816,1cyclictest0-21swapper/120:00:141
16930991815,2cyclictest23388-21sed22:35:001
1693099180,2cyclictest0-21swapper/123:46:091
1693099180,2cyclictest0-21swapper/121:41:291
1693099180,1cyclictest0-21swapper/100:36:401
1692599187,5cyclictest13-21ksoftirqd/023:46:520
1692599180,18cyclictest0-21swapper/021:23:500
16939991714,1cyclictest9745-21diskmemload22:22:133
16939991712,5cyclictest39-21ksoftirqd/323:59:183
16935991715,2cyclictest32-21ksoftirqd/223:32:202
1693599170,13cyclictest0-21swapper/221:11:452
1693099171,15cyclictest2647-21head23:00:221
1693099171,0cyclictest0-21swapper/100:28:531
1693099170,2cyclictest0-21swapper/122:21:031
1693099170,1cyclictest0-21swapper/123:44:321
1692599170,0cyclictest0-21swapper/022:44:100
1692599170,0cyclictest0-21swapper/022:44:100
1693999161,5cyclictest39-21ksoftirqd/321:33:513
16939991614,1cyclictest9745-21diskmemload23:44:143
16939991611,4cyclictest26191-21rm21:57:503
16935991614,2cyclictest32-21ksoftirqd/221:52:462
16935991614,1cyclictest21067-21apache_processe21:55:112
1693599160,16cyclictest0-21swapper/222:30:102
1693599160,16cyclictest0-21swapper/222:26:082
1693599160,15cyclictest0-21swapper/222:20:522
1693599160,12cyclictest0-21swapper/221:38:522
16930991615,1cyclictest0-21swapper/121:12:351
1693099160,2cyclictest0-21swapper/122:05:181
1693099160,1cyclictest0-21swapper/121:46:351
1693099160,15cyclictest0-21swapper/123:35:261
203742150,0chrt0-21swapper/200:08:562
16939991513,1cyclictest9745-21diskmemload21:39:223
16939991512,3cyclictest39-21ksoftirqd/300:30:183
16939991511,3cyclictest28755-21sort23:35:173
16939991510,4cyclictest9745-21diskmemload00:02:293
1693999150,2cyclictest0-21swapper/321:45:123
1693599157,5cyclictest25136-21apt-get23:52:102
1693599150,14cyclictest0-21swapper/200:31:222
1693599150,12cyclictest0-21swapper/223:13:022
1692599154,6cyclictest18311-21awk20:20:010
1692599153,6cyclictest17691-21awk23:10:000
16925991512,1cyclictest0-21swapper/021:38:530
1692599151,13cyclictest7095-21sensors22:05:220
1692599150,4cyclictest141rcu_preempt23:13:350
1693999148,5cyclictest9745-21diskmemload21:44:053
1693999141,2cyclictest39-21ksoftirqd/322:08:213
16939991412,1cyclictest7565-21apt-get21:05:153
1693999140,6cyclictest0-21swapper/323:52:263
1693999140,2cyclictest141rcu_preempt22:48:143
1693599142,5cyclictest14362-21cstates22:10:142
16935991411,1cyclictest0-21swapper/223:20:122
1693599140,8cyclictest0-21swapper/223:00:462
1693599140,1cyclictest0-21swapper/222:53:002
1693599140,13cyclictest0-21swapper/223:57:432
1693599140,13cyclictest0-21swapper/200:18:542
1693599140,12cyclictest0-21swapper/221:32:372
16930991412,1cyclictest0-21swapper/121:57:501
1693099141,12cyclictest1077-21ntp_offset22:40:191
1693099141,12cyclictest1077-21ntp_offset22:40:191
1693099140,6cyclictest0-21swapper/122:15:461
1693099140,3cyclictest0-21swapper/123:32:251
1693099140,11cyclictest14317-21sensors21:50:251
1692599144,0cyclictest0-21swapper/020:40:130
1693999139,3cyclictest17264-21grep20:15:203
1693999136,6cyclictest32314-21aten2_r7power_p22:40:143
1693999136,6cyclictest32314-21aten2_r7power_p22:40:133
1693999135,7cyclictest0-21swapper/322:50:273
1693999134,8cyclictest0-21swapper/321:25:073
1693999134,4cyclictest225-21systemd-journal19:32:173
16939991313,0cyclictest19036-21kworker/u8:222:36:273
16939991312,1cyclictest9745-21diskmemload23:48:253
16939991311,2cyclictest39-21ksoftirqd/300:35:263
16939991311,2cyclictest0-21swapper/323:18:163
16939991310,3cyclictest39-21ksoftirqd/322:55:443
16939991310,2cyclictest3100-21grep19:45:233
16939991310,2cyclictest10393-21grep21:10:143
1693999130,12cyclictest6868-21sed21:00:273
1693599135,6cyclictest0-21swapper/223:18:032
1693599133,4cyclictest0-21swapper/221:05:132
1693599131,1cyclictest0-21swapper/223:44:512
1693599130,7cyclictest0-21swapper/222:19:052
1693099139,2cyclictest9178-21ssh22:45:171
16930991311,2cyclictest0-21swapper/123:23:481
16930991311,1cyclictest2508-21cron21:25:001
16930991310,2cyclictest30264-21wc20:45:001
16930991310,2cyclictest24554-21grep19:25:141
16930991310,2cyclictest14524-21grep20:10:171
1692599138,2cyclictest31772-21wc19:40:140
1692599137,3cyclictest13-21ksoftirqd/022:49:580
1692599136,7cyclictest0-21swapper/000:21:580
1692599132,6cyclictest5187-21apt-get21:00:130
16925991312,1cyclictest0-21swapper/021:11:450
16925991310,2cyclictest13-21ksoftirqd/000:28:230
1692599130,1cyclictest0-21swapper/021:56:000
162162130,0sleep20-21swapper/223:08:532
1693999129,2cyclictest9551-21head20:00:183
1693999129,2cyclictest381rcuc/319:20:223
1693999129,2cyclictest29885-21expr22:00:153
1693999129,2cyclictest27835-21sleep20:40:013
1693999129,2cyclictest21210-21grep19:15:253
1693999129,2cyclictest20174-21grep23:10:273
1693999128,3cyclictest1514-21tr23:00:133
1693999128,2cyclictest6662-21grep19:55:133
1693999124,5cyclictest0-21swapper/322:15:083
1693999122,8cyclictest0-21swapper/322:31:583
1693599129,2cyclictest31208-21rm21:22:062
1693599129,2cyclictest29569-21wc20:40:252
1693599129,2cyclictest22015-21cut19:20:142
1693599129,2cyclictest16208-21ssh23:27:452
1693599129,2cyclictest14366-21ssh22:48:382
1693599129,2cyclictest1323-21sed19:45:002
1693599125,3cyclictest9745-21diskmemload21:17:102
1693599121,10cyclictest8800-21apache222:36:182
1693599121,10cyclictest12483-21apache_volume23:45:132
1693599120,11cyclictest31464-21grep22:00:282
1693099129,2cyclictest7850-21tr21:05:151
1693099129,2cyclictest3382-21expr21:25:141
1693099129,2cyclictest26964-21tr19:30:131
1693099129,2cyclictest20689-21munin-plugin-st20:25:001
1693099129,2cyclictest20173-21cut21:35:151
1693099129,2cyclictest19508-21sed21:15:171
1693099129,2cyclictest19437-21grep23:10:211
1693099129,2cyclictest14746-21basename00:25:001
1693099129,2cyclictest10062-21tr23:05:151
1693099128,3cyclictest506-21ssh23:56:411
1693099128,3cyclictest31697-21grep22:40:001
1693099127,4cyclictest351-21ssh22:01:261
1693099127,2cyclictest15579-21ls00:05:281
16930991211,1cyclictest0-21swapper/100:31:361
1693099120,1cyclictest11449-21ssh00:03:321
1692599129,2cyclictest8877-21wc21:05:220
1692599129,2cyclictest5604-21ssh00:00:160
1692599129,2cyclictest4873-21grep23:40:160
1692599129,2cyclictest4856-21sleep00:00:000
1692599129,2cyclictest3549-21df21:25:150
1692599129,2cyclictest32405-21find00:15:230
1692599129,2cyclictest30599-21grep20:45:140
1692599129,2cyclictest30588-21ssh21:40:420
1692599129,2cyclictest2334-21ssh23:39:050
1692599129,2cyclictest21450-21ssh00:09:410
1692599129,2cyclictest20072-21ssh22:32:290
1692599129,2cyclictest11698-21expr23:25:160
1692599127,4cyclictest9745-21diskmemload00:30:390
1692599123,2cyclictest0-21swapper/022:00:140
1692599122,8cyclictest4758-21/usr/sbin/munin23:20:260
1692599122,8cyclictest26004-21cstates20:35:150
1692599122,10cyclictest28225-21kworker/u8:322:15:120
16925991210,2cyclictest0-21swapper/023:02:460
1692599120,5cyclictest0-21swapper/022:26:290
1692599120,1cyclictest0-21swapper/022:20:150
1692599120,11cyclictest11203-21apache_accesses20:05:120
1693999119,1cyclictest20950-21apt-get20:25:133
1693999119,1cyclictest174502sleep300:06:413
1693999118,3cyclictest0-21swapper/322:10:023
1693999118,2cyclictest39-21ksoftirqd/319:25:143
1693999118,2cyclictest28859-21head20:40:163
1693999118,2cyclictest15148-21sendmail_mailtr20:10:233
1693999118,2cyclictest1182-21head20:50:173
1693999118,2cyclictest1182-21head20:50:173
1693599119,2cyclictest32437-21chrt20:48:282
1693599119,2cyclictest1065-21grep20:50:172
1693599119,2cyclictest1065-21grep20:50:172
1693599119,1cyclictest8481-21apt20:00:002
1693599119,1cyclictest5032-21sed21:45:162
1693599119,1cyclictest3779-21wc20:55:202
1693599119,1cyclictest30923-21taskset19:36:562
1693599119,1cyclictest27384-21grep20:35:282
1693599119,1cyclictest26975-21tr19:30:132
1693599119,1cyclictest18691-21cut20:20:152
1693599118,2cyclictest21493-21grep20:25:162
1693599118,2cyclictest20295-21sed19:15:162
1693599118,2cyclictest17251-21basename19:10:142
1693599114,2cyclictest225-21systemd-journal19:48:082
1693099118,2cyclictest6644-21grep19:55:131
1693099118,2cyclictest32059-21sort19:40:151
1693099118,2cyclictest25798-21grep20:35:141
1693099118,2cyclictest22788-21ls00:10:161
1693099118,2cyclictest1840-21expr19:45:121
1693099118,2cyclictest16481-21ssh22:50:061
1693099118,2cyclictest14173-21sh23:26:111
1693099117,3cyclictest32185-21chrt20:45:271
1693099117,3cyclictest25433-21grep20:35:001
1693099117,3cyclictest12348-21sed21:30:191
1693099116,4cyclictest12651-21ssh22:28:141
1692599119,2cyclictest4657-21taskset20:57:050
1692599119,1cyclictest9290-21ls20:00:150
1692599119,1cyclictest6063-21grep21:45:250
1692599119,1cyclictest30776-21sed19:35:260
1692599119,1cyclictest22813-21expr00:10:150
1692599119,1cyclictest19849-21cat19:15:140
1692599119,1cyclictest18576-21ssh21:15:100
1692599119,1cyclictest12381-21ssh21:50:000
1692599119,1cyclictest11208-21ssh21:30:010
1692599118,2cyclictest25414-21cat19:25:200
1692599118,2cyclictest22076-21hostname19:20:140
1692599118,2cyclictest20679-21sh20:25:000
1692599118,2cyclictest16476-21ssh22:50:070
1693999108,1cyclictest48252sleep320:59:153
1693999108,1cyclictest23384-21cut20:30:143
1693999108,1cyclictest12996-21chrt20:06:063
1693999107,2cyclictest32224-21head19:40:163
1693999107,2cyclictest29501-21expr19:35:163
1693999103,5cyclictest39-21ksoftirqd/319:11:383
1693999102,4cyclictest5663-21sshd19:52:083
1693599109,1cyclictest70752sleep221:02:192
1693599109,1cyclictest4336-21sed19:50:162
1693599109,1cyclictest24924-21grep19:25:152
1693599108,1cyclictest14618-21sort20:10:182
1693599108,1cyclictest11522-21expr20:05:152
1693599108,1cyclictest11117-21cat20:05:012
1693599107,2cyclictest18169-21chrt20:18:392
1693099108,1cyclictest20991-21cut20:25:151
1693099107,1cyclictest19012-21taskset19:11:441
1693099105,2cyclictest225-21systemd-journal19:35:171
1693099100,9cyclictest7110-21taskset21:02:471
1693099100,9cyclictest22034-21grep19:20:141
1693099100,9cyclictest19510-21chrt19:15:071
1692599109,1cyclictest3275-21taskset19:46:580
1692599109,1cyclictest22868-21chrt20:27:470
1692599109,1cyclictest15791-21chrt20:13:440
1692599108,1cyclictest27134-21ls19:30:140
1692599108,1cyclictest23557-21grep20:30:150
1692599107,2cyclictest5433-21tail19:50:260
1692599107,2cyclictest1027-21grep20:50:170
1692599107,2cyclictest1027-21grep20:50:170
1692599103,5cyclictest6378-21apt-get19:55:110
169309997,1cyclictest16839-21cut20:15:171
169309996,2cyclictest11978-21tail20:05:171
169309992,5cyclictest0-21swapper/119:50:151
169259997,1cyclictest17178-21aten2_r7power_c19:10:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional