You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-21 - 09:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu May 21, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
309921250,0sleep10-21swapper/122:17:001
26784210490,9sleep00-21swapper/019:08:100
2683429073,12sleep30-21swapper/319:08:493
2674628774,9sleep20-21swapper/219:07:402
2479128069,6sleep10-21swapper/119:05:061
15152590,0sleep20-21swapper/200:31:482
27014992826,2cyclictest0-21swapper/122:57:421
2701099289,10cyclictest6050-21munin-run22:00:000
2702099274,8cyclictest141rcu_preempt23:38:303
2701499270,26cyclictest0-21swapper/122:05:001
2701099270,1cyclictest0-21swapper/022:28:080
2701099270,1cyclictest0-21swapper/022:28:070
2701499261,24cyclictest19814-21diskmemload23:18:211
2701499258,14cyclictest19814-21diskmemload23:56:401
2701499251,21cyclictest0-21swapper/121:10:161
27010992522,2cyclictest13-21ksoftirqd/023:09:210
27010992519,2cyclictest13-21ksoftirqd/022:54:500
2701699240,23cyclictest0-21swapper/221:36:502
2701499245,19cyclictest19814-21diskmemload00:00:331
2701499240,21cyclictest0-21swapper/121:27:571
27010992422,1cyclictest13-21ksoftirqd/023:12:270
2702099238,2cyclictest141rcu_preempt00:32:293
2701699230,0cyclictest0-21swapper/200:17:322
2701499237,11cyclictest19814-21diskmemload22:35:071
2701499231,21cyclictest18655-21sshd20:00:101
2701499230,23cyclictest0-21swapper/119:20:101
2701499230,17cyclictest0-21swapper/123:40:151
2701099230,1cyclictest141rcu_preempt00:10:160
2702099220,21cyclictest27440-21kworker/3:100:29:333
2702099220,21cyclictest0-21swapper/321:40:153
2701499224,17cyclictest19814-21diskmemload22:05:211
27014992221,1cyclictest0-21swapper/123:10:131
2701499221,20cyclictest19814-21diskmemload22:21:291
2701499221,20cyclictest19814-21diskmemload22:21:291
2701499221,18cyclictest19814-21diskmemload00:33:281
2701499220,22cyclictest0-21swapper/123:54:061
2701499220,21cyclictest0-21swapper/121:49:061
27010992218,1cyclictest13-21ksoftirqd/000:08:490
2701099220,1cyclictest0-21swapper/021:47:290
310462210,0sleep30-21swapper/321:55:133
2702099210,20cyclictest14860-21kworker/3:023:47:433
27016992121,0cyclictest0-21swapper/223:56:072
2701699210,21cyclictest0-21swapper/222:35:302
2701499213,17cyclictest11149-21munin-run23:40:001
2701499212,17cyclictest0-21swapper/100:11:341
2701499211,19cyclictest6153-21latency_hist22:00:001
2701499211,19cyclictest19814-21diskmemload23:34:371
2701499211,19cyclictest19814-21diskmemload22:33:051
2701499211,19cyclictest0-21swapper/100:25:201
2701499210,20cyclictest0-21swapper/100:24:201
2701499210,18cyclictest5617-21/usr/sbin/munin21:40:161
27010992120,1cyclictest13-21ksoftirqd/000:24:440
2701099211,1cyclictest0-21swapper/022:06:230
2701099210,2cyclictest0-21swapper/021:10:410
2701699201,5cyclictest141rcu_preempt21:44:112
2701499207,12cyclictest19814-21diskmemload00:35:021
2701499203,10cyclictest19814-21diskmemload23:02:081
27014992012,7cyclictest19814-21diskmemload00:05:271
2701499201,18cyclictest18897-21ssh22:45:261
2701499200,19cyclictest0-21swapper/120:05:161
2701499200,18cyclictest0-21swapper/122:53:561
2701499200,18cyclictest0-21swapper/100:18:551
2701099203,3cyclictest141rcu_preempt22:45:120
27010992018,2cyclictest13-21ksoftirqd/022:00:570
27010992014,5cyclictest13-21ksoftirqd/023:30:520
2701099200,1cyclictest0-21swapper/022:55:590
2702099195,10cyclictest1235-21ntpd22:25:313
2702099195,10cyclictest1235-21ntpd22:25:313
2702099190,19cyclictest25981-21kworker/3:222:07:233
2701699190,18cyclictest0-21swapper/221:47:572
2701499193,14cyclictest19814-21diskmemload23:49:301
27014991916,3cyclictest0-21swapper/123:27:031
2701499190,18cyclictest0-21swapper/121:50:151
2701499190,17cyclictest0-21swapper/121:24:381
2701499190,17cyclictest0-21swapper/119:30:121
2701099196,2cyclictest141rcu_preempt22:41:550
2701099190,1cyclictest0-21swapper/022:15:320
27020991817,1cyclictest0-21swapper/321:31:053
27020991812,1cyclictest0-21swapper/300:05:283
2701699181,3cyclictest141rcu_preempt23:39:072
27016991810,5cyclictest452-21systemd-logind00:12:042
2701699180,5cyclictest31100-21taskset23:12:332
2701699180,17cyclictest0-21swapper/221:58:472
2701499181,14cyclictest19814-21diskmemload21:38:531
2701499180,4cyclictest0-21swapper/122:10:141
2701499180,17cyclictest0-21swapper/123:08:451
2701499180,16cyclictest0-21swapper/120:45:161
2701099182,3cyclictest13-21ksoftirqd/023:43:360
27010991815,2cyclictest13-21ksoftirqd/019:35:000
27010991813,5cyclictest13-21ksoftirqd/020:45:130
27010991812,1cyclictest13-21ksoftirqd/021:40:310
2701099180,3cyclictest13-21ksoftirqd/000:18:410
2701699170,1cyclictest0-21swapper/223:28:252
2701499176,10cyclictest16429-21taskset19:55:141
2701499174,12cyclictest19814-21diskmemload22:40:511
2701499172,14cyclictest19814-21diskmemload22:26:581
2701499172,14cyclictest19814-21diskmemload22:26:571
2701499171,15cyclictest0-21swapper/121:31:491
2701499170,17cyclictest0-21swapper/121:15:261
2701499170,16cyclictest17659-21sshd21:05:101
2701499170,13cyclictest0-21swapper/123:24:181
2701499170,13cyclictest0-21swapper/120:10:131
27010991716,1cyclictest161rcuc/023:26:100
27010991714,3cyclictest13-21ksoftirqd/023:35:160
2701099170,17cyclictest0-21swapper/000:31:010
2701099170,14cyclictest0-21swapper/000:00:130
2702099160,16cyclictest0-21swapper/323:04:373
27016991614,1cyclictest0-21swapper/200:09:212
2701499160,10cyclictest0-21swapper/119:45:121
27010991613,2cyclictest19814-21diskmemload21:27:580
27010991613,2cyclictest0-21swapper/021:22:370
2701099160,7cyclictest13-21ksoftirqd/021:36:010
2702099153,8cyclictest16298-21apt-get19:55:143
27020991514,1cyclictest25981-21kworker/3:222:20:433
27020991514,1cyclictest25981-21kworker/3:222:20:433
27020991513,1cyclictest25981-21kworker/3:221:19:233
2702099150,1cyclictest0-21swapper/322:15:233
2701699150,1cyclictest0-21swapper/222:10:122
2701499153,11cyclictest11018-21fschecks_time20:50:141
2701499151,13cyclictest0-21swapper/120:45:001
2701499151,10cyclictest27520-21tr19:10:101
2701499150,14cyclictest0-21swapper/121:00:001
2701499150,13cyclictest0-21swapper/120:35:131
2701099152,1cyclictest141rcu_preempt23:53:450
2701099150,4cyclictest13-21ksoftirqd/022:37:420
2701099150,1cyclictest0-21swapper/019:49:510
2701099150,0cyclictest0-21swapper/021:19:180
27020991414,0cyclictest0-21swapper/323:10:503
27020991413,1cyclictest25981-21kworker/3:221:21:243
27016991412,1cyclictest0-21swapper/223:19:542
27016991411,2cyclictest14427-21awk22:05:002
27016991410,3cyclictest28340-21grep19:10:162
27016991410,3cyclictest1304-21grep22:55:122
2701699140,7cyclictest0-21swapper/222:24:262
2701699140,7cyclictest0-21swapper/222:24:262
2701499148,5cyclictest32575-21taskset20:28:311
2701499147,6cyclictest8210-21cat19:35:261
2701499147,6cyclictest1499-21grep20:30:191
27014991410,3cyclictest1980-21apt-get19:25:131
2701499140,13cyclictest0-21swapper/121:00:141
2701099148,2cyclictest13-21ksoftirqd/021:54:370
2701099147,6cyclictest0-21swapper/023:56:410
2701099146,7cyclictest10397-21apache220:20:140
27010991410,3cyclictest12649-21grep23:20:250
27010991410,3cyclictest10518-21tail23:00:180
27010991410,1cyclictest225-21systemd-journal20:00:100
2701099140,12cyclictest23557-21head21:30:320
2702099139,3cyclictest14864-21cut19:50:173
2702099139,2cyclictest1558-21grep22:55:133
2702099135,3cyclictest0-21swapper/322:35:213
27020991311,1cyclictest39-21ksoftirqd/322:11:073
27020991310,2cyclictest6314-21grep20:40:243
27020991310,2cyclictest5581-21grep00:15:143
27020991310,2cyclictest3858-21grep20:35:163
27020991310,2cyclictest30336-21grep19:15:153
27020991310,2cyclictest30329-21dump-pmu-power20:25:003
27020991310,2cyclictest23973-21wc21:50:243
27020991310,2cyclictest15021-21wc21:00:013
27020991310,2cyclictest0-21swapper/300:10:103
2702099130,2cyclictest0-21swapper/300:00:153
2702099130,12cyclictest30210-21grep23:50:273
2701699139,3cyclictest5965-21grep19:30:232
2701699139,3cyclictest19431-21cron23:45:002
2701699134,7cyclictest31757-21runrttasks22:34:462
2701699131,2cyclictest0-21swapper/222:45:152
27016991310,2cyclictest17476-21ls23:04:592
27016991310,2cyclictest15383-21grep21:00:152
27016991310,2cyclictest14361-21grep20:55:262
27016991310,2cyclictest10929-21grep20:50:142
2701699130,11cyclictest0-21swapper/200:25:172
2701499132,9cyclictest28227-21apt-get20:20:131
27014991311,2cyclictest0-21swapper/120:15:151
2701099139,4cyclictest26792-21kworker/u8:200:26:210
2701099139,3cyclictest7220-21grep20:40:320
2701099139,3cyclictest24734-21fgrep22:10:210
27010991310,2cyclictest13811-21grep20:55:190
27010991310,2cyclictest1301-21head19:20:260
2701099130,0cyclictest0-21swapper/000:35:320
2702099129,2cyclictest6950-21ssh00:35:173
2702099129,2cyclictest2832-21taskset20:35:013
2702099129,2cyclictest28132-21sh21:15:003
2702099129,2cyclictest2726-21head19:25:163
2702099129,2cyclictest25036-21head22:30:163
2702099129,2cyclictest20786-21cron20:05:003
2702099129,2cyclictest17412-21cat22:45:153
27020991211,1cyclictest39-21ksoftirqd/323:07:533
27020991211,1cyclictest25981-21kworker/3:222:40:533
27020991211,1cyclictest25981-21kworker/3:221:35:243
27020991211,1cyclictest0-21swapper/323:34:403
2702099120,9cyclictest0-21swapper/323:27:273
2702099120,10cyclictest4339-21aten2_r7power_p23:55:143
2701699129,2cyclictest28831-21cut21:15:152
2701699129,2cyclictest2702-21head19:25:162
2701699129,2cyclictest22055-21cat20:05:212
2701699129,2cyclictest18679-21cat21:05:192
2701699129,2cyclictest15174-21tr22:05:142
2701699129,2cyclictest14670-21grep19:50:152
2701699126,5cyclictest21419-21cat21:10:202
2701699125,4cyclictest0-21swapper/222:25:142
2701699125,4cyclictest0-21swapper/222:25:142
2701699124,6cyclictest32-21ksoftirqd/220:10:132
2701699122,8cyclictest20381-21awk00:05:002
27016991211,1cyclictest0-21swapper/221:30:112
2701699120,11cyclictest19861-21sed23:05:252
2701499129,2cyclictest29890-21tr19:15:131
2701499126,5cyclictest15125-21cut19:50:191
2701099129,2cyclictest8342-21grep19:35:260
2701099128,3cyclictest2412-21cut23:15:160
2701099128,3cyclictest15810-21head21:00:170
2701099122,6cyclictest21463-21find23:45:230
27010991210,2cyclictest13-21ksoftirqd/020:40:000
2702099118,2cyclictest9653-21ssh22:01:183
2702099118,2cyclictest7593-21ssh23:18:003
2702099118,2cyclictest7395-21grep19:35:183
2702099118,2cyclictest26594-21grep20:15:173
2702099118,2cyclictest25609-21tr22:50:133
2702099118,2cyclictest19392-21grep21:05:263
2702099118,2cyclictest17895-21ssh21:46:543
2702099118,2cyclictest15437-21expr21:00:153
2702099118,2cyclictest15269-21ssh21:25:323
2702099118,2cyclictest14374-21head00:20:163
2702099118,2cyclictest14036-21ssh23:21:283
2702099118,2cyclictest11535-21ssh23:40:083
2702099117,3cyclictest9509-21grep20:45:253
2702099110,10cyclictest11807-21wc20:50:233
2701699119,1cyclictest24903-21taskset23:47:462
2701699118,2cyclictest9922-21gzip22:40:212
2701699118,2cyclictest9882-21ssh00:36:532
2701699118,2cyclictest967-21expr20:30:152
2701699118,2cyclictest7157-21expr19:35:152
2701699118,2cyclictest6852-21ssh21:20:292
2701699118,2cyclictest5778-21grep20:40:212
2701699118,2cyclictest3861-21head20:35:162
2701699118,2cyclictest27297-21rm23:30:102
2701699118,2cyclictest26570-21head20:15:162
2701699118,2cyclictest24859-21sh21:50:522
2701699118,2cyclictest19473-21sed20:00:172
2701699118,2cyclictest18645-21ssh21:28:112
2701699118,2cyclictest18360-21taskset19:59:382
2701699118,2cyclictest13708-21aten2_r7power_v00:20:142
2701699118,2cyclictest12946-21chrt23:20:362
2701699110,1cyclictest0-21swapper/223:53:192
2701699110,10cyclictest32496-21ssh22:54:282
2701699110,10cyclictest2586-21sh22:16:362
2701499110,10cyclictest0-21swapper/119:40:141
2701099118,2cyclictest3196-21head19:25:220
2701099118,2cyclictest31725-21cron19:19:590
2701099118,2cyclictest19814-21diskmemload22:20:060
2701099118,2cyclictest19814-21diskmemload22:20:060
2701099118,2cyclictest18397-21grep21:05:170
2701099118,2cyclictest17616-21wc19:55:230
2701099118,2cyclictest13-21ksoftirqd/020:20:000
2701099118,2cyclictest12368-21chrt20:54:110
2701099117,3cyclictest11388-21wc19:45:010
2701099111,5cyclictest13-21ksoftirqd/020:30:250
2701099110,7cyclictest31719-21ntp_kernel_err20:25:190
2701099110,7cyclictest24923-21fschecks_count22:30:150
2702099108,1cyclictest90792sleep319:40:103
2702099108,1cyclictest5140-21sh19:30:153
2702099108,1cyclictest291352sleep319:12:223
2702099108,1cyclictest229102sleep320:06:463
2702099108,1cyclictest13672sleep319:20:473
2702099108,1cyclictest0-21swapper/320:10:163
2702099107,2cyclictest31519-21uname20:25:183
2701699108,1cyclictest8681-21expr20:45:162
2701699108,1cyclictest31059-21expr20:25:142
2701699108,1cyclictest10753-21cat19:40:262
2701699107,2cyclictest32019-21apache_accesses19:20:122
2701699107,2cyclictest11886-21expr19:45:132
2701699103,3cyclictest225-21systemd-journal19:17:102
2701099108,1cyclictest22225-21ntp_states20:05:240
2701099108,1cyclictest16214-21cat19:55:000
2701099100,7cyclictest161rcuc/020:10:160
270209997,1cyclictest12211-21tune2fs19:45:153
270169997,1cyclictest28866-21tune2fs20:20:172
270109997,1cyclictest28254-21load19:10:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional