You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-18 - 03:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon May 18, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
553029785,8sleep00-21swapper/019:07:420
561629561,30sleep20-21swapper/219:08:492
357929566,10sleep30-21swapper/319:05:083
544428972,13sleep10-21swapper/119:06:341
579499278,2cyclictest31071-21ssh22:07:461
579099245,3cyclictest141rcu_preempt21:25:140
5790992422,1cyclictest13-21ksoftirqd/000:09:270
579899230,23cyclictest0-21swapper/222:38:052
579499233,17cyclictest8073-21df21:15:191
5790992312,1cyclictest13-21ksoftirqd/023:00:430
580099220,20cyclictest0-21swapper/321:05:153
579899220,21cyclictest0-21swapper/223:20:042
579099220,22cyclictest0-21swapper/021:16:540
580099210,3cyclictest0-21swapper/322:59:263
579499212,19cyclictest0-21swapper/122:45:201
579499210,19cyclictest0-21swapper/123:06:231
5790992116,2cyclictest13-21ksoftirqd/022:15:190
5790992114,4cyclictest13-21ksoftirqd/023:45:180
580099200,19cyclictest0-21swapper/300:23:583
580099200,19cyclictest0-21swapper/300:23:583
579499203,15cyclictest30955-21diskmemload00:12:051
579099202,4cyclictest141rcu_preempt22:25:140
5790992014,6cyclictest13-21ksoftirqd/023:30:220
579099200,2cyclictest0-21swapper/019:45:150
579099200,2cyclictest0-21swapper/000:18:080
580099191,2cyclictest17059-21kworker/u8:023:35:543
580099190,1cyclictest141rcu_preempt23:03:483
5798991918,1cyclictest0-21swapper/220:35:142
5798991918,1cyclictest0-21swapper/200:20:522
5798991918,1cyclictest0-21swapper/200:20:522
5798991917,2cyclictest0-21swapper/200:28:112
579899190,18cyclictest0-21swapper/223:00:422
579499190,2cyclictest0-21swapper/120:40:141
579099192,3cyclictest141rcu_preempt00:35:340
5790991918,1cyclictest0-21swapper/021:21:460
5790991915,2cyclictest13-21ksoftirqd/023:27:560
5790991910,4cyclictest13-21ksoftirqd/000:22:020
5790991910,4cyclictest13-21ksoftirqd/000:22:010
5800991817,1cyclictest25419-21kworker/3:123:51:333
579899184,3cyclictest141rcu_preempt00:08:402
579899180,18cyclictest0-21swapper/222:30:592
579499186,7cyclictest1-21systemd19:20:551
579499183,14cyclictest30955-21diskmemload22:42:181
5794991814,3cyclictest30955-21diskmemload21:31:371
579499180,16cyclictest0-21swapper/121:10:531
579099189,1cyclictest141rcu_preempt23:16:540
5790991818,0cyclictest0-21swapper/022:40:010
5790991816,2cyclictest13-21ksoftirqd/022:09:410
5790991814,3cyclictest13-21ksoftirqd/022:38:120
5790991811,6cyclictest13-21ksoftirqd/020:00:120
580099174,5cyclictest30955-21diskmemload21:29:443
580099174,11cyclictest24819-21/usr/sbin/munin19:50:003
580099170,1cyclictest0-21swapper/322:26:393
580099170,12cyclictest0-21swapper/321:33:223
579899170,17cyclictest0-21swapper/223:47:382
579899170,17cyclictest0-21swapper/221:49:172
579899170,17cyclictest0-21swapper/221:49:172
579899170,17cyclictest0-21swapper/221:12:482
579899170,16cyclictest0-21swapper/222:25:492
579499179,4cyclictest38150irq/131-snd_hda22:00:141
5794991716,1cyclictest0-21swapper/100:23:581
5794991716,1cyclictest0-21swapper/100:23:581
5794991715,1cyclictest0-21swapper/121:40:491
579499170,0cyclictest0-21swapper/123:16:591
579099178,3cyclictest13-21ksoftirqd/021:51:500
579099178,2cyclictest141rcu_preempt23:10:210
579099172,2cyclictest141rcu_preempt21:56:560
5790991716,1cyclictest13-21ksoftirqd/000:30:250
5790991715,2cyclictest13-21ksoftirqd/023:00:000
5790991715,2cyclictest13-21ksoftirqd/021:38:370
5790991714,3cyclictest13-21ksoftirqd/023:52:500
5790991710,7cyclictest13-21ksoftirqd/023:20:140
579099170,2cyclictest141rcu_preempt23:39:270
579099170,1cyclictest0-21swapper/020:40:180
5800991615,1cyclictest39-21ksoftirqd/323:49:053
5800991615,1cyclictest25419-21kworker/3:100:00:033
5800991612,3cyclictest55650irq/127-enp2s0-00:08:303
579899161,12cyclictest23398-21sendmail00:00:002
579899160,16cyclictest0-21swapper/222:03:282
579899160,15cyclictest0-21swapper/223:53:272
5794991615,1cyclictest0-21swapper/122:39:081
579499160,14cyclictest0-21swapper/122:13:291
579099162,1cyclictest141rcu_preempt20:30:270
5790991615,1cyclictest161rcuc/023:07:200
5790991611,1cyclictest13-21ksoftirqd/000:00:240
5800991514,1cyclictest14843-21kworker/3:221:12:333
5800991514,1cyclictest0-21swapper/321:39:393
5800991514,1cyclictest0-21swapper/300:27:423
5800991513,1cyclictest25195-21ssh21:45:143
5800991513,1cyclictest25195-21ssh21:45:133
580099150,14cyclictest0-21swapper/320:20:163
579899150,0cyclictest0-21swapper/223:31:442
579499159,5cyclictest21415-21cron20:50:001
5794991515,0cyclictest0-21swapper/121:20:241
579099156,5cyclictest32484-21apache_accesses20:05:120
579099155,5cyclictest13-21ksoftirqd/022:01:290
579099155,2cyclictest13-21ksoftirqd/022:50:510
5790991511,3cyclictest13-21ksoftirqd/019:15:000
5790991511,2cyclictest17120-21wc19:30:240
579099150,1cyclictest0-21swapper/022:22:550
580099147,7cyclictest0-21swapper/322:38:493
580099143,3cyclictest0-21swapper/323:42:473
579899142,11cyclictest19918-21apache_accesses22:40:122
5798991413,1cyclictest0-21swapper/223:07:202
579899140,14cyclictest0-21swapper/221:56:582
579499148,2cyclictest0-21swapper/123:00:201
579499146,6cyclictest18074-21kworker/1:223:20:441
579499146,5cyclictest452-21systemd-logind22:24:041
579499146,1cyclictest0-21swapper/119:25:131
5794991410,4cyclictest0-21swapper/122:25:201
579099144,9cyclictest30955-21diskmemload00:25:230
579099143,3cyclictest141rcu_preempt22:13:110
579099142,1cyclictest0-21swapper/022:34:190
5790991412,2cyclictest13-21ksoftirqd/023:58:130
5790991412,2cyclictest13-21ksoftirqd/022:48:480
580099132,9cyclictest13088-21sh21:18:183
579899139,4cyclictest31847-21apache223:16:022
579899138,4cyclictest0-21swapper/223:13:332
5798991312,1cyclictest0-21swapper/221:43:392
5798991310,2cyclictest9113-21grep20:20:292
5798991310,2cyclictest30801-21cron23:45:002
5798991310,2cyclictest20073-21grep20:45:182
5798991310,2cyclictest10036-21sed19:15:282
579899130,4cyclictest0-21swapper/200:10:172
579899130,1cyclictest0-21swapper/221:28:082
579899130,13cyclictest0-21swapper/221:15:522
579499139,4cyclictest10493-21taskset19:19:521
579499136,5cyclictest30955-21diskmemload22:50:151
579499135,7cyclictest38150irq/131-snd_hda23:39:481
5794991312,1cyclictest30955-21diskmemload00:33:271
5794991312,1cyclictest0-21swapper/123:10:141
579499130,5cyclictest19140-21ssh23:56:391
579499130,1cyclictest0-21swapper/121:05:151
579499130,11cyclictest0-21swapper/100:38:531
579099139,3cyclictest26158-21ssh21:45:190
579099139,3cyclictest26158-21ssh21:45:190
579099139,3cyclictest19495-21wc19:35:260
579099132,4cyclictest6663-21kworker/0:100:14:340
5790991311,2cyclictest13-21ksoftirqd/021:30:190
5790991310,2cyclictest6948-21ls21:15:000
5790991310,2cyclictest13-21ksoftirqd/020:55:130
579099130,12cyclictest23670-21ssh23:40:160
579099130,12cyclictest17925-21wc21:40:190
580099129,2cyclictest3693-21wc20:10:213
580099129,2cyclictest31677-21sed20:00:263
580099129,2cyclictest29401-21wc22:45:203
580099129,2cyclictest27502-21wc22:05:223
580099129,2cyclictest19918-21cut20:45:163
580099129,2cyclictest19119-21grep19:35:213
580099129,2cyclictest16799-21ssh22:18:473
580099129,2cyclictest16431-21rm23:16:033
580099129,2cyclictest15798-21tr19:30:143
580099129,2cyclictest13931-21chrt00:32:143
580099128,3cyclictest7038-21date20:20:013
580099128,3cyclictest31445-21ssh23:05:313
580099128,3cyclictest18790-21ssh21:21:123
580099126,4cyclictest1281-21cpu21:50:143
580099124,7cyclictest21960-21/usr/sbin/munin22:40:273
580099124,5cyclictest30955-21diskmemload22:50:133
580099123,9cyclictest0-21swapper/320:40:233
580099122,8cyclictest30955-21diskmemload00:14:063
580099122,4cyclictest0-21swapper/322:20:143
580099121,7cyclictest0-21swapper/300:18:293
580099120,5cyclictest0-21swapper/323:10:443
580099120,11cyclictest22948-21wc20:50:223
580099120,11cyclictest15941-21sensors20:35:253
580099120,10cyclictest0-21swapper/322:13:093
579899129,2cyclictest5411-21grep20:15:172
579899129,2cyclictest5179-21sh22:50:172
579899129,2cyclictest28063-21taskset00:01:562
579899129,2cyclictest27685-21grep21:00:222
579899129,2cyclictest22309-21grep20:50:172
579899129,2cyclictest18364-21basename22:19:592
579899129,2cyclictest11523-21grep19:20:182
579899129,2cyclictest10474-21tail21:35:272
579899128,3cyclictest23014-21taskset00:18:382
579899121,2cyclictest0-21swapper/221:30:132
579899120,1cyclictest0-21swapper/222:59:332
579899120,1cyclictest0-21swapper/221:50:022
579899120,12cyclictest0-21swapper/223:36:422
579899120,11cyclictest26917-21head22:05:172
579899120,11cyclictest0-21swapper/200:36:162
579499129,2cyclictest3037-21grep20:10:161
579499129,2cyclictest18321-21sed00:15:231
579499129,2cyclictest15025-21cat20:35:161
579499129,2cyclictest12122-21sh21:56:001
579499125,4cyclictest38150irq/131-snd_hda22:33:281
5794991212,0cyclictest0-21swapper/123:50:421
5794991212,0cyclictest0-21swapper/100:09:091
5794991210,1cyclictest208992sleep119:40:151
579499120,7cyclictest0-21swapper/123:46:181
579499120,6cyclictest0-21swapper/100:00:131
579499120,1cyclictest0-21swapper/120:20:161
579499120,11cyclictest8642-21basename21:35:141
579099129,3cyclictest28734-21kworker/u8:021:05:490
579099129,2cyclictest21578-21wc19:40:220
57092120,1sleep10-21swapper/100:27:201
580099119,1cyclictest11052-21tr19:20:163
580099118,2cyclictest8547-21grep19:15:163
580099118,2cyclictest826-21sed20:05:193
580099118,2cyclictest7605-21cat19:10:263
580099118,2cyclictest28619-21wc19:55:203
580099118,2cyclictest25984-21seq20:57:403
580099118,2cyclictest223312sleep319:43:553
580099118,2cyclictest21939-21sh23:58:523
580099118,2cyclictest20173-21sh21:41:283
580099118,2cyclictest19151-21sed22:00:233
580099118,2cyclictest17864-21ssh00:35:093
580099118,2cyclictest12992-21sort20:30:183
580099118,2cyclictest11813-21chrt21:55:463
580099118,2cyclictest1059-21ssh23:26:183
580099117,3cyclictest6011-21rm22:31:053
580099114,4cyclictest30955-21diskmemload23:20:023
580099112,4cyclictest0-21swapper/323:30:133
579899119,1cyclictest18674-21cut19:35:172
579899119,1cyclictest17255-21grep21:20:232
579899119,1cyclictest16287-21grep19:30:162
579899118,2cyclictest8150-21ssh22:13:302
579899118,2cyclictest32263-21grep23:25:262
579899118,2cyclictest30238-21python21:05:262
579899118,2cyclictest28194-21ssh22:45:102
579899118,2cyclictest2487-21basename20:10:132
579899118,2cyclictest24785-21grep20:55:162
579899118,2cyclictest19423-21expr22:20:162
579899118,2cyclictest17683-21head20:40:172
579899118,2cyclictest10725-21ssh00:30:192
579899118,2cyclictest10061-21grep20:25:142
579499119,1cyclictest956-21ssh21:50:081
579499119,1cyclictest32099-21ssh21:29:571
579499119,1cyclictest27304-21if_enp0s31f621:00:181
579499119,1cyclictest25331-21basename19:50:151
579499119,1cyclictest24910-21rm21:45:021
579499119,1cyclictest24910-21rm21:45:021
579499119,1cyclictest24903-21basename20:55:171
579499119,1cyclictest11943-21ls22:15:221
579499118,3cyclictest29562-21taskset19:57:321
579499118,3cyclictest16110-21ssh22:56:371
579499118,2cyclictest6191-21ssh23:30:081
579499118,2cyclictest5523-21head20:15:181
579499118,2cyclictest4125-21ssh23:28:421
579499118,2cyclictest30067-21hostname20:00:131
579499118,2cyclictest23174-21expr19:45:161
579499118,2cyclictest22406-21head20:50:171
579499118,2cyclictest10530-21head20:25:171
579499110,10cyclictest24112-21cat23:40:191
579099118,2cyclictest8530-21grep20:20:220
579099118,2cyclictest4841-21basename20:15:130
579099118,2cyclictest28085-21grep21:00:260
579099118,2cyclictest2667-21grep20:10:140
579099118,2cyclictest25532-21grep19:50:150
579099118,2cyclictest22368-21sort20:50:170
579099118,2cyclictest16763-21sed20:40:000
579099118,2cyclictest12021-21grep19:20:230
579099117,3cyclictest21417-21cron20:50:000
579099114,1cyclictest141rcu_preempt20:25:130
580099108,2cyclictest11561-21taskset20:25:403
580099108,1cyclictest25092-21sed19:50:003
580099108,1cyclictest13191-21cut19:25:143
580099107,2cyclictest26796-21expr21:00:153
579899108,2cyclictest25478-21ls19:50:152
579899108,1cyclictest80772sleep219:14:252
579899108,1cyclictest30030-21acpi20:00:112
579899108,1cyclictest20703-21expr19:40:152
579899108,1cyclictest151682sleep219:28:392
579899107,2cyclictest29756-21cron20:00:002
579899107,2cyclictest12509-21ls20:30:152
579499108,1cyclictest0-21swapper/119:35:131
579499107,2cyclictest2168-21cron20:10:001
579499107,2cyclictest16329-21tail19:30:161
579499107,2cyclictest13079-21cat20:30:191
579099108,1cyclictest10283-21chrt19:17:070
579099107,2cyclictest28415-21cut19:55:170
579099106,3cyclictest161rcuc/019:25:190
57989997,1cyclictest24553-21chrt19:46:542
57949998,1cyclictest6337-21cstates19:10:141
57989986,2cyclictest32512-21basename20:05:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional