You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-30 - 11:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Apr 30, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1485529366,9sleep20-21swapper/219:09:092
1461529378,10sleep30-21swapper/319:05:593
1279529383,6sleep00-21swapper/019:05:090
1283528463,15sleep10-21swapper/119:05:141
139542700,0sleep30-21swapper/322:12:413
296232460,0sleep20-21swapper/222:01:172
1501699330,33cyclictest0-21swapper/222:16:132
1501499260,1cyclictest0-21swapper/100:19:551
1500999260,4cyclictest0-21swapper/021:40:230
1501999254,3cyclictest11430-21users20:05:273
1501499242,21cyclictest8866-21diskmemload22:20:231
1501499240,20cyclictest0-21swapper/122:46:371
1501999230,1cyclictest0-21swapper/300:01:063
15014992320,1cyclictest0-21swapper/119:30:001
1501999211,1cyclictest9163-21kworker/3:220:04:153
15019992015,2cyclictest39-21ksoftirqd/322:09:393
15019992015,2cyclictest39-21ksoftirqd/322:09:383
15014992020,0cyclictest0-21swapper/121:54:201
15014992017,2cyclictest25-21ksoftirqd/100:38:441
1501499201,2cyclictest141rcu_preempt00:00:151
1501499200,19cyclictest0-21swapper/123:49:091
1501999192,14cyclictest141rcu_preempt21:56:373
15019991914,1cyclictest39-21ksoftirqd/322:36:583
15014991916,2cyclictest0-21swapper/100:26:211
1501499191,16cyclictest30838-21ssh23:07:401
15009991919,0cyclictest0-21swapper/023:37:310
15019991816,2cyclictest39-21ksoftirqd/322:44:113
15019991815,2cyclictest39-21ksoftirqd/323:28:423
1501999181,4cyclictest19250-21aten2_r7power_e23:00:133
15019991813,2cyclictest39-21ksoftirqd/323:21:573
15014991815,3cyclictest0-21swapper/122:44:071
1501499180,0cyclictest0-21swapper/100:08:521
1500999180,3cyclictest0-21swapper/021:53:450
1500999180,1cyclictest0-21swapper/022:35:290
1500999180,12cyclictest26562-21ssh22:00:140
1501999178,4cyclictest141rcu_preempt23:58:393
1501999178,4cyclictest141rcu_preempt23:58:383
1501999174,3cyclictest39-21ksoftirqd/321:30:203
1501999174,10cyclictest39-21ksoftirqd/321:50:143
1501999173,4cyclictest141rcu_preempt21:23:223
15019991716,1cyclictest0-21swapper/300:38:463
15019991715,2cyclictest39-21ksoftirqd/322:27:203
15019991711,4cyclictest12219-21ssh23:16:483
15019991711,2cyclictest39-21ksoftirqd/322:32:563
1501699170,1cyclictest0-21swapper/219:15:142
1501699170,16cyclictest0-21swapper/200:15:242
1500999172,0cyclictest0-21swapper/023:17:240
15009991717,0cyclictest0-21swapper/022:25:460
15009991716,1cyclictest0-21swapper/000:35:480
1500999170,16cyclictest0-21swapper/022:09:080
1500999170,16cyclictest0-21swapper/022:09:080
1501999169,1cyclictest141rcu_preempt21:43:353
1501999162,3cyclictest141rcu_preempt23:07:493
15019991616,0cyclictest0-21swapper/323:37:203
1501999160,15cyclictest0-21swapper/323:48:183
1501699161,14cyclictest0-21swapper/223:01:332
1501499168,7cyclictest38150irq/131-snd_hda21:33:091
1501499160,15cyclictest0-21swapper/122:54:481
1500999164,6cyclictest18974-21munin-run20:25:000
1500999164,6cyclictest1582-21idleruntime-cro21:25:000
1500999163,10cyclictest0-21swapper/019:35:130
15009991616,0cyclictest0-21swapper/023:23:510
1501999159,5cyclictest39-21ksoftirqd/321:13:033
1501999154,2cyclictest141rcu_preempt22:56:593
1501999153,10cyclictest39-21ksoftirqd/323:40:143
1501999152,13cyclictest39-21ksoftirqd/300:29:413
15019991511,3cyclictest26390-21pmu-power00:30:203
15019991511,3cyclictest17372-21idleruntime-cro19:15:013
15016991513,2cyclictest32-21ksoftirqd/223:23:412
1501699150,15cyclictest0-21swapper/200:10:122
1501699150,10cyclictest0-21swapper/222:20:262
15014991515,0cyclictest0-21swapper/122:27:261
15014991512,2cyclictest0-21swapper/123:40:151
1501499150,0cyclictest0-21swapper/123:16:011
1500999153,8cyclictest499-21munin-run00:35:000
1500999150,0cyclictest0-21swapper/000:02:190
1501999146,3cyclictest225-21systemd-journal21:01:533
15019991412,2cyclictest8866-21diskmemload00:20:003
15019991412,1cyclictest381rcuc/322:00:063
15019991411,2cyclictest39-21ksoftirqd/300:23:063
1501699149,5cyclictest24290-21apache223:32:482
1501699142,11cyclictest30374-21ssh23:50:162
15016991411,2cyclictest0-21swapper/200:28:472
1501699140,13cyclictest0-21swapper/220:45:142
1501499149,4cyclictest6597-21wc21:05:001
1501499149,3cyclictest20231-21cat23:00:191
1501499147,5cyclictest25989-21idleruntime-cro22:00:001
15014991413,1cyclictest38150irq/131-snd_hda23:13:081
15014991410,3cyclictest20206-21sed21:35:211
15014991410,2cyclictest20160-21grep20:25:171
1501499140,14cyclictest0-21swapper/121:40:121
1500999143,11cyclictest449-21irqbalance22:11:120
15009991414,0cyclictest0-21swapper/023:26:200
15009991411,2cyclictest20858-21wc20:25:260
15009991410,3cyclictest26070-21grep20:35:270
15009991410,3cyclictest18923-21grep21:35:140
15009991410,2cyclictest1238-21wc19:45:250
1501999139,4cyclictest39-21ksoftirqd/322:47:153
1501999139,3cyclictest39-21ksoftirqd/321:17:213
1501999137,3cyclictest39-21ksoftirqd/322:18:563
1501999132,10cyclictest0-21swapper/321:26:113
15019991311,1cyclictest381rcuc/321:49:223
15019991310,2cyclictest6546-21sed19:55:253
15019991310,2cyclictest27604-21grep00:10:143
15019991310,2cyclictest2528-21head20:55:163
15019991310,2cyclictest20166-21grep20:25:163
15019991310,2cyclictest17112-21grep20:20:153
1501999130,2cyclictest0-21swapper/322:20:173
1501999130,1cyclictest0-21swapper/323:12:543
1501699139,3cyclictest18965-21ls20:25:002
1501699133,2cyclictest16332-21timerwakeupswit23:40:262
15016991311,1cyclictest1880-21perf19:50:002
15016991310,2cyclictest3977-21ssh21:25:272
15016991310,2cyclictest3944-21grep21:45:172
15016991310,2cyclictest27592-21grep20:40:182
15016991310,2cyclictest23502-21sed22:40:262
15016991310,2cyclictest14495-21cron22:35:012
1501699130,2cyclictest0-21swapper/200:21:302
1501699130,1cyclictest0-21swapper/222:47:012
1501699130,13cyclictest0-21swapper/223:15:382
1501699130,13cyclictest0-21swapper/223:14:192
1501499139,3cyclictest3865-21cron21:00:011
1501499138,4cyclictest10979-21ssh22:32:141
1501499138,2cyclictest7432-21basename23:35:001
15014991312,1cyclictest0-21swapper/121:25:201
1501499130,1cyclictest0-21swapper/122:00:321
1501499130,12cyclictest14753-21ssh00:22:341
1500999139,3cyclictest26244-21sh00:09:340
1500999133,2cyclictest18977-21ntpq21:15:190
15009991310,2cyclictest7489-21tail21:05:190
15009991310,2cyclictest6540-21grep19:55:240
15009991310,2cyclictest3282-21grep23:10:210
15009991310,2cyclictest10437-21ssh21:30:110
1500999130,4cyclictest4570-21apt-config19:55:000
1500999130,2cyclictest0-21swapper/023:49:480
1500999130,1cyclictest0-21swapper/023:04:550
1500999130,12cyclictest3410-21ntpq21:25:210
1500999130,12cyclictest28548-21head20:40:270
1500999130,11cyclictest30048-21cut19:40:150
1501999129,3cyclictest25874-21kworker/u8:020:50:363
1501999129,2cyclictest8476-21ssh22:52:133
1501999129,2cyclictest7382-21sort21:05:183
1501999129,2cyclictest30207-21cut23:50:163
1501999129,2cyclictest23105-21grep19:25:173
1501999129,2cyclictest20739-21grep21:35:263
1501999129,2cyclictest19044-21sort19:15:263
1501999120,1cyclictest0-21swapper/320:15:373
1501699129,2cyclictest5441-21tr22:50:182
1501699129,2cyclictest4327-21ssh22:28:362
1501699129,2cyclictest2010-21tr20:55:152
1501699129,2cyclictest19337-21cron21:55:002
1501699129,2cyclictest11445-21cron23:59:592
1501699129,2cyclictest11445-21cron00:00:002
1501699129,2cyclictest10735-21tr21:30:142
1501699126,2cyclictest225-21systemd-journal20:40:002
1501699126,2cyclictest225-21systemd-journal20:00:002
15016991212,0cyclictest0-21swapper/223:38:272
1501699120,3cyclictest0-21swapper/200:30:252
1501699120,2cyclictest0-21swapper/222:37:272
1501499129,2cyclictest8949-21grep20:00:251
1501499129,2cyclictest5174-21grep19:55:141
1501499129,2cyclictest3997-21grep19:50:251
1501499129,2cyclictest3748-21df21:45:161
1501499129,2cyclictest30294-21ssh23:50:151
1501499129,2cyclictest26733-21sed21:20:151
1501499129,2cyclictest25325-21tr00:30:141
1501499129,2cyclictest23676-21grep20:30:251
1501499129,2cyclictest23449-21ssh23:23:531
1501499129,2cyclictest17941-21grep20:20:201
1501499129,2cyclictest1280-21cron20:55:001
1501499129,2cyclictest11477-21munin-run00:00:011
1501499129,2cyclictest11477-21munin-run00:00:011
1501499129,2cyclictest10447-21sed22:10:211
1501499129,2cyclictest0-21swapper/119:40:271
1501499128,3cyclictest21117-21sendmail22:40:011
1501499124,4cyclictest225-21systemd-journal22:10:011
1501499124,4cyclictest225-21systemd-journal22:10:011
15014991211,1cyclictest0-21swapper/123:27:241
15014991210,1cyclictest28656-21tail00:10:181
1500999129,2cyclictest29777-21grep20:45:160
1500999129,2cyclictest15333-21grep23:40:160
1500999129,2cyclictest1532-21grep23:30:260
1500999126,5cyclictest21671-21sh21:56:390
15009991210,1cyclictest24276-21apache222:50:200
1501999118,2cyclictest27688-21tr19:35:153
1501999118,2cyclictest26828-21chrt19:31:053
1501999118,2cyclictest24505-21aten2_r7power_e20:35:143
1501999118,2cyclictest22235-21cat20:30:143
1501999117,3cyclictest12703-21sort20:10:183
1501999115,5cyclictest39-21ksoftirqd/323:30:213
1501999114,7cyclictest0-21swapper/300:09:593
1501699118,2cyclictest801-21taskset00:35:032
1501699118,2cyclictest7438-21ssh22:09:082
1501699118,2cyclictest7438-21ssh22:09:072
1501699118,2cyclictest6396-21taskset21:03:582
1501699118,2cyclictest32628-21cat20:50:192
1501699118,2cyclictest29888-21ssh21:21:472
1501699118,2cyclictest29527-21ssh23:06:382
1501699118,2cyclictest28714-21uname21:40:212
1501699118,2cyclictest28631-21grep19:35:222
1501699118,2cyclictest28411-21ssh23:27:332
1501699118,2cyclictest25471-21systemd-cgroups00:08:582
1501699118,2cyclictest25036-21ssh23:46:542
1501699118,2cyclictest20264-21grep19:20:132
1501699118,2cyclictest20219-21cut20:25:162
1501699118,2cyclictest19585-21uniq21:35:162
1501699118,2cyclictest18203-21ssh22:59:412
1501699118,2cyclictest17334-21rm00:03:042
1501699118,2cyclictest11826-21basename20:10:002
1501699118,2cyclictest10343-21ntpq22:10:202
1501699113,4cyclictest225-21systemd-journal21:20:002
1501699110,10cyclictest20029-21sh21:55:232
1501499118,2cyclictest28951-21taskset20:44:591
1501499118,2cyclictest25783-21grep19:30:161
1501499118,2cyclictest24210-21ssh21:18:571
1501499118,2cyclictest2101-21grep19:50:001
1501499118,2cyclictest16459-21taskset20:18:451
1501499118,2cyclictest16173-21kernelversion19:10:181
1501499118,2cyclictest13250-21ssh22:55:481
1501499118,2cyclictest10554-21ssh23:37:171
1501499118,2cyclictest10410-21ssh21:10:181
1501499117,3cyclictest18177-21ssh22:15:481
1500999119,1cyclictest7471-21grep22:30:170
1500999118,2cyclictest8837-21ssh00:19:300
1500999118,2cyclictest8359-21ssh21:47:560
1500999118,2cyclictest7076-21ssh23:56:360
1500999118,2cyclictest7076-21ssh23:56:360
1500999118,2cyclictest30069-21ssh22:45:180
1500999118,2cyclictest27349-21ssh00:10:080
1500999118,2cyclictest27297-21ssh22:21:200
1500999118,2cyclictest2567-21ssh23:53:130
1500999118,2cyclictest24987-21ssh22:41:300
1500999118,2cyclictest23146-21ntpq20:30:190
1500999118,2cyclictest16824-21ssh00:24:110
1500999118,2cyclictest16588-21perf20:20:000
1500999118,2cyclictest16500-21wc19:10:200
1500999118,2cyclictest14337-21ssh21:12:470
1500999118,2cyclictest13159-21ssh22:55:450
1500999116,3cyclictest27333-21wc19:34:590
1500999115,5cyclictest22137-21basename19:24:590
15009991110,1cyclictest0-21swapper/023:05:520
1501999108,1cyclictest4216-21taskset19:52:063
1501999108,1cyclictest15232sleep319:45:433
1501999107,2cyclictest31071-21chrt20:46:453
1501999107,2cyclictest29843-21tr19:40:133
1501999107,2cyclictest27687-21basename20:40:193
1501699108,1cyclictest8261-21sed20:00:182
1501699108,1cyclictest7815-21ls21:05:222
1501699108,1cyclictest24980-21sh19:30:002
1501699108,1cyclictest23958-21wc19:25:262
1501699108,1cyclictest15490-21cut19:10:142
1501699108,1cyclictest12543-21expr20:10:162
1501699107,2cyclictest3122-21head19:50:162
1501699107,2cyclictest30533-21head19:40:162
1501699107,2cyclictest14795-21expr20:15:132
1501699103,6cyclictest8866-21diskmemload21:10:152
1501499108,1cyclictest31242-21seq20:48:591
1501499108,1cyclictest17610-21basename19:15:121
1501499108,1cyclictest13869-21taskset20:11:051
1501499107,2cyclictest6726-21cat21:05:131
1501499107,2cyclictest28133-21expr19:35:171
1501499107,2cyclictest24694-21expr20:35:161
1501499107,2cyclictest21071-21wc19:20:171
1501499107,2cyclictest10524-21taskset20:05:171
1500999108,1cyclictest18661-21taskset19:15:200
1500999108,1cyclictest17578-21sh22:15:200
1500999108,1cyclictest12119-21aten2_r7power_e20:10:150
1500999108,1cyclictest1053-21taskset20:52:120
1500999107,2cyclictest24732-21cat19:30:000
1500999107,2cyclictest19676-21ssh00:26:140
1500999105,3cyclictest2022-21tr20:55:150
1500999103,6cyclictest7983-21fschecks_time20:00:160
150199997,1cyclictest20155-21wget19:20:133
150169997,1cyclictest21924-21readlink20:30:012
150099997,1cyclictest10673-21cat20:05:190
150099996,2cyclictest6263-21chrt21:02:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional