You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-05 - 21:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri Jun 05, 2026 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
90022182109,28sleep10-21swapper/107:06:571
91502164112,29sleep30-21swapper/307:08:533
90032162108,29sleep20-21swapper/207:06:582
89042161107,29sleep00-21swapper/007:05:410
354521560,2sleep20-21swapper/211:15:132
181292760,2sleep30-21swapper/312:37:533
99772640,2sleep39981-21sort08:25:153
5702640,2sleep10-21swapper/111:36:311
118482600,2sleep10-21swapper/110:09:271
220872560,2sleep30-21swapper/309:28:483
52342540,1sleep20-21swapper/209:40:152
137882510,19chrt0-21swapper/311:22:283
137882510,19chrt0-21swapper/311:22:283
5452500,2sleep00-21swapper/011:36:310
188352500,1sleep20-21swapper/207:30:172
122252500,2sleep00-21swapper/012:09:570
122252500,2sleep00-21swapper/012:09:560
194282480,2sleep10-21swapper/110:15:051
113502450,1sleep20-21swapper/210:32:282
76102420,1sleep30-21swapper/312:30:153
164282420,1sleep10-21swapper/112:36:101
260882410,2chrt0-21swapper/211:55:232
193812410,2sleep20-21swapper/212:39:092
951199367,26cyclictest26703-21latency_hist07:50:012
9512993520,8cyclictest23468-21sh11:30:123
951199357,12cyclictest32412-21grep10:00:172
4972350,2chrt0-21swapper/312:00:203
950999334,27cyclictest28311-21sh09:57:000
951199323,27cyclictest9482-21ssh10:07:052
951099320,29cyclictest16095-21sh09:48:181
951299311,6cyclictest9492-21ssh11:19:563
951299311,28cyclictest24448-21ssh09:54:533
951299311,28cyclictest2156-21ssh10:01:303
951299310,7cyclictest29714-21ssh09:34:393
951199313,4cyclictest24066-21df09:30:122
951199312,10cyclictest21158-21sh11:51:492
951199310,20cyclictest9906-21cat09:20:002
951299301,27cyclictest23085-21ssh12:17:223
951299300,27cyclictest4262-21ssh10:50:553
9510993021,7cyclictest24592-21perf09:55:001
951099301,27cyclictest19789-21ssh09:26:301
951099300,6cyclictest1854-21ssh11:14:001
950999301,4cyclictest0-21swapper/009:35:500
950999301,27cyclictest31614-21kworker/0:009:51:440
27282300,2chrt0-21swapper/110:02:041
9512992921,6cyclictest18690-21ssh10:14:333
951299291,26cyclictest22967-21ssh10:17:063
951299290,26cyclictest32367-21ssh11:12:063
951199294,9cyclictest19565-21sed08:45:242
951199293,3cyclictest32503-21ssh09:35:422
951199293,11cyclictest21878-21latency_hist11:05:002
9511992920,6cyclictest28665-21diskmemload09:25:272
951199291,10cyclictest26680-21ssh10:44:212
9510992924,3cyclictest18842-21ssh12:14:511
951099291,3cyclictest28718-21ssh09:33:391
951099291,26cyclictest0-21swapper/109:57:111
951099290,4cyclictest4280-21sh10:50:551
9509992924,3cyclictest52850irq/140-enp3s0-09:28:190
9509992923,3cyclictest0-21swapper/009:20:470
951299281,25cyclictest29758-21diskmemload09:58:283
951199283,2cyclictest28-21ksoftirqd/210:53:172
9511992826,1cyclictest0-21swapper/209:55:242
951199282,10cyclictest2057-21unixbench-2d12:25:252
9511992812,6cyclictest28-21ksoftirqd/211:33:262
951199280,26cyclictest25980-21sh11:07:272
9510992824,3cyclictest0-21swapper/112:16:131
9510992823,3cyclictest7775-21ssh09:41:371
9510992823,3cyclictest32476-21ssh10:48:261
951099281,26cyclictest52950irq/141-enp3s0-10:12:161
951099281,25cyclictest0-21swapper/110:25:111
951099281,25cyclictest0-21swapper/109:35:491
950999281,2cyclictest0-21swapper/010:01:440
950999281,25cyclictest0-21swapper/012:14:080
950999281,25cyclictest0-21swapper/011:15:060
950999281,25cyclictest0-21swapper/010:50:550
950999281,19cyclictest0-21swapper/011:13:020
108532280,2sleep10-21swapper/110:31:581
9512992722,3cyclictest15340-21ssh09:47:323
9512992722,3cyclictest12115-21ssh10:09:433
951299271,3cyclictest32409-21diskmemload09:35:373
951299271,24cyclictest0-21swapper/309:05:533
951299271,24cyclictest0-21swapper/309:05:523
951199274,9cyclictest7266-21cut10:30:002
951199274,6cyclictest26302-21apt-get10:20:112
951199273,8cyclictest29308-21fschecks_count07:55:152
951199273,7cyclictest29260-21expr11:10:142
951199271,9cyclictest13293-21tail12:10:142
951199271,20cyclictest11536-21ssh09:45:122
951199271,11cyclictest4107-21idleruntime-cro11:40:012
951199270,25cyclictest13614-21ssh09:22:042
951199270,10cyclictest23558-21ps07:40:222
951099271,25cyclictest0-21swapper/111:09:031
951099271,25cyclictest0-21swapper/109:07:371
951099271,25cyclictest0-21swapper/109:07:371
951099271,25cyclictest0-21swapper/109:00:291
951099271,19cyclictest0-21swapper/111:15:531
951099270,2cyclictest12754-21ssh09:21:131
951099270,25cyclictest12202-21ssh12:09:551
951099270,25cyclictest12202-21ssh12:09:541
9509992724,2cyclictest0-21swapper/010:36:110
9509992724,2cyclictest0-21swapper/009:09:000
9509992724,2cyclictest0-21swapper/009:08:590
9509992723,2cyclictest0-21swapper/009:30:130
9509992722,3cyclictest21593-21ssh10:15:430
9509992718,6cyclictest31599-21ssh10:47:320
950999271,2cyclictest28665-21diskmemload10:10:050
950999271,2cyclictest0-21swapper/009:49:390
950999271,25cyclictest0-21swapper/009:42:030
950999271,24cyclictest0-21swapper/011:27:020
950999271,24cyclictest0-21swapper/011:06:550
950999271,24cyclictest0-21swapper/009:00:200
950999271,19cyclictest0-21swapper/010:06:470
63752270,4sleep20-21swapper/212:05:142
63752270,4sleep20-21swapper/212:05:132
9512992621,2cyclictest0-21swapper/312:13:113
951299261,2cyclictest0-21swapper/311:07:103
951299261,24cyclictest0-21swapper/310:49:133
951299261,24cyclictest0-21swapper/310:21:303
951299261,23cyclictest0-21swapper/309:15:433
951299261,23cyclictest0-21swapper/309:00:223
951299260,24cyclictest15680-21ssh09:24:083
951199269,12cyclictest19409-21apt-get12:15:102
951199264,7cyclictest28-21ksoftirqd/212:00:562
951199264,7cyclictest22367-21sed07:40:012
951199261,8cyclictest13416-21sh11:45:492
951199261,2cyclictest0-21swapper/210:36:202
951199261,2cyclictest0-21swapper/210:11:202
951199261,23cyclictest0-21swapper/209:54:362
951199261,23cyclictest0-21swapper/209:07:062
951199261,23cyclictest0-21swapper/209:07:062
951199261,23cyclictest0-21swapper/209:02:592
951199260,5cyclictest0-21swapper/212:25:002
951199260,24cyclictest23608-21ssh10:17:442
9510992624,1cyclictest0-21swapper/110:36:361
9510992624,1cyclictest0-21swapper/109:18:161
9510992623,2cyclictest0-21swapper/111:27:271
9510992623,2cyclictest0-21swapper/110:44:291
951099261,24cyclictest28665-21diskmemload10:24:561
951099261,24cyclictest0-21swapper/111:20:531
951099261,24cyclictest0-21swapper/111:20:521
9509992623,2cyclictest0-21swapper/010:44:100
950999261,24cyclictest0-21swapper/012:16:120
950999261,24cyclictest0-21swapper/009:18:170
950999261,1cyclictest0-21swapper/011:20:530
950999261,1cyclictest0-21swapper/011:20:520
950999261,1cyclictest0-21swapper/010:21:300
63642260,1sleep30-21swapper/308:15:243
63642260,1sleep30-21swapper/308:15:243
255792260,2sleep10-21swapper/111:30:501
157272260,3sleep3331rcuc/311:00:133
9512992522,2cyclictest0-21swapper/310:36:113
9512992520,3cyclictest4479-21ssh10:27:313
951299251,23cyclictest0-21swapper/312:09:363
951299251,23cyclictest0-21swapper/312:09:353
951299251,23cyclictest0-21swapper/310:57:303
951299251,23cyclictest0-21swapper/309:40:523
951299251,1cyclictest0-21swapper/310:43:493
951199254,8cyclictest17765-21taskset08:42:472
951199254,6cyclictest22343-21sed08:55:002
9511992522,2cyclictest0-21swapper/211:26:002
9511992521,3cyclictest271rcuc/208:03:262
951199251,8cyclictest8992-21taskset08:23:052
9511992517,5cyclictest1839-21ssh09:13:512
951199251,6cyclictest29180-21ls10:45:232
951199251,10cyclictest7923-21sh11:42:052
9511992510,6cyclictest28-21ksoftirqd/211:24:592
9511992510,6cyclictest28-21ksoftirqd/211:24:582
951099255,4cyclictest141rcu_preempt07:35:001
9510992515,8cyclictest9875-21df07:10:151
9509992523,1cyclictest0-21swapper/010:34:160
951299242,4cyclictest141rcu_preempt12:20:113
951299240,23cyclictest0-21swapper/310:34:153
951199244,6cyclictest11956-21expr08:30:132
951199243,8cyclictest12727-21ssh10:57:392
951199242,6cyclictest26782-21taskset07:50:072
9510992415,6cyclictest2967-21apt-get08:10:111
950999240,1cyclictest0-21swapper/010:25:100
9512992321,1cyclictest0-21swapper/309:13:433
9512992311,9cyclictest16719-21apt-get11:25:113
951199235,5cyclictest14245-21diskstats08:35:162
951199233,6cyclictest28-21ksoftirqd/212:32:052
9509992321,1cyclictest0-21swapper/011:03:070
72062230,1sleep20-21swapper/208:20:012
72062230,1sleep20-21swapper/208:20:002
282272230,2sleep10-21swapper/112:20:451
951199224,5cyclictest17156-21python307:25:252
951199224,5cyclictest17156-21python307:25:242
9511992219,2cyclictest271rcuc/208:12:302
951199221,7cyclictest99792grep08:25:152
951099226,7cyclictest22-21ksoftirqd/111:53:051
9510992218,3cyclictest14656-21ssh10:59:381
9510992216,4cyclictest22-21ksoftirqd/108:40:011
9510992213,6cyclictest29875-21ntpq07:55:201
9510992213,6cyclictest27752-21awk07:50:221
950999223,3cyclictest28665-21diskmemload11:48:520
951299219,8cyclictest24677-21apt-get07:45:133
9512992113,6cyclictest28537-21seq11:57:283
9512992113,6cyclictest22306-21sendmail-msp07:40:003
9512992112,6cyclictest23433-21ntp_kernel_pll_07:40:213
9511992117,3cyclictest271rcuc/208:08:402
9511992116,3cyclictest24575-21cat09:00:002
951199210,8cyclictest13462-21taskset07:18:322
9510992116,4cyclictest0-21swapper/111:03:061
9510992112,6cyclictest25313-21grep11:55:141
950999214,3cyclictest28665-21diskmemload11:53:360
950999213,3cyclictest28665-21diskmemload12:37:000
950999213,3cyclictest28665-21diskmemload11:55:090
9509992113,6cyclictest10538-21ntp_states08:25:200
9509992113,5cyclictest31287-21sh12:23:510
9509992112,7cyclictest26896-21tr07:50:140
9509992110,7cyclictest52850irq/140-enp3s0-12:30:120
951299207,11cyclictest34-21ksoftirqd/308:40:003
951299206,3cyclictest19322run-parts08:05:233
9512992013,5cyclictest2932-21seq11:38:503
9512992012,6cyclictest12137-21tune2fs08:30:143
9512992010,7cyclictest22908-21tune2fs08:55:133
951099209,8cyclictest30134-21ntpq09:10:191
9510992014,4cyclictest6123-21sh12:29:221
9510992013,5cyclictest278-21systemd-journal08:05:001
9510992012,6cyclictest6800-21seq08:17:421
9510992012,6cyclictest6800-21seq08:17:411
9510992012,6cyclictest22211-21seq07:39:291
9510992012,5cyclictest28665-21diskmemload12:34:261
951099200,7cyclictest5417-21/usr/sbin/munin11:40:191
950999209,9cyclictest6982-21sh11:41:090
9509992018,1cyclictest0-21swapper/009:13:430
950999201,5cyclictest0-21swapper/012:00:190
950999201,5cyclictest0-21swapper/010:56:330
9509992012,6cyclictest23176-21sed08:55:160
9509992012,6cyclictest22280-21systemctl07:40:000
9512991913,4cyclictest6572-21sh11:40:443
9512991913,4cyclictest5478-21sh12:28:433
9512991911,6cyclictest27186-21tune2fs07:50:173
951199190,7cyclictest28-21ksoftirqd/207:23:102
951199190,6cyclictest28-21ksoftirqd/207:14:202
951099192,12cyclictest7269-21apt-get08:20:121
9510991912,5cyclictest21257-21ntpq08:50:191
9510991912,5cyclictest18412-21cat08:45:141
9510991912,5cyclictest16288-21cat08:40:131
9510991912,5cyclictest13084-21seq11:45:291
950999191,5cyclictest0-21swapper/012:29:290
9509991914,3cyclictest15740-21sh07:25:000
9509991911,6cyclictest17380-21grep08:40:250
9509991911,6cyclictest1527-21ls08:05:180
9509991910,6cyclictest31565-21tune2fs08:00:140
951299182,11cyclictest18204-21apt-get08:45:133
9512991813,3cyclictest19930-21sh11:50:353
9512991813,3cyclictest15271-21unixbench_multi07:20:263
9512991812,4cyclictest14815-21sh11:47:123
9512991811,5cyclictest8567-21cat08:20:243
9512991811,5cyclictest20941-21if_enp3s008:50:173
9512991811,5cyclictest17451-21grep08:40:253
9510991811,5cyclictest17402-21unixbench_multi07:25:261
9510991811,5cyclictest17402-21unixbench_multi07:25:261
9510991811,5cyclictest12989-21cat08:30:221
9510991811,5cyclictest12779-21sendmail_mailqu07:15:211
9510991810,6cyclictest9604-21cat08:25:121
9510991810,6cyclictest22937-21fw_packets08:55:141
9510991810,5cyclictest26765-21date07:50:011
951099180,8cyclictest3212350tune2fs12:00:141
950999182,11cyclictest11698-21apt-get07:15:120
950999181,6cyclictest0-21swapper/011:30:140
9509991810,6cyclictest7632-21cut08:20:160
9509991810,6cyclictest10721-21awk07:10:230
951299179,6cyclictest11927-21expr07:15:143
951299172,10cyclictest15999-21apt-get07:25:143
951299172,10cyclictest15999-21apt-get07:25:133
951299171,11cyclictest31107-21apt-get08:00:113
9512991710,5cyclictest30433-21grep07:55:243
9512991710,5cyclictest18350-21ls07:30:133
951099179,6cyclictest24513-21cut07:45:001
9510991712,3cyclictest802-21/usr/sbin/munin08:05:111
9510991712,3cyclictest14835-21pmu-power07:20:221
950999179,5cyclictest18342-21tr08:45:130
9509991712,3cyclictest24652-21/usr/sbin/munin07:45:210
9509991711,4cyclictest20189-21sed07:35:010
9509991710,5cyclictest29450-21grep07:55:150
9509991710,5cyclictest24568-21latency_hist07:45:000
9509991710,5cyclictest13737-21sed08:35:010
951299162,10cyclictest9552-21apt-get07:10:123
9512991611,3cyclictest3325-21fschecks_count08:10:133
950999162,10cyclictest20338-21apt-get08:50:120
950999161,13cyclictest7012-21apt08:20:000
950999161,13cyclictest7012-21apt08:20:000
9509991610,4cyclictest4306-21timerwakeupswit08:10:230
9509991610,4cyclictest14736-21needreboot08:35:210
950999156,7cyclictest16630-21if_err_enp3s007:25:180
950999156,7cyclictest16630-21if_err_enp3s007:25:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional