You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-20 - 06:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Sun Sep 20, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
152252175101,29sleep00-21swapper/019:07:150
152852174102,29sleep30-21swapper/319:08:023
151122166111,28sleep10-21swapper/119:05:481
152322162106,27sleep20-21swapper/219:07:212
185632710,1sleep20-21swapper/219:15:192
57612700,3sleep15749-21apt-get00:25:131
81252670,7sleep08129-21tr20:05:250
306512500,3sleep3331rcuc/322:42:503
284552500,1sleep20-21swapper/200:17:282
97772490,1sleep30-21swapper/322:51:013
295602470,2sleep20-21swapper/222:17:452
295602470,2sleep20-21swapper/222:17:452
24902460,1sleep20-21swapper/222:45:262
184912460,1sleep10-21swapper/123:21:531
15710994611,21cyclictest6557-21seq20:00:551
15710994611,17cyclictest636-21runrttasks19:48:141
1571299451,26cyclictest17935-21/usr/sbin/munin21:45:143
1571299440,29cyclictest26224-21ssh23:51:243
1571299431,28cyclictest2351-21diskmemload21:17:293
1571299431,27cyclictest2351-21diskmemload22:05:333
1571299431,25cyclictest30771-21ssh23:30:333
15712994310,23cyclictest5358-21sh23:35:513
1571299430,29cyclictest25603-21ssh21:26:223
1571299430,28cyclictest13165-21ssh21:40:593
1571299430,25cyclictest11749-21chrt20:15:153
1571299430,23cyclictest27679-21ssh22:15:493
1571299430,23cyclictest27679-21ssh22:15:493
15710994312,15cyclictest2425-21sh00:22:081
263972420,1sleep00-21swapper/021:50:500
1571299421,27cyclictest32213-21sh23:55:423
1571299421,26cyclictest2351-21diskmemload23:15:323
1571299421,26cyclictest14816-21ssh22:55:183
1571299421,26cyclictest10104-21apt-get23:40:113
1571299421,23cyclictest0-21swapper/321:31:213
1571299420,28cyclictest19909-21ssh00:10:343
1571299420,25cyclictest1057-21ssh00:20:453
1571299420,24cyclictest29252-21ssh21:53:433
1571299420,24cyclictest22563-21sshd22:12:243
1571099427,18cyclictest21343-21sh21:47:281
1571299416,26cyclictest10875-21sh22:04:083
1571299411,30cyclictest18140-21ssh00:34:273
1571299411,30cyclictest0-21swapper/323:03:163
1571299411,24cyclictest4396-21sh23:11:143
1571299411,23cyclictest2351-21diskmemload22:24:073
1571299410,26cyclictest8001-21ssh00:02:003
1571299410,24cyclictest24860-21ssh23:26:213
1571299410,24cyclictest20164-21ssh21:22:383
1571299410,24cyclictest18425-21ssh22:33:593
1571299401,25cyclictest2351-21diskmemload00:26:423
1571299401,24cyclictest2351-21diskmemload00:18:343
1571299401,20cyclictest2351-21diskmemload23:46:393
1571299401,20cyclictest2351-21diskmemload23:46:393
1571299400,27cyclictest6236-21sh21:35:473
1571299400,24cyclictest2351-21diskmemload21:13:583
1571299400,23cyclictest23599-21rm00:38:123
1571299400,22cyclictest30004-21seq19:40:253
1571099406,20cyclictest15533-21sh22:07:051
1571099406,18cyclictest12383-21sh22:29:391
1571099401,24cyclictest2388-21ssh21:57:361
1571299391,35cyclictest0-21swapper/320:30:193
1571299391,25cyclictest3021-21sh22:45:573
1571299391,24cyclictest2351-21diskmemload23:21:463
1571299391,22cyclictest2351-21diskmemload23:07:033
1571299391,22cyclictest2351-21diskmemload22:39:023
1571299390,29cyclictest9244-21sh22:26:273
1571299390,25cyclictest11200-21sh00:05:033
1571099397,17cyclictest28152-21taskset20:53:011
1571099391,23cyclictest0-21swapper/123:49:371
1571099391,23cyclictest0-21swapper/123:49:371
1571299381,27cyclictest4284-21sh21:59:283
1571099388,17cyclictest23855-21sh23:01:401
1571099380,1cyclictest0-21swapper/120:30:191
15710993713,21cyclictest12175-21sendmail_mailtr21:40:231
1571099371,21cyclictest13721-21sh23:42:191
15710993710,15cyclictest2351-21diskmemload21:27:061
1571299361,19cyclictest0-21swapper/319:27:353
1571299361,15cyclictest0-21swapper/319:15:253
1571299360,21cyclictest26165-21chrt20:49:583
1571099369,15cyclictest28246-21sh00:17:151
1571099366,16cyclictest0-21swapper/120:55:321
1571099365,17cyclictest22147-21ssh22:35:591
1571099361,4cyclictest29644-21ssh23:30:181
1571099361,21cyclictest2351-21diskmemload21:32:151
1571299350,20cyclictest8381-21ls20:05:273
1571099358,18cyclictest2351-21diskmemload21:11:401
1571099351,20cyclictest2351-21diskmemload00:09:131
1571099351,19cyclictest0-21swapper/122:02:221
1571099350,7cyclictest32740-21rm23:56:141
1571299341,16cyclictest0-21swapper/319:30:123
1571299340,19cyclictest0-21swapper/320:29:423
1571099341,19cyclictest2351-21diskmemload00:39:081
1571299331,30cyclictest0-21swapper/320:56:343
1571299330,19cyclictest0-21swapper/320:03:383
1571299330,18cyclictest0-21swapper/319:35:253
1571099332,28cyclictest23950-21sort23:50:151
1571099331,30cyclictest0-21swapper/119:44:551
1571099331,1cyclictest0-21swapper/119:39:031
1571099330,2cyclictest0-21swapper/100:30:171
1571299321,29cyclictest15476-21munin-run20:25:013
15710993212,18cyclictest9645-21expr21:15:141
1571099321,1cyclictest0-21swapper/120:45:131
1571099321,1cyclictest0-21swapper/119:50:381
15710993210,19cyclictest17356-21awk21:20:221
1571099320,4cyclictest16143-21ssh22:55:401
1571299311,28cyclictest0-21swapper/320:50:523
1571299311,17cyclictest0-21swapper/321:00:263
1571299311,15cyclictest0-21swapper/320:10:113
1571299310,16cyclictest0-21swapper/319:45:223
1571099312,25cyclictest75-21kswapd022:40:121
1571099312,24cyclictest13166-21apt-get22:30:111
1571099311,27cyclictest22532-21diskstats19:25:161
1571099311,24cyclictest0-21swapper/119:31:281
1571099311,1cyclictest0-21swapper/121:02:111
1571099311,1cyclictest0-21swapper/120:13:071
15710993111,17cyclictest7832-21seq23:38:211
1571099310,2cyclictest0-21swapper/100:01:501
1571099310,1cyclictest2351-21diskmemload21:53:061
1571299301,6cyclictest0-21swapper/320:41:393
1571299300,28cyclictest0-21swapper/319:50:383
1571099309,11cyclictest2351-21diskmemload22:19:001
1571099309,11cyclictest2351-21diskmemload22:19:001
1571099306,22cyclictest17136-21grep19:10:251
1571099306,22cyclictest17136-21grep19:10:251
1571099303,12cyclictest2351-21diskmemload22:47:431
15710993023,5cyclictest0-21swapper/119:59:511
15710993023,5cyclictest0-21swapper/119:59:511
1571099301,1cyclictest0-21swapper/120:24:411
15710993011,16cyclictest17716-21cat20:29:591
1571099300,7cyclictest6811-21ssh21:36:221
1571099300,6cyclictest29107-21wc23:05:261
1571099300,4cyclictest3660-21rm22:22:331
1571099300,1cyclictest0-21swapper/123:10:141
1571299291,14cyclictest0-21swapper/321:05:253
1571199291,25cyclictest2209-21users19:50:252
1571099298,18cyclictest9520-21expr23:15:191
1571099291,25cyclictest7541-21grep20:05:181
1571099291,15cyclictest141rcu_preempt22:14:051
15710992911,16cyclictest25735-21sh23:27:141
1571099290,3cyclictest19597-21/usr/sbin/munin00:10:261
1571099290,26cyclictest23797-21sshd20:42:001
1570999291,3cyclictest640-21snmpd19:55:010
1570999291,3cyclictest640-21snmpd19:55:010
1571199281,25cyclictest0-21swapper/219:34:202
15710992813,13cyclictest7956-21cut22:50:171
1571099281,25cyclictest0-21swapper/120:38:031
1571099280,1cyclictest0-21swapper/120:15:141
1570999281,25cyclictest0-21swapper/020:21:060
1570999281,25cyclictest0-21swapper/020:03:550
1570999281,25cyclictest0-21swapper/019:29:100
1571299276,18cyclictest20412-21sort19:20:153
1571299270,24cyclictest17140-21grep19:10:253
1571299270,24cyclictest17140-21grep19:10:243
15711992723,2cyclictest0-21swapper/220:49:092
15711992722,3cyclictest3280-21perl19:55:172
15711992722,3cyclictest3280-21perl19:55:172
1571199271,24cyclictest0-21swapper/221:03:102
1571199271,24cyclictest0-21swapper/220:01:582
1571099270,3cyclictest1954-21/usr/sbin/munin21:05:261
1571099270,1cyclictest0-21swapper/119:20:141
15709992725,1cyclictest0-21swapper/020:41:460
15709992723,2cyclictest0-21swapper/019:50:270
1570999271,2cyclictest0-21swapper/020:28:200
1570999271,25cyclictest0-21swapper/020:57:110
1570999271,25cyclictest0-21swapper/020:45:080
1570999271,24cyclictest0-21swapper/021:00:340
1570999271,24cyclictest0-21swapper/020:50:340
1570999271,24cyclictest0-21swapper/020:38:520
1570999271,24cyclictest0-21swapper/020:30:010
1570999271,24cyclictest0-21swapper/020:10:460
1570999271,24cyclictest0-21swapper/019:35:420
1570999271,24cyclictest0-21swapper/019:32:330
1571299261,22cyclictest3241-21tune2fs19:55:173
1571299261,22cyclictest3241-21tune2fs19:55:173
1571299260,23cyclictest20717-21cat20:35:193
1571199261,23cyclictest135462sleep220:20:082
1571199261,23cyclictest0-21swapper/220:55:102
1571199261,23cyclictest0-21swapper/220:40:522
1571199261,23cyclictest0-21swapper/220:38:462
1571199261,23cyclictest0-21swapper/220:28:352
1571199261,23cyclictest0-21swapper/220:16:422
1571199261,23cyclictest0-21swapper/220:10:162
1571199261,23cyclictest0-21swapper/220:05:302
1571199261,23cyclictest0-21swapper/219:46:482
1571199261,23cyclictest0-21swapper/219:41:052
1571099264,20cyclictest18732-21expr19:15:201
1570999263,16cyclictest22227-21ssh23:25:120
15709992624,1cyclictest0-21swapper/020:17:360
1570999261,1cyclictest0-21swapper/019:46:300
1570999261,1cyclictest0-21swapper/019:40:500
15711992523,1cyclictest0-21swapper/221:08:242
15711992523,1cyclictest0-21swapper/219:27:422
1571199251,23cyclictest0-21swapper/220:50:072
1571199251,23cyclictest0-21swapper/220:32:542
1571199251,1cyclictest0-21swapper/219:36:352
15709992512,10cyclictest1363-21timerwakeupswit22:20:260
176512240,3sleep00-21swapper/021:45:120
1570999249,9cyclictest20914-21sh23:00:120
1570999249,12cyclictest28002sleep021:58:010
15709992422,1cyclictest0-21swapper/021:07:470
15711992321,1cyclictest0-21swapper/219:22:232
15711992314,7cyclictest8532-21ntp_states22:50:232
15711992313,7cyclictest20397-21wc22:10:282
1570999239,12cyclictest31906-21wc22:20:000
1570999239,12cyclictest31906-21wc22:20:000
15709992312,8cyclictest1-21systemd00:00:000
163012220,1sleep00-21swapper/021:44:080
15711992213,6cyclictest25154-21ls21:50:192
15711992212,6cyclictest52950irq/141-enp3s0-00:00:132
1570999222,13cyclictest19180-21ssh00:35:140
34122210,1sleep20-21swapper/223:10:262
15711992115,4cyclictest20452-21sendmail23:00:002
15711992114,5cyclictest28-21ksoftirqd/222:35:002
15711992114,4cyclictest14752-21sh22:06:182
15711992112,7cyclictest10490-21diskstats23:40:142
1571199211,16cyclictest21848-21sed23:25:012
15709992113,6cyclictest21585-21awk22:35:260
15709992113,6cyclictest15952-21head23:20:170
15709992113,4cyclictest636-21runrttasks21:20:250
15709992111,7cyclictest4175-21apt-get21:35:130
1571199209,7cyclictest52950irq/141-enp3s0-22:35:132
1571199207,11cyclictest18187-21expr00:10:142
1571199206,12cyclictest10698-21proc_pri21:15:232
15711992014,4cyclictest29518-21sh23:05:382
15711992013,5cyclictest278-21systemd-journal00:40:002
15711992012,6cyclictest20889-21seq21:47:012
15711992011,7cyclictest32448-21ntpq00:20:202
1570999202,12cyclictest8585-21apt-get23:15:120
15709992018,1cyclictest0-21swapper/019:22:510
15709992014,5cyclictest12-21ksoftirqd/021:35:000
15709992013,4cyclictest25554-21sh23:50:450
15709992012,6cyclictest20649-21awk22:10:280
15709992012,6cyclictest12567-21entropy22:05:160
1570999200,4cyclictest0-21swapper/000:17:270
31542190,4sleep23153-21df_inode21:10:152
1571199198,7cyclictest52950irq/141-enp3s0-21:35:032
1571199192,15cyclictest8963-21ssh22:26:102
1571199192,13cyclictest4831-21apt-get22:00:042
1571199191,8cyclictest29468-21/usr/sbin/munin23:30:172
15711991914,3cyclictest30283-21sed21:30:182
15711991914,3cyclictest25224-21uptime23:50:252
15711991913,4cyclictest467-21sh21:55:412
15711991913,4cyclictest18332-21sh21:20:472
15711991913,4cyclictest13549-21sh21:41:232
15711991912,5cyclictest4123-21sed23:35:182
15711991912,5cyclictest25438-21sh21:26:122
15711991912,5cyclictest23067-21memory23:25:192
15711991912,5cyclictest16576-21seq19:10:192
15711991912,5cyclictest16576-21seq19:10:182
15711991912,5cyclictest12721-21sensors00:05:242
15711991910,6cyclictest17754-21grep23:45:202
15711991910,6cyclictest17754-21grep23:45:202
1570999192,12cyclictest27633-21ssh23:05:140
15709991914,3cyclictest19141-21timerandwakeup19:15:250
15709991913,4cyclictest6022-21cut22:00:150
15709991912,5cyclictest375-21cat00:20:240
15709991912,4cyclictest636-21runrttasks23:35:230
15709991911,6cyclictest5831-21ntpq00:00:230
1570999191,16cyclictest5497-21ssh21:11:260
15709991911,6cyclictest16525-21sed19:10:190
15709991911,6cyclictest16525-21sed19:10:190
15709991911,5cyclictest52750irq/139-enp3s0-23:30:020
1570999191,15cyclictest20457-21run-parts23:00:000
15709991910,6cyclictest2168-21expr23:10:150
1571199188,8cyclictest27059-1kworker/u9:123:55:262
1571199182,6cyclictest32229-21apt-get22:20:132
15711991813,3cyclictest22137-21processes23:00:232
15711991812,4cyclictest12713-21sh23:17:502
15711991812,4cyclictest11622-21sh00:29:372
1571199181,15cyclictest26930-21diskstats22:40:152
15711991810,6cyclictest14115-21unixbench_singl00:30:262
1570999185,3cyclictest2351-21diskmemload22:49:010
1570999182,5cyclictest18718-21munin-run00:35:000
1570999182,10cyclictest26553-21apt-get22:40:130
15709991811,5cyclictest16593-21ls23:45:000
15709991811,5cyclictest12112-21cat00:05:180
15709991810,5cyclictest636-21runrttasks00:27:510
1570999178,6cyclictest18236-21cpuspeed_turbos00:10:140
1570999174,3cyclictest2351-21diskmemload22:50:100
1570999173,4cyclictest13735-21gltestperf22:30:150
1570999172,10cyclictest9533-21apt-get21:15:140
15709991712,4cyclictest0-21swapper/023:47:100
15709991712,4cyclictest0-21swapper/023:47:100
1570999171,12cyclictest7114-21/usr/sbin/munin22:25:160
15709991611,3cyclictest2351-21diskmemload21:25:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional